Implementing a custom ExUnit assert to test PDF output

Making a custom assert can clean up and DRY your tests. Here is how I refactored a few ExUnit tests to use a single assert to test PDF output.

I needed to test that an output PDF looks visually the same as what is expected. I say visually, because the binary representation can be a little different. There might be metadata that changes, but I am only interested in what the PDF looks like.

To test it I am using a Python utility called diff-pdf-visually. If we want to call it from Elixir, we can use System.cmd/2 to shell out and run it. In the context of an ExUnit test, this could look like:

  ...
  describe "pdf_crop/1" do
    test "crops the passed PDF", %{} do
      result_path = Example.pdf_crop(...)

      case System.cmd("diff-pdf-visually", [
                 result_path,
                 @example_cropped_pdf_path
               ]) do
            {_out, 0} ->
              assert true

            {_out, _error_code} ->
              flunk "Files are not visually the same"
      end
    end
  end
  ...

As you can tell this utility returns successfully if the PDF files are the same and errors if not. We can use that to simply call the usual assert to pass or fail the test. We can also leverage ExUnit’s flunk/1to fail with a message.

This is how we might start, but not how we want to finish. Having similar tests across your test suite is a maintenance pain and it doesn’t look very good.

To that end let’s implement a custom assert as a macro:

# test/support/assertions/visual_assert.ex
defmodule Assertions.VisualAssert do
  @doc """
  Asserts that the two PDF files are visually the same.

  Uses diff-pdf-visually under the hood.
  """
  defmacro assert_same_looking_pdf(original, new) do
    quote do
      case System.cmd("diff-pdf-visually", [
             "--time",
             "120",
             unquote(original),
             unquote(new)
           ]) do
        {_out, 0} ->
          assert true

        {_out, _error_code} ->
          flunk "Files are not visually the same"
      end
    end
  end
end

We cannot just create a helper function, because we need access to assert/1 and flunk/1 functions. Also read about quotes and unquotes.

Once saved, we might want to import it inside the test/support/conn_case.ex file:

defmodule DigiWeb.ConnCase do
  ...

  using do
    quote do
      ...
      # Import our new assertions
      import Assertions.VisualAssert

Then we can rewrite our test as:

  ...
  describe "pdf_crop/1" do
    test "crops the passed PDF", %{} do
      result_path = Example.pdf_crop(...)

      assert_same_looking_pdf result_path, @example_cropped_pdf_path
    end
  end
  ...

Much much better. A failed test reports now as:

1) test pdf_crop/1 crops the passed label PDF (PdfCropperTest)
     test/pdf_cropper_test.exs:14
     Files are not visually the same
     code: assert_same_looking_pdf(
     stacktrace:
       test/pdf_cropper_test.exs:29: (test)

.

As you can imagine, debugging why those PDF files are not the same can take a while.

diff-pdf-visually actually supports --time option to give you some time to investigate these files, but that’s not something we can rely on in these kinds of tests.

We can do something else though. Let’s make new temporary files on failures and report them to the user:

defmodule Assertions.VisualAssert do
  ...
  defmacro assert_same_looking_pdf(original, new) do
    quote do
      case System.cmd("diff-pdf-visually", [
             unquote(original),
             unquote(new)
           ]) do
        {_out, 0} ->
          assert true

        {_out, _error_code} ->
          tmp_dir = Temp.path!()
          File.mkdir(tmp_dir)
          File.cp(unquote(original), "#{tmp_dir}/original.pdf")
          File.cp(unquote(new), "#{tmp_dir}/new.pdf")

          flunk("""
          Files are not visually the same

          You can inspect the files:

          #{tmp_dir}/original.pdf
          #{tmp_dir}/new.pdf
          """)
      end
    end
  end
end

We create a new temporary directory, copy the files in, and report the file names to the user running the test suite!

Now a fail ends up being:

1) test pdf_crop/1 crops the passed label PDF (PdfCropperTest)
     test/pdf_cropper_test.exs:14
     Files are not visually the same

     You can inspect the files:

     /tmp/f-1592801823-597798-1p6adva/original.pdf
     /tmp/f-1592801823-597798-1p6adva/new.pdf

     code: assert_same_looking_pdf(
     stacktrace:
       test/pdf_cropper_test.exs:29: (test)

.

With the final change in place, we can now simply run xdg-open (on Linux) with the reported path and investigate what’s wrong with the PDF output.

Any comments? Write me a DM on Twitter.

Before you leave…

I am writing an introductory book on web application deployment. Networking, processes, systemd, backups, and all your usual suspects.

Check it out →