Making a custom assert can clean up and DRY your tests. Here is how I refactored a few ExUnit tests to use a single assert to test PDF output.
I needed to test that an output PDF looks visually the same as what is expected. I say visually, because the binary representation can be a little different. There might be metadata that changes, but I am only interested in what the PDF looks like.
To test it I am using a Python utility called diff-pdf-visually. If we want to call it from Elixir, we can use System.cmd/2 to shell out and run it. In the context of an ExUnit test, this could look like:
...
describe "pdf_crop/1" do
test "crops the passed PDF", %{} do
result_path = Example.pdf_crop(...)
case System.cmd("diff-pdf-visually", [
result_path,
@example_cropped_pdf_path
]) do
{_out, 0} ->
assert true
{_out, _error_code} ->
flunk "Files are not visually the same"
end
end
end
...
As you can tell this utility returns successfully if the PDF files are the same and errors if not. We can use that to simply call the usual assert to pass or fail the test. We can also leverage ExUnit’s flunk/1to fail with a message.
This is how we might start, but not how we want to finish. Having similar tests across your test suite is a maintenance pain and it doesn’t look very good.
To that end let’s implement a custom assert as a macro:
# test/support/assertions/visual_assert.ex
defmodule Assertions.VisualAssert do
@doc """
Asserts that the two PDF files are visually the same.
Uses diff-pdf-visually under the hood.
"""
defmacro assert_same_looking_pdf(original, new) do
quote do
case System.cmd("diff-pdf-visually", [
"--time",
"120",
unquote(original),
unquote(new)
]) do
{_out, 0} ->
assert true
{_out, _error_code} ->
flunk "Files are not visually the same"
end
end
end
end
We cannot just create a helper function, because we need access to assert/1 and flunk/1 functions. Also read about quotes and unquotes.
Once saved, we might want to import it inside the test/support/conn_case.ex file:
defmodule DigiWeb.ConnCase do
...
using do
quote do
...
# Import our new assertions
import Assertions.VisualAssert
Then we can rewrite our test as:
...
describe "pdf_crop/1" do
test "crops the passed PDF", %{} do
result_path = Example.pdf_crop(...)
assert_same_looking_pdf result_path, @example_cropped_pdf_path
end
end
...
Much much better. A failed test reports now as:
1) test pdf_crop/1 crops the passed label PDF (PdfCropperTest)
test/pdf_cropper_test.exs:14
Files are not visually the same
code: assert_same_looking_pdf(
stacktrace:
test/pdf_cropper_test.exs:29: (test)
.
As you can imagine, debugging why those PDF files are not the same can take a while.
diff-pdf-visually actually supports --time option to give you some time to investigate these files, but that’s not something we can rely on in these kinds of tests.
We can do something else though. Let’s make new temporary files on failures and report them to the user:
defmodule Assertions.VisualAssert do
...
defmacro assert_same_looking_pdf(original, new) do
quote do
case System.cmd("diff-pdf-visually", [
unquote(original),
unquote(new)
]) do
{_out, 0} ->
assert true
{_out, _error_code} ->
tmp_dir = Temp.path!()
File.mkdir(tmp_dir)
File.cp(unquote(original), "#{tmp_dir}/original.pdf")
File.cp(unquote(new), "#{tmp_dir}/new.pdf")
flunk("""
Files are not visually the same
You can inspect the files:
#{tmp_dir}/original.pdf
#{tmp_dir}/new.pdf
""")
end
end
end
end
We create a new temporary directory, copy the files in, and report the file names to the user running the test suite!
Now a fail ends up being:
1) test pdf_crop/1 crops the passed label PDF (PdfCropperTest)
test/pdf_cropper_test.exs:14
Files are not visually the same
You can inspect the files:
/tmp/f-1592801823-597798-1p6adva/original.pdf
/tmp/f-1592801823-597798-1p6adva/new.pdf
code: assert_same_looking_pdf(
stacktrace:
test/pdf_cropper_test.exs:29: (test)
.
With the final change in place, we can now simply run xdg-open (on Linux) with the reported path and investigate what’s wrong with the PDF output.
