Takashi Kokubun (known as k0kubun), a Ruby JIT committer, asked the community for more benchmarks on upcoming Ruby 2.6 JIT. I put together one for generating PDF documents with InvoicePrinter (based on Prawn).
Making a JIT benchmark
To benchmark the code with and without JIT I used Takashi’s own gem benchmark_driver. The README will guide you on how to create a simple YAML file.
Since I don’t use rbenv as Takshi, but ruby-install and chruby, I just pass the executables with -e option (I even sent a patch some time ago for this option since I didn’t realize it’s there!).
However there was still one gotcha in handling the Ruby arguments. If you use old version of the gem you need to pass arguments after the executable path separated by comma (notice the ,--jit):
$ benchmark-driver benchmarks/render.yml -e '/home/strzibny/.rubies/ruby-2.6.0-preview3/bin/ruby;/home/strzibny/.rubies/ruby-2.6.0-preview3/bin/ruby,--jit'
This is now fixed in the latest versions so feel free to run it as:
$ benchmark-driver benchmarks/render.yml -e '/home/strzibny/.rubies/ruby-2.6.0-preview3/bin/ruby;/home/strzibny/.rubies/ruby-2.6.0-preview3/bin/ruby --jit'
I sent a patch to fix the README.
What’s been benchmarked?
In this blog post I am benchmarking the generation of a simple invoice using InvoicePrinter’s render method. It’s variant on this example. InvoicePrinter is based on Prawn PDF library.
I am running the benchmark on my Thinkpad X1 Carbon 5th generation with i7-7600U processor, 16 GB RAM and SSD. I already committed this benchmark to the InvoicePrinter repository, feel free to suggest improvements!
Results
Without further ado here is the graph:
As you can see Ruby 2.6 is a little bit better, but pretty much the same in iterations per second. Here is the text output for loop_count of 1000:
Comparison:
render
/home/strzibny/.rubies/ruby-2.6.0-preview3/bin/ruby: 100.4 i/s
/home/strzibny/.rubies/ruby-2.5.0/bin/ruby: 96.9 i/s - 1.04x slower
/home/strzibny/.rubies/ruby-2.6.0-preview3/bin/ruby,--jit: 86.6 i/s - 1.16x slower
And 10000:
Comparison:
render
/home/strzibny/.rubies/ruby-2.6.0-preview3/bin/ruby: 99.8 i/s
/home/strzibny/.rubies/ruby-2.5.0/bin/ruby: 95.8 i/s - 1.04x slower
/home/strzibny/.rubies/ruby-2.6.0-preview3/bin/ruby,--jit: 64.5 i/s - 1.55x slower
Comparison:
render
/home/strzibny/.rubies/ruby-2.5.0/bin/ruby: 106.1 i/s
/home/strzibny/.rubies/ruby-2.6.0-preview3/bin/ruby: 104.6 i/s - 1.01x slower
/home/strzibny/.rubies/ruby-2.6.0-preview3/bin/ruby --jit: 73.5 i/s - 1.44x slower
Depending on a run Ruby 2.6 can render here around 100 invoices per second. 4-5 more or less than 2.5. In other words it seems the performance is pretty much the same. The –jit version runs 1,1-1,6 times slower which is not yet great.
What about memory?
Comparison:
render
/home/strzibny/.rubies/ruby-2.5.0/bin/ruby: 18316000.0 bytes
/home/strzibny/.rubies/ruby-2.6.0-preview3/bin/ruby: 22812000.0 bytes - 1.25x larger
/home/strzibny/.rubies/ruby-2.6.0-preview3/bin/ruby,--jit: 75672000.0 bytes - 4.13x larger
Ruby 2.6 uses a little more memory than 2.5 on my example. The –jit version even 4 times as Ruby 2.5.
Feel free to run it yourself and report.
I hope this benchmark can help Takashi somehow and we see some improvements for JIT when using Prawn in future.
