RUBI 2012 implementation race

A set of benchmarks from ruby-1.9.3-p125 was used. All tests were run on:

OS: OSX Lion 10.7.3

Processor: 2.3GHz i5

Memory: 8GB 1333 MHz DDR3

SSD: OCZ Vertex 3 Max IOPS SATA III 2.5 "120GB

Implementations:

- ruby 1.8.7p249 (system ruby)

- ruby 1.9.3p125

- ruby 2.0.0dev (2012-02-25 trunk 34796)

- MacRuby 0.12 (ruby 1.9.2) (Nightly build)

- maglev 1.0.0 (ruby 1.8.7)

- rubinius 1.2.4 (1.8.7 release 2011-07-05 JI)

- rubinius 2.0.0dev (1.9.3 e22ed173 JI)

- jruby 1.7.0.dev (ruby-1.9.3-p28) (Java HotSpot (TM) 64-Bit Server VM 1.7.0_04-ea)

- jruby 1.6.7 (ruby-1.8.7-p357) (Java HotSpot (TM) 64-Bit Server VM 1.7.0_04-ea)

')

JRuby started with the flags - server -Xinvokedynamic.constants = true

Compiler matter

From time to time I see posts about increasing ruby performance by applying patches, but what if we go even further and try to increase performance by compiling it with the latest compilers? I decided to check.

Here is a list of compilers:

- gcc version 4.2.1 (build 5658) (LLVM build 2336.9.00)

- Apple clang version 3.1 (tags / Apple / clang-318.0.45) (based on LLVM 3.1svn)

- gcc version 4.2.1 (Apple Inc. build 5666)

- gcc version 4.7.0 20120218 (experimental) (GCC)

#!/bin/bash compilers=( gcc gcc-4.2 gcc-4.7 clang ) for i in "${compilers[@]}"; do CC=$i ./configure --disable-install-doc --prefix ~/Projects/benches/mri/1.9.3-p125-$i time make -j4 make install done $ ruby driver.rb -v -o ~/Projects/benches/compilers-bench.txt \ --executables='~/Projects/benches/mri/1.9.3-p125-gcc/bin/ruby; ~/Projects/benches/mri/1.9.3-p125-gcc-4.2/bin/ruby; ~/Projects/benches/mri/1.9.3-p125-gcc-4.7/bin/ruby; ~/Projects/benches/mri/1.9.3-p125-clang/bin/ruby'

Results:

~ 20% of the difference (the benchmark was run several times and I chose the best result) between llvm-gcc, used by default, and gcc-4.7 in synthetic tests. Not bad, I think.

Make sure nothing breaks with gcc-4.7:

 PASS all 943 tests KNOWNBUGS.rb . PASS all 1 tests

I want to make sure yourself

This can be easily done if homebrew is installed:

 $ brew install https://raw.github.com/etehtsea/formulary/009735e66ccabc5867331f64a406073d1623c683/Formula/gcc.rb --enable-cxx --enable-profiled-build --use-gcc

... an hour later:

 $ CC=gcc-4.7 ruby-build 1.9.3-p125 ~/.rbenv/versions/1.9.3-p125

Note trans. About ruby-build here .

And what about% any other implementation of ruby%?

I could not stop and went about my curiosity and launched a benchmark on other popular implementations of ruby and versions of MRI. I do not cite full logs here, just highlight interesting moments.

Do not use the default ruby

This is a catch!

bm_vm_thread_mutex3.rb

 # 1000 , 1  require 'thread' m = Mutex.new r = 0 max = 2000 (1..max).map{ Thread.new{ i=0 while i<max i+=1 m.synchronize{ r += 1 } end } }.each{|e| e.join } raise r.to_s if r != max * max

 $ time ~/.rbenv/versions/1.8.7-p357/bin/ruby bm_vm_thread_mutex3.rb real 0m3.093s user 0m3.078s sys 0m0.013s $ /usr/bin/ruby -v ruby 1.8.7 (2011-12-28 patchlevel 357) [i686-darwin11.3.0] $ time /usr/bin/ruby bm_vm_thread_mutex3.rb ^Cbm_vm_thread_mutex3.rb:18:in `join': Interrupt from bm_vm_thread_mutex3.rb:18 from bm_vm_thread_mutex3.rb:7:in `each' from bm_vm_thread_mutex3.rb:7 real 3m54.930s user 3m54.122s sys 0m0.918s

Even if you are not going to use Thread, here are the results without this test:

ruby 1.8.7 (2010-01-10) - 572.863 seconds

ruby 1.9.3p125 (2012-02-16) - 211.655 seconds

Not passed tests for 1.8:

- bm_app_factorial.rb

- bm_so_ackermann.rb

Rubinius 1.2.4 vs. 2.0.0-dev

I read that in 2.0.0-dev there is no more GIL, etc. etc., but this upcoming version is noticeably slower.

The most noticeable slowdown is still in the same test bm_vm_thread_mutext3.rb:

- rubinius 1.2.4 (1.8.7 release 2011-07-05 JI) - 3.260 seconds

- rubinius 2.0.0dev (1.9.3 e22ed173 yyyy-mm-dd JI) - 207.711 seconds

Here are the tests where the difference is most noticeable:

But the results without these tests:

1.2.4 - 518.861 seconds

2.0.0dev - 606.811 seconds

Rubinius was never fast, but was still 15% slower.

Did not pass the tests:

- factorial 4k instead of 5k

- bm_loop_generator.rb

- bm_so_ackermann.rb

- bm_vm_thread_pass_flood.rb (test timed out waiting for execution)

MacRuby 0.12 (Nightly)

MacRuby is what you need if you are writing a desktop application for OS X, or simply using the OS X API, but from the point of view of performance, there is no sense to use it.

First of all, eval in MacRuby (bm_vm2_eval.rb) is rather slow:

ruby 1.9.3p125 (2012-02-16) - 29.681 seconds

MacRuby 0.12 (ruby 1.9.2) - 232.257 seconds

bm_vm2_eval.rb

 i=0 while i<6_000_000 i+=1 eval("1") end

So as erb parsing and creation Class instances:

bm_app_erb.rb

 # # Create many HTML strings with ERB. # require 'erb' data = DATA.read max = 15_000 title = "hello world!" content = "hello world!\n" * 10 max.times{ ERB.new(data).result(binding) } __END__ <html> <head> <%= title %> </head> <body> <h1> <%= title %> </h1> <p> <%= content %> </p> </body> </html>

1.9.3p125 - 1.817 seconds

MacRuby - 81.808 second

bm_vm3_clearmethodcache.rb

 i=0 while i<200_000 i+=1 Class.new{ def m; end } end

1.9.3p125 - 0.748 seconds

MacRuby - 86.573 seconds

Not passed tests:

- bm_loop_generator.rb

- bm_so_count_words.rb

- bm_so_nsieve_bits.rb (timed out)

- bm_vm_thread_create_join.rb (timeout exceeded)

Maglev 1.0

Interestingly, MagLev has similar problems:

bm_vm2_eval.rb - 754.028 seconds

bm_vm3_clearmethodcache.rb - 33.785 seconds

JRuby 1.6 vs. 1.7.0-dev

JRuby 1.7.0-dev has similar performance c 1.6.6, with a slight improvement on the bm_vm_thread_mutex3.rb test:

1.7.0-dev - 14.381 seconds

1.6.6 - 202.552 seconds

The overall result is:

1.7.0-dev - 257.584 seconds

1.6.6 - 229.502 seconds

Not passed tests:

- bm_io_select.rb

MRI 2.0.0-dev vs 1.9.3-p125

The same situation with the dev branch of MRI. Improvement only in the bm_vm_thread_create_join.rb test:

ruby 2.0.0dev (2012-02-25 trunk 34796) - 2.806 seconds

ruby 1.9.3p125 (2012-02-16) - 9.239 seconds