'The Java Faster than C++' Benchmark Revisited

Introduction

"There are lies, damn lies and Benchmarks".

A while back I was Googling something and and happened across this comparison: The Java Faster than C++ Benchmark. Having witnessed the "real world" performance of comparable applications between the two languages, that phrase (along with a red flashing B.S.) kept popping into my mind as I reviewed the methodology and code used in that article. Digging a little further, I found out that the article was based primarily on code from the Great Computer Language Shootout, so changes made to the code, build settings, etc. here may apply to that code as well.

Also, I thought that GCC was really under-represented in the original article as well as a follow-up that featured the Intel compiler (where the author suggested GCC was a dog and C++ IO was slow). The follow-up article can be found here: Comparing C++ performance with Java.

Given that all three premises (Java is faster than C++, GCC is a dog and C++ IO is slow) are highly questionable from my experience at least, and apparently 'The Java Faster' article was 'featured' (given wide circulation) on Slashdot I thought it would be worthwhile to revisit this series of benchmarks and post the results on the web for all to flame.

Reading further into the 'The Java Faster than C++ Benchmark' article, I also noticed several links to other benchmarks. This article also addresses (the mostly false) claims made by those references. Some of the claims are pretty remarkable - three of them use incorrect results from the same test to conclude Java performs better than C when correct results actually show the opposite by a good margin (more on that below). Also, the samples provided by one Java vendor (Jet) use debug builds for the C code in their comparisons. A couple of those benchmarks (linpack and penta) are compared more fairly below as well.

But first, a few thoughts..

For the most part, these benchmarks are highly 'artificial' in that the same code probably won't be found in many applications (with maybe a few exceptions). However, that doesn't mean that the stuff measured in them doesn't give an indication of 'real-world' performance for the same general types of problems. Also, for the most part, these tests really do not measure a lot of OOP specific performance even though Java highly encourages the use of OOP constructs when possible (and is typically used that way in most IS shops from what I've seen). So, I guess the original rational behind this particular code set may be questionable in that regard.

Methodology

I reviewed the build/runtime methodology and code from the original "Java Faster than C++" benchmark and modified both sets as described below. There are quite a few code changes; the original code had some "apples-to-oranges" inconsistencies and was not really optimized very well either, especially the C++ code. IMHO, if you're going to compare languages (heck, if you going to use a language at all), at least try and use the best each language reasonably has to offer. I found it really curious that the original Language Shootout Java code had annotations on how it was changed to optimize performance, while the C++ code had few.

• Synced OOP'ness. Some of the C++ code was using OOP constructs to compete against 'native' (primitive type) constructs in Java. More on that here. The modified code here changes that where it makes sense. Rest assured, I didn't convert OOP to C except in those cases where the corresponding Java code was doing its best to mimic C. If the Java code used objects, so does the C++ version of the benchmark to the extent possible given the two languages and my knowledge of each.
• Remove code that pre-allocated object buffers in a way specific to the benchmark, unless it is done in a constructor. For example, I removed some std::string::reserve() code for the C++ version of the strcat benchmark because it was doing this in the timed loop. The reasons behind this are 1) the class should be designed and written to handle stuff like that (buffer expansion) efficiently, 2) pre-allocating buffers (as in 'strcat') actually adds overhead if the class is efficient, and 3) I rarely if ever see that applied in 'real-world' code.
• Statically initialize objects if the language and algorithm allow it. IMHO, this is one of the most basic and important differences between Java and C++, especially where OOP performance is concerned. Just because Java has to allocate objects on the heap doesn't mean C++ does, or should in many cases.
• "inline", "const" and "finalize". If 'inline' was used in C++, "static final" or "final" (depending on context) was used for the corresponding Java method. Along those same lines, C++ const variables where made final variables in the corresponding Java code.
• Java Runtime settings. I added the Sun Java runtime switch -Xbatch because that is often used by Sun when it is fixing, err, running it's own benchmarks (If you find yourself complaining about static compiler switches for C & C++, take a look at this: SPECjbb.). As an aside, I ended up doing a lot more experimenting with the various Java runtime 'X' switches than any C++ compiler switches, the opposite of which is oft quoted as something that Java improves upon over static compilers.
• C++ Compiler settings. For the C++ builds, the compiler settings were changed to common and well documented optimization switches that allow the same executable to be run across different x86 chips: "g++ -O3 -fomit-frame-pointer -funroll-loops -mtune=pentium4" and "icc -O3 -mcpu=pentium4". More thoughts here.
• Compilers and runtimes. I significantly expanded the java runtimes competing in the test to better represent the choices available, because I was curious and also wanted to give Java the benefit of doubt whenever possible. I also added the Intel C++ compiler. I limited the C++ compiler selection due to time, ease of availability and money. Another rant here.
• Timing the Code. I wrapped all of the test loops with a timer so runtime start-up overhead was not a factor in the results.

For further information or to give the benchmarks a try yourself, download this zip file of the benchmark code along with the build and run scripts and more detailed system information.

Results

Below is a summary of the results after 'optimizing' the code and compiler/runtime settings. The tests were run on an Intel P4 2.2Ghz box with 512 MB RAM. Click here for the complete results or here for the run log. Also include in the run log and results are several other tests that do floating point and an OOP sort comparison.

As you can see, the C++ compilers did very well, turning in the best performance for 12 of the 14 benchmarks and often by a pretty significant margin. Contrary to opinions expressed in Comparing C++ performance with Java, GCC fairs pretty well, placing either 1st or 2nd (ahead of all of the Java systems) for 10 of the 14 tests, and C++ performs the best in both of the IO tests.

Interestingly, gcj does not perform nearly as well as g++ even though it is implemented as another front-end to GCC (as g++ is). Some of the difference I'm sure can be attributed to different levels of focus, but the large differences for most of the tests are probably also somewhat indicative of the relative performance potential of C++ and Java, especially for OOP constructs, where gcj really seems to lag. I always get a kick out of compiler vendors who advertise their new 4GL language (or whatever) as using the same "backend optimizer" as their C/C++ compiler, leaving the impression that the different languages will perform the same. In my experience, optimizations performed in the 'front-end' that are effected by language semantics (e.g.: cse, inlining, loop unrolling, etc.) often account for a greater performance improvement  than things like peephole optimizations done on the backend.. Garbage-in/Garbage-out.

Intel is clearly the top system for these tests, but I don't think that is a huge surprise.

Other Thoughts and Comparisons

At the end of the The Java Faster than C++ Benchmark, several links were given to other articles to support the author's conclusions. As I took a closer look at those, the good old B.S. detector went off again. The original links are bulleted, my comments are further indented.

• Java vs. C benchmark
• Java Pulling Ahead?
• Runtime Vs Static Compilation

This is my favorite example of how not to benchmark. Problem is, these results have propagated themselves across the web over the years and seem to be a favorite of Java zealots. I wonder how many PHB's have been brought to the Java religion based in-part on this bogus benchmark ;)

Each of the above articles has FFT code that was using a bass-ackward algorithm for computing the relative performance results. Basically, what the programmer was doing was this:

rate = run_test_rig(fraction_of_min_time);           // (rate is higher for faster runtimes)
cycles = run_test_rig(rate * min_time_required); // test rig has overhead not related to the measured work
result = cycles / run_time;                                   // the faster runtimes have a higher ratio of overhead to time

In a perfect world where there is no 'overhead' or rounding errors, every runtime would have scored the same.

The reported results primarily reflected overhead involved in running the "test rig" for more iterations. In effect, the faster systems were reported as being slower because the overhead is cumulative (more iterations = more overhead). What should have been reported was the time taken for the same amount of work, like 99 44/100% of benchmarks. The corrected code, the code from the links above, and results from my system can be found in this zip file. The FFT algorithm in the links above was adapted from here (the original SciMark code, not surprisingly, confirms my hypothesis - look for the SciMark FFT results here).

• FreeTTS performance case study - at Sun Research

First off, consider the source. Secondly, notice that (page 12 of the PDF) the final results are based on optimized algorithms to the Java code, but not the C code. Thirdly, no specific methodology (compiler settings, etc.) or Java source code was provided.

• Linux Number Crunching

See my results for the almabench test here. They seem to confirm the original, and then some.

• SNAP matrix benchmark - Java vs. C++ for matrix multiplication.

Not sure why this was included, unless it was to indicate that code compiled for debugging is relatively slow. When production quality code is compiled SNAP seems to indicate the opposite of the intent.

• Nine-Language Performance Round-Up - math and file I/O performance among nine languages

These are generally even more 'simplistic' and 'artificial' than the Shootout tests so there is not much to really glean from the results other than to maybe indicate that Sun Java math libraries can be outperformed by Python.

• Performance comparison C++, C# and Java

This contains code from both 'Nine-language Performance Round-Up' and 'The Shootout'. It looks like the 'Shootout' parts of the C++ code could be optimized as they were here. The author inexplicably turns-off "Omit-frame-pointers" and turns-on "Improve Float Consistency" for the C++ code in the last test. There is absolutely no reason to turn off "Omit-frame-pointers" and turn on "Improve Float Consistency", unless you want to handicap the results, which kind-of seems like the intent.

• Java vs. C

This link is apparently broken. Here's another one of interest though.

• Comparing C++ performance with Java

Mentioned above.

This website is hosted for free by . Get your own Free Website now!