Pendragon Software Corporation, developer of the widely-used and recognized CaffeineMark, said yesterday that a Sun Microsystems Java compiler purposefully identifies part of CaffeineMark and then produce a misleading score. Sun issued a press release in October using this bogus result to claim that its Solaris operating system runs Java faster than any other environment. Sun is denying the charges.
Pendragon Software analyzed the results and found that Solaris inexplicably scored 50 times higher than any other operating system.
"There were three possible explanations for this result: either the compiler was performing some impressive optimizations, there was a bug in the compiler, or there was a bug in the benchmark," Pendragon said in a press release.
To test the results, the company introduced a small change in the benchmark and ran it on Solaris and other platforms. For all operating systems except Solaris, the differences were minimal. On Solaris, however, the benchmark ran 300 times slower. The company then determined that the Java Virtual Machine in Solaris was simply pattern matching for the CaffeineMark benchmark "and performing optimisations that do not work in the general case." Further inspection led Pendragon to a block of code in the JVM that matches the CaffeineMark bytecode