Pet vs Pet: MS opens .NET benchmarking wars

How fast is that doggy in the window?

Updated Microsoft thinks .NET is now mature to be put in the front line of the benchmarketing wars. Not that "mature" is ever a word you can use in the context of benchmark battles.

The Beast has taken aim at a piece of sample code devised by Sun and used principally to demonstrate the portability of Java2 Enterprise Edition, which Sun posts here.

In response, Microsoft has built its own pet shop here and claims that the .NET version (.PET?) uses less than a third as many lines of code as the Java pet store and runs up to 28 times faster (or eight times faster uncached). And using the CPU cycles it has saved you, it performs background Internet searches for your long-lost relatives, and interesting pictures from binary newsgroups (We made the last one up, but you get the general idea).

Microsoft uses Oracle's app server benchmark in which an unnamed application server is used as the J2EE front end. The Oracle back end is 8i, Microsoft uses SQL Server but says it populated the data exactly as Oracle did in its example.

The Oracle/J2EE example uses fairly old Sun hardware: A 4-way E3500 for the app server, and a 10-way Ultra running Solaris 2.6 for the database. Microsoft uses a 2-way ProLiant for the app server and a 4-way 550Mhz ProLiant for the database in its .NET Pet Shop run.

Sun described the .PET exercise as "highly deceptive tactics".

The J2EE Pet Store, Sun told us, " is a sample application meant to demonstrate certain design patterns, and is not a benchmark."

"The J2EE version of this application is not tuned for performance. Microsoft claims that the version tested by Oracle was "an optimized and highly tuned version of the Java Pet Store".

However, Oracle's white paper detailing the results in question state that the 'Pet Store was not designed as a fully-fledged performance workload and would require substantial modifications to be used as one.'

A more detailed rebuttal can be found here, where poster Ray Cromwell concludes that the Sun demo code is "'overengineered' for maximum abstraction... Almost no one writes apps according to the Sun blueprints, which are an ideal "top down" OO approach."

And a more direct clue to the performance of the .PET comes from one if its designers, Scott Stanfield, at MSDN itself:

The .PET demo is really "a relatively thin layer that wraps stored procedures," he notes.

J2EE isn't the only target of the exercise. At GotDotNet, a site designed to look like a SourceForge/FreshMeat-style community developer but really an alternative front end to MSDN, Microsoft also takes aim at IBM's WebSphere app server. IBM has already responded here.

So what does all this sound and fury signify, exactly? Well Microsoft clearly intends to use performance as a key marketing plus. Microsoft field sales staff who have already been requested to tattoo anti-Linux intelligence on their backsides by Windows VP Brian Valentine may well need to find more earth-facing real estate for .PET benchmarks.

But benchmark tactics can easily backfire. Microsoft made bold claims with a Visual Basic benchmark several years ago, that famously rebounded. Microsoft published test results purporting to show VB outperforming Delphi. In practice the Microsoft application was making local calls, while the Delphi test was querying a remote database, and unsurprisingly VB ran several thousand times faster.

Perhaps someone can try and convince us why .NET is intrinsically faster than J2EE, when the architectures are essentially similar. .NET may well be faster, it may not, but since each rival platform can be highly optimised, or in the case of Java, re-compiled to provide native performance, neither architecture has performance bragging rights.

The .PET example proves little beyond that Wintel hardware provides more bang per buck than than old Sun hardware, but then we knew that anyway. ®

Sponsored: A modern data architecture with Apache Hadoop