Feeds

Benchmarks, bunkum and baloney

Performance figure hype

  • alert
  • submit to reddit

Business security measures using SSL

Comment There are three types of claims made by vendors with respect to benchmarks: pure performance figures, competitive performance comparison, and TPC figures. In my view, none of these has marketing value, though the second of these potentially has some utility for the supplier. Let me explain.

A good example of the over-hyping of pure performance figures can be found in the event streaming market. More or less every vendor in the market puts out figures about its event handling capability, ranging from tens, through hundreds of thousands of transactions per second to claims in the millions of transactions per second.

Now, what don't these claims encompass? First, a definition of what an event consists of: if my event is 10 bytes and yours is 20 bytes then, all other things being equal (which they aren't), I am going to get better results than you are. Second, even if we agree on an event size, then this takes no account of the fact that some vendors can process deltas and some can't. That is, the ability to process only changes to the data rather than having to process events in their entirety.

Thirdly, there is the processor issue. Unless everybody is running on the same hardware the relative merits of one performance figure versus another are opaque to say the least. Moreover, if you are going to have to spend hundreds of thousands of dollars for an implementation and you need a two CPU system as opposed to a single CPU system (or even 16 instead of eight) then how much difference is the extra cost of hardware going to make?

And finally, how much throughput do you really need? In equity trading in New York 150,000 transactions per sec is roughly the current rate. On the other hand, for mobile call rate look-up less than a tenth of this figure is more than enough. While rates are likely to increase, the performance of all the vendors is more than enough or, where it is not, you will find that these vendors do not target the very highest throughput requirements such as algorithmic trading.

Anyway, enough of pure performance and on to competitive performance benchmarks. An example of this that recently came up was an ETL vendor that claimed to have the fastest throughput on the market. Apart from the fact that it had done a performance comparison with just one other vendor (and that was not Ab Initio) as opposed to the 40 or so in the market, it, too, made a big thing of the fact that its figures were for a single CPU system as opposed to a two CPU system—see my remarks above.

However, in the ETL space it isn't just about throughput, there are also issues depending on the transformations you are doing, for example, and this vendor did a number of different tests, in some of which it came off better than others.

Naturally, it published only the ones most favourable to it (incidentally, I am not naming the company here since, in my experience, every vendor does the same thing), which is why the figures are nothing more than marketing puff. However, where there was real value in doing these comparisons was in allowing the supplier to discover where it did not do so well, so that it could work on this in the labs and improve its performance in the future.

Finally, as far as TPC figures are concerned: these are artificial, easily suborned and are won by the companies that have the most money to throw at them. There is, however, one other class of benchmark that has value, and these are the proofs of concept and benchmarks that users undertake. They have value because they directly relate to the user's requirements and then you can get a true feel for the cost of ownership.

To conclude, benchmarks have value when they are conducted for the benefit of individual users and for vendors wanting to identify weaknesses in their product. As for the marketing hype, it's like boys in the locker room claiming "mine's bigger than yours" followed by "no, it isn't - yes, it is" arguments. And it's of about as much value. I could go on with this metaphor but I think I'd better stop.

Copyright © 2006, IT-Analysis.com

New hybrid storage solutions

More from The Register

next story
'Windows 9' LEAK: Microsoft's playing catchup with Linux
Multiple desktops and live tiles in restored Start button star in new vids
Not appy with your Chromebook? Well now it can run Android apps
Google offers beta of tricky OS-inside-OS tech
New 'Cosmos' browser surfs the net by TXT alone
No data plan? No WiFi? No worries ... except sluggish download speed
iOS 8 release: WebGL now runs everywhere. Hurrah for 3D graphics!
HTML 5's pretty neat ... when your browser supports it
Greater dev access to iOS 8 will put us AT RISK from HACKERS
Knocking holes in Apple's walled garden could backfire, says securo-chap
NHS grows a NoSQL backbone and rips out its Oracle Spine
Open source? In the government? Ha ha! What, wait ...?
Google extends app refund window to two hours
You now have 120 minutes to finish that game instead of 15
Intel: Hey, enterprises, drop everything and DO HADOOP
Big Data analytics projected to run on more servers than any other app
SUSE Linux owner Attachmate gobbled by Micro Focus for $2.3bn
Merger will lead to mainframe and COBOL powerhouse
prev story

Whitepapers

Providing a secure and efficient Helpdesk
A single remote control platform for user support is be key to providing an efficient helpdesk. Retain full control over the way in which screen and keystroke data is transmitted.
WIN a very cool portable ZX Spectrum
Win a one-off portable Spectrum built by legendary hardware hacker Ben Heck
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Security and trust: The backbone of doing business over the internet
Explores the current state of website security and the contributions Symantec is making to help organizations protect critical data and build trust with customers.