Feeds

Benchmarks, bunkum and baloney

Performance figure hype

  • alert
  • submit to reddit

Intelligent flash storage arrays

Comment There are three types of claims made by vendors with respect to benchmarks: pure performance figures, competitive performance comparison, and TPC figures. In my view, none of these has marketing value, though the second of these potentially has some utility for the supplier. Let me explain.

A good example of the over-hyping of pure performance figures can be found in the event streaming market. More or less every vendor in the market puts out figures about its event handling capability, ranging from tens, through hundreds of thousands of transactions per second to claims in the millions of transactions per second.

Now, what don't these claims encompass? First, a definition of what an event consists of: if my event is 10 bytes and yours is 20 bytes then, all other things being equal (which they aren't), I am going to get better results than you are. Second, even if we agree on an event size, then this takes no account of the fact that some vendors can process deltas and some can't. That is, the ability to process only changes to the data rather than having to process events in their entirety.

Thirdly, there is the processor issue. Unless everybody is running on the same hardware the relative merits of one performance figure versus another are opaque to say the least. Moreover, if you are going to have to spend hundreds of thousands of dollars for an implementation and you need a two CPU system as opposed to a single CPU system (or even 16 instead of eight) then how much difference is the extra cost of hardware going to make?

And finally, how much throughput do you really need? In equity trading in New York 150,000 transactions per sec is roughly the current rate. On the other hand, for mobile call rate look-up less than a tenth of this figure is more than enough. While rates are likely to increase, the performance of all the vendors is more than enough or, where it is not, you will find that these vendors do not target the very highest throughput requirements such as algorithmic trading.

Anyway, enough of pure performance and on to competitive performance benchmarks. An example of this that recently came up was an ETL vendor that claimed to have the fastest throughput on the market. Apart from the fact that it had done a performance comparison with just one other vendor (and that was not Ab Initio) as opposed to the 40 or so in the market, it, too, made a big thing of the fact that its figures were for a single CPU system as opposed to a two CPU system—see my remarks above.

However, in the ETL space it isn't just about throughput, there are also issues depending on the transformations you are doing, for example, and this vendor did a number of different tests, in some of which it came off better than others.

Naturally, it published only the ones most favourable to it (incidentally, I am not naming the company here since, in my experience, every vendor does the same thing), which is why the figures are nothing more than marketing puff. However, where there was real value in doing these comparisons was in allowing the supplier to discover where it did not do so well, so that it could work on this in the labs and improve its performance in the future.

Finally, as far as TPC figures are concerned: these are artificial, easily suborned and are won by the companies that have the most money to throw at them. There is, however, one other class of benchmark that has value, and these are the proofs of concept and benchmarks that users undertake. They have value because they directly relate to the user's requirements and then you can get a true feel for the cost of ownership.

To conclude, benchmarks have value when they are conducted for the benefit of individual users and for vendors wanting to identify weaknesses in their product. As for the marketing hype, it's like boys in the locker room claiming "mine's bigger than yours" followed by "no, it isn't - yes, it is" arguments. And it's of about as much value. I could go on with this metaphor but I think I'd better stop.

Copyright © 2006, IT-Analysis.com

Top 5 reasons to deploy VMware with Tegile

More from The Register

next story
Preview redux: Microsoft ships new Windows 10 build with 7,000 changes
Latest bleeding-edge bits borrow Action Center from Windows Phone
Google opens Inbox – email for people too thick to handle email
Print this article out and give it to someone tech-y if you get stuck
Microsoft promises Windows 10 will mean two-factor auth for all
Sneak peek at security features Redmond's baking into new OS
UNIX greybeards threaten Debian fork over systemd plan
'Veteran Unix Admins' fear desktop emphasis is betraying open source
Entity Framework goes 'code first' as Microsoft pulls visual design tool
Visual Studio database diagramming's out the window
Google+ goes TITSUP. But WHO knew? How long? Anyone ... Hello ...
Wobbly Gmail, Contacts, Calendar on the other hand ...
DEATH by PowerPoint: Microsoft warns of 0-day attack hidden in slides
Might put out patch in update, might chuck it out sooner
prev story

Whitepapers

Choosing cloud Backup services
Demystify how you can address your data protection needs in your small- to medium-sized business and select the best online backup service to meet your needs.
Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.