Feeds

Benchmarks, bunkum and baloney

Performance figure hype

  • alert
  • submit to reddit

Securing Web Applications Made Simple and Scalable

Comment There are three types of claims made by vendors with respect to benchmarks: pure performance figures, competitive performance comparison, and TPC figures. In my view, none of these has marketing value, though the second of these potentially has some utility for the supplier. Let me explain.

A good example of the over-hyping of pure performance figures can be found in the event streaming market. More or less every vendor in the market puts out figures about its event handling capability, ranging from tens, through hundreds of thousands of transactions per second to claims in the millions of transactions per second.

Now, what don't these claims encompass? First, a definition of what an event consists of: if my event is 10 bytes and yours is 20 bytes then, all other things being equal (which they aren't), I am going to get better results than you are. Second, even if we agree on an event size, then this takes no account of the fact that some vendors can process deltas and some can't. That is, the ability to process only changes to the data rather than having to process events in their entirety.

Thirdly, there is the processor issue. Unless everybody is running on the same hardware the relative merits of one performance figure versus another are opaque to say the least. Moreover, if you are going to have to spend hundreds of thousands of dollars for an implementation and you need a two CPU system as opposed to a single CPU system (or even 16 instead of eight) then how much difference is the extra cost of hardware going to make?

And finally, how much throughput do you really need? In equity trading in New York 150,000 transactions per sec is roughly the current rate. On the other hand, for mobile call rate look-up less than a tenth of this figure is more than enough. While rates are likely to increase, the performance of all the vendors is more than enough or, where it is not, you will find that these vendors do not target the very highest throughput requirements such as algorithmic trading.

Anyway, enough of pure performance and on to competitive performance benchmarks. An example of this that recently came up was an ETL vendor that claimed to have the fastest throughput on the market. Apart from the fact that it had done a performance comparison with just one other vendor (and that was not Ab Initio) as opposed to the 40 or so in the market, it, too, made a big thing of the fact that its figures were for a single CPU system as opposed to a two CPU system—see my remarks above.

However, in the ETL space it isn't just about throughput, there are also issues depending on the transformations you are doing, for example, and this vendor did a number of different tests, in some of which it came off better than others.

Naturally, it published only the ones most favourable to it (incidentally, I am not naming the company here since, in my experience, every vendor does the same thing), which is why the figures are nothing more than marketing puff. However, where there was real value in doing these comparisons was in allowing the supplier to discover where it did not do so well, so that it could work on this in the labs and improve its performance in the future.

Finally, as far as TPC figures are concerned: these are artificial, easily suborned and are won by the companies that have the most money to throw at them. There is, however, one other class of benchmark that has value, and these are the proofs of concept and benchmarks that users undertake. They have value because they directly relate to the user's requirements and then you can get a true feel for the cost of ownership.

To conclude, benchmarks have value when they are conducted for the benefit of individual users and for vendors wanting to identify weaknesses in their product. As for the marketing hype, it's like boys in the locker room claiming "mine's bigger than yours" followed by "no, it isn't - yes, it is" arguments. And it's of about as much value. I could go on with this metaphor but I think I'd better stop.

Copyright © 2006, IT-Analysis.com

Bridging the IT gap between rising business demands and ageing tools

More from The Register

next story
NO MORE ALL CAPS and other pleasures of Visual Studio 14
Unpicking a packed preview that breaks down ASP.NET
DARPA-derived secure microkernel goes open source tomorrow
Hacker-repelling, drone-protecting code will soon be yours to tweak as you see fit
Cheer up, Nokia fans. It can start making mobes again in 18 months
The real winner of the Nokia sale is *drumroll* ... Nokia
Put down that Oracle database patch: It could cost $23,000 per CPU
On-by-default INMEMORY tech a boon for developers ... as long as they can afford it
Google shows off new Chrome OS look
Athena springs full-grown from Chromium project's head
Apple: We'll unleash OS X Yosemite beta on the MASSES on 24 July
Starting today, regular fanbois will be guinea pigs, it tells Reg
HIDDEN packet sniffer spy tech in MILLIONS of iPhones, iPads – expert
Don't panic though – Apple's backdoor is not wide open to all, guru tells us
prev story

Whitepapers

Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Seven Steps to Software Security
Seven practical steps you can begin to take today to secure your applications and prevent the damages a successful cyber-attack can cause.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.