Feeds

Benchmarks, bunkum and baloney

Performance figure hype

  • alert
  • submit to reddit

Top three mobile application threats

Comment There are three types of claims made by vendors with respect to benchmarks: pure performance figures, competitive performance comparison, and TPC figures. In my view, none of these has marketing value, though the second of these potentially has some utility for the supplier. Let me explain.

A good example of the over-hyping of pure performance figures can be found in the event streaming market. More or less every vendor in the market puts out figures about its event handling capability, ranging from tens, through hundreds of thousands of transactions per second to claims in the millions of transactions per second.

Now, what don't these claims encompass? First, a definition of what an event consists of: if my event is 10 bytes and yours is 20 bytes then, all other things being equal (which they aren't), I am going to get better results than you are. Second, even if we agree on an event size, then this takes no account of the fact that some vendors can process deltas and some can't. That is, the ability to process only changes to the data rather than having to process events in their entirety.

Thirdly, there is the processor issue. Unless everybody is running on the same hardware the relative merits of one performance figure versus another are opaque to say the least. Moreover, if you are going to have to spend hundreds of thousands of dollars for an implementation and you need a two CPU system as opposed to a single CPU system (or even 16 instead of eight) then how much difference is the extra cost of hardware going to make?

And finally, how much throughput do you really need? In equity trading in New York 150,000 transactions per sec is roughly the current rate. On the other hand, for mobile call rate look-up less than a tenth of this figure is more than enough. While rates are likely to increase, the performance of all the vendors is more than enough or, where it is not, you will find that these vendors do not target the very highest throughput requirements such as algorithmic trading.

Anyway, enough of pure performance and on to competitive performance benchmarks. An example of this that recently came up was an ETL vendor that claimed to have the fastest throughput on the market. Apart from the fact that it had done a performance comparison with just one other vendor (and that was not Ab Initio) as opposed to the 40 or so in the market, it, too, made a big thing of the fact that its figures were for a single CPU system as opposed to a two CPU system—see my remarks above.

However, in the ETL space it isn't just about throughput, there are also issues depending on the transformations you are doing, for example, and this vendor did a number of different tests, in some of which it came off better than others.

Naturally, it published only the ones most favourable to it (incidentally, I am not naming the company here since, in my experience, every vendor does the same thing), which is why the figures are nothing more than marketing puff. However, where there was real value in doing these comparisons was in allowing the supplier to discover where it did not do so well, so that it could work on this in the labs and improve its performance in the future.

Finally, as far as TPC figures are concerned: these are artificial, easily suborned and are won by the companies that have the most money to throw at them. There is, however, one other class of benchmark that has value, and these are the proofs of concept and benchmarks that users undertake. They have value because they directly relate to the user's requirements and then you can get a true feel for the cost of ownership.

To conclude, benchmarks have value when they are conducted for the benefit of individual users and for vendors wanting to identify weaknesses in their product. As for the marketing hype, it's like boys in the locker room claiming "mine's bigger than yours" followed by "no, it isn't - yes, it is" arguments. And it's of about as much value. I could go on with this metaphor but I think I'd better stop.

Copyright © 2006, IT-Analysis.com

Maximizing your infrastructure through virtualization

More from The Register

next story
Apple fanbois SCREAM as update BRICKS their Macbook Airs
Ragegasm spills over as firmware upgrade kills machines
HIDDEN packet sniffer spy tech in MILLIONS of iPhones, iPads – expert
Don't panic though – Apple's backdoor is not wide open to all, guru tells us
Mozilla fixes CRITICAL security holes in Firefox, urges v31 upgrade
Misc memory hazards 'could be exploited' - and guess what, one's a Javascript vuln
NO MORE ALL CAPS and other pleasures of Visual Studio 14
Unpicking a packed preview that breaks down ASP.NET
Captain Kirk sets phaser to SLAUGHTER after trying new Facebook app
William Shatner less-than-impressed by Zuck's celebrity-only app
Cheer up, Nokia fans. It can start making mobes again in 18 months
The real winner of the Nokia sale is *drumroll* ... Nokia
EU dons gloves, pokes Google's deals with Android mobe makers
El Reg cops a squint at investigatory letters
Chrome browser has been DRAINING PC batteries for YEARS
Google is only now fixing ancient, energy-sapping bug
prev story

Whitepapers

Top three mobile application threats
Prevent sensitive data leakage over insecure channels or stolen mobile devices.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
The Essential Guide to IT Transformation
ServiceNow discusses three IT transformations that can help CIO's automate IT services to transform IT and the enterprise.