Feeds

A case for software benchmarking

Wheat and chaff

  • alert
  • submit to reddit

Secure remote control for conventional and virtual desktops

Comment I do not normally hold much truck with benchmarking, at least for such public tests as TPC-C (the standard for transaction processing) and TPC-H (for data warehousing). These are typically marketing figures – the tests are artificial, limited, and vendors with big bucks can throw enough money at the tests to ensure that they do well. Thus, for example, I did not publicise IBM's TPC-C figure for the latest release of DB2 last year, despite the fact that it was more than twice as good as any previous figure. No, to my mind the only useful benchmarks are those that are performed specifically for individual customers with their data and their workload. But these, of course, are expensive to run.

This, at least, was my thought until recently. However, I have recently been discussing benchmarking with one of the founders of benchmarking, then at the University of Wisconsin, back in the 80s. The point was made to me that back in the early days, benchmarks really did have a useful purpose. For example, it turned out that some of the databases that were tested way back then couldn't do such things as three-way joins: the optimiser just thrashed about and produced nothing. The contention therefore, is that benchmarking can be useful where there is only uncertainty and doubt and a lot of marketing hype about performance – in other words, in the early days of a technology.

Is there such a technology? Yes, there is. Vendors (recent examples include Software AG, Sunopsis and Informatica) are pouring into the EII (enterprise information integration) space to offer federated query capability. That is, supporting real-time queries against heterogeneous data sources.

To date, much of the focus in this market has been on how you set up and define these queries and only a relatively few vendors (notably Composite Software, Callixa and IBM) have been putting much emphasis on performance. But performance is a big issue. Indeed, if you don't get good performance across a spectrum of queries, then it is not worth investing in the technology in the first place. In fact, I know of one major company (which will remain nameless) that, to its credit, has not introduced a product into this space precisely because it cannot get it to perform.

I am inclined to think that some benchmarking, perhaps based on a distributed TPC-H model, would be of benefit in this instance; at least in the short term. Vendors can currently wave performance figures around willy-nilly without fear of contradiction, and it is pretty much certain that some federated query projects will fail because the selected solution fails. While benchmarks are only a snapshot, in this case they may serve as an initial method for sorting the wheat from the chaff (if only because the chaff will not participate). To that extent I am in favour of them.

© IT-Analysis.com

Related stories

Sun could quell database hunger with Unify buy
IBM benchmark leaves server rivals breathless
IBM and HP take phony benchmark war up several notches

The essential guide to IT transformation

More from The Register

next story
Apple promises to lift Curse of the Drained iPhone 5 Battery
Have you tried turning it off and...? Never mind, here's a replacement
China: You, Microsoft. Office-Windows 'compatibility'. You have 20 days to explain
Told to cough up more details as antitrust probe goes deeper
BBC: We're going to slip CODING into kids' TV
Pureed-carrot-in-ice cream C++ surprise
Linux turns 23 and Linus Torvalds celebrates as only he can
No, not with swearing, but by controlling the release cycle
Scratched PC-dispatch patch patched, hatched in batch rematch
Windows security update fixed after triggering blue screens (and screams) of death
This is how I set about making a fortune with my own startup
Would you leave your well-paid job to chase your dream?
prev story

Whitepapers

Endpoint data privacy in the cloud is easier than you think
Innovations in encryption and storage resolve issues of data privacy and key requirements for companies to look for in a solution.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Advanced data protection for your virtualized environments
Find a natural fit for optimizing protection for the often resource-constrained data protection process found in virtual environments.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.