Feeds

Benchmarks are $%#&@!!

Secrets and solutions from a reformed benchmarketer

  • alert
  • submit to reddit

Boost IT visibility and business value

At SC11 I ran into Henry Newman, CEO of HPC consulting firm Instrumental Inc. After exchanging the usual pleasantries and deeply offensive personal insults, we got to talking about some of the recently released benchmark results – and how irrelevant most of them are to the real world.

In the course of the conversation, Henry told me that he was once a “slimy benchmarker". He told me about some of the tricks that vendor benchmarkers use to make sure their systems shine brightly in customer bake-offs. Basically, if a particular trick (special tuning, hardware, etc.) “isn’t clearly and explicitly banned by the customer, then it’s fair game.”

I get the feeling that back in the day, Henry was a particularly savage and merciless benchmarker with a pretty good win/loss record. But now that he’s on the customer side of the industry, Henry has a different point of view, and wants to become part of the solution to the benchmarking problem (which he helped promulgate, damn it.)

Toward that end, he’s spent a lot of time working on a project with DARPA to develop a set of benchmarks that get to the heart of what customers really need to know: how well systems scale from small to large.

In the webcast we talk about all of the above topics, drill down into specific scalability benchmarks, and give Henry a chance to confess his benchmarking sins. Give it a listen…

Watch Video

The essential guide to IT transformation

More from The Register

next story
The Return of BSOD: Does ANYONE trust Microsoft patches?
Sysadmins, you're either fighting fires or seen as incompetents now
Microsoft: Azure isn't ready for biz-critical apps … yet
Microsoft will move its own IT to the cloud to avoid $200m server bill
Oracle reveals 32-core, 10 BEEELLION-transistor SPARC M7
New chip scales to 1024 cores, 8192 threads 64 TB RAM, at speeds over 3.6GHz
US regulators OK sale of IBM's x86 server biz to Lenovo
Now all that remains is for gov't offices to ban the boxes
Flash could be CHEAPER than SAS DISK? Come off it, NetApp
Stats analysis reckons we'll hit that point in just three years
Object storage bods Exablox: RAID is dead, baby. RAID is dead
Bring your own disks to its object appliances
Nimble's latest mutants GORGE themselves on unlucky forerunners
Crossing Sandy Bridges without stopping for breath
prev story

Whitepapers

5 things you didn’t know about cloud backup
IT departments are embracing cloud backup, but there’s a lot you need to know before choosing a service provider. Learn all the critical things you need to know.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.
Rethinking backup and recovery in the modern data center
Combining intelligence, operational analytics, and automation to enable efficient, data-driven IT organizations using the HP ABR approach.
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.