Feeds

Transmeta speed debate – damned lies and benchmarks?

Information in this story may be 30 per cent inaccurate...

  • alert
  • submit to reddit

Bridging the IT gap between rising business demands and ageing tools

Transmeta notebooks have been available for a week now, and although it's been a fascinating week for benchmark mavens, the casual punter hasn't been much enlightened.

Why? Well three serious IT publications have run benchmarks for Sony's new Crusoe laptop in the past week, published them... and disavowed the results immediately.

Sorta, kinda.

The problem is that Transmeta's new chip confounds the traditional one-pass benchmarks that their propeller head authors have spent years perfecting. Since the Transmeta architecture allows for dynamic and smart execution of code in software, the first pass of any test sample is guaranteed to be if not crap, then not quite optimal. With the instruction set and performance heuristics in software - a compromise that allows the chip itself to throttle back its frequency and voltage - judging a Crusoe on first-pass benchmarks is like judging a marrow competition based on the seeds. It isn't going to give you the full picture.

So unsurprisingly, the three benchmarks we've looked at pronounce the Crusoe a dog. To the magazines' credit (or at least most of them), that's qualified by a note that their benchmark shouldn't be taken seriously.

PC Magazine (US) stated: "The PictureBook is a bit anemic [sic], even when set to maximum performance," but then qualifies this by pointing out "the Winstone benchmark test does not repeat tasks, which minimizes opportunities for the Code Morphing technology to have an effect."

The British publication IT Week splashed with a screaming headline 'CRUSOE FAILS TO DELIVER ON PROMISE' using similar benchmarks - but then torpedoed its own assertion with a comment from Microprocessor Report's Peter Glaskowsky pointing out that... um... a 30 per cent speed improvement could be gained from running the benchmark again.

Reviewing the Transmeta picture book, PC World (US) failed to point this out at all, noting that "its score of 66 is shockingly low". Why? They don't even tell us.

Transmeta insists "Our best benchmark is the fact that major notebook manufacturers are seeing the benefits of our chip and shipping systems based on Crusoe."

Now clearly the labs guys have a duty to make like-for-like comparisons with what they receive. These are smart folk (we claim some insight here for all three publications) with a professional duty to provide empirical evidence for their readers. But plainly, sticking to a benchmark written for one architecture is rendered meaningless another - particularly when it's as radical as Crusoe. Both IT Week and PC Mag acknowledged this, in their own way, but still couldn't quite resist the urge to scratch an itch. "These benchmarks are probably worthless," they're saying, "but here they are anyway."

In the week that The Nation published an account of how the New York Times' persecuted of Los Alamos scientist Wen Ho Lee (an innocent man branded a spy) and compounded it with a now notorious non-apology, we were reminded of how the most august publications can continue to propound information they acknowledge is worthless. And that everyone, but themselves, knows to be worthless too. No, Crusoe benchmarks don't morally equate to the Lee case, but you wonder how long the public prints can show the same ambivalence, superiority and essential stupidity as the NYT, without conceding a little tweak to reality.

And how Transmeta thinks benchmarks really should be, we hope to tell you tomorrow. ®

Build a business case: developing custom apps

More from The Register

next story
BBC goes offline in MASSIVE COCKUP: Stephen Fry partly muzzled
Auntie tight-lipped as major outage rolls on
Nadella: Apps must run on ALL WINDOWS – on PCs, slabs and mobes
Phone egg, meet desktop chicken - your mother
ITC: Seagate and LSI can infringe Realtek patents because Realtek isn't in the US
Land of the (get off scot) free, when it's a foreign owner
HP and Microsoft prove it again: Big Business Doesn't Create Jobs
SMEs get lip service - what they need is dinner at the Club
Samsung threatens to cut ties with supplier over child labour allegations
Vows to uphold 'zero tolerance' policy on underage workers
Dude, you're getting a Dell – with BITCOIN: IT giant slurps cryptocash
1. Buy PC with Bitcoin. 2. Mine more coins. 3. Goto step 1
There's NOTHING on TV in Europe – American video DOMINATES
Even France's mega subsidies don't stop US content onslaught
You! Pirate! Stop pirating, or we shall admonish you politely. Repeatedly, if necessary
And we shall go about telling people you smell. No, not really
prev story

Whitepapers

Seven Steps to Software Security
Seven practical steps you can begin to take today to secure your applications and prevent the damages a successful cyber-attack can cause.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.