The Benchmark's Bunk – BAPCo a ‘Front’ for Intel

Or so rival claims

The two most widely used industry CPU benchmarks are effectively bunk, as they are under Intel's thumb, a rival maker claims.

Speaking at last week's Platform Conference held in San Jose, Randall Kennedy, the Director of Research for Competitive Systems Analysis, stated that BAPCo, the organization responsible for the most popular application level benchmark, SysMark 2000, was simply a "front" for the Satan Clara chip maker.

"It's my understanding that all of the other companies listed on BAPCo’s website have effectively just fallen by the wayside" in guiding the benchmark effort, he said.

Competitive Systems Analysis, Inc., also called "CSA Research," has developed a competitive benchmark suite marketed under the name Benchmark Studio.

According to Kennedy, Benchmark Studio was ironically borne from work he conducted under contract with Intel to demonstrate the superiority of the Pentium 4.

A demo application that preceded Benchmark Studio was used in a public presentation by Intel’s Pat Gelsinger last February, he said. Unfortunately these efforts eventually showed that dual-processing Pentium III systems, even when running at much slower clock speeds, easily outperformed even the fastest P4.

These results eventually led to a nasty falling out with the chip giant, according to Kennedy.

The current iteration of Benchmark Studio, version 1.0, shows a 733 MHz dual PIII system performing about twice as fast as a 1.5 GHz Pentium 4. Additionally, the benchmark also indicates that a 1.2 GHz AMD Athlon DDR system clobbered the same 1.5 GHz P4 system by about 15 per cent. The slowest system Kennedy discussed was one based on the 1 GHz Pentium III which was about 30 per cent slower than the P4.

Besides BAPCo's SysMark 2000, the second benchmark Kennedy singled out for being a Chipzilla chew toy was Ziff-Davis' Winstone suite. "Basically the benchmarks out there today say what Intel wants said" about computing performance, Kennedy claimed.

He explained that Benchmark Studio differs from other mainstream benchmarks by executing common tasks in the background. Built around out-of-process ActiveX COM objects which stress multitasking performance, Benchmark Studio gives a more realistic idea of system throughput, Kennedy claims.

Although Intel wants customers to believe that computing performance is like a stripped-down drag racer going from point A to point B, in real life computing is more like an SUV "running down the road with a load of gear in the back, a canoe on top and two screaming kids in the back shouting 'are we there yet?'" Kennedy stated.

You can take Benchmark Studio out for a spin by visiting CSA's website. Currently it's a free download.

And you can check out EETimes' take on the subject here. ®

Every benchmark tells a different story

Benchmarks - Itanic 32bit emulation is 'unusable'
P4 - Tom's Hardware has another rethink
P4: first Sandra 2001 benchmarks
Transmeta talks benchmarks...and 10 hour charges
Transmeta speed debate - damned lies and benchmarks?