Feeds

Egad! Could Samsung be CHEATING in Galaxy benchmark tests?

What's this lipstick-marked 'BenchmarkBooster' we found in your system file, Sammy?

High performance access to file storage

Samsung has reportedly been cheating in benchmark tests, artificially boosting the scores of its latest and greatest system-on-chip, the Exynos 5 Octa, on those performance-ranking number generators so beloved by reviewers and product evaluators.

"Oh hell Samsung, shame on you!" wrote a Beyond3D forum member in a posting on one of that site's forums last month.

The poster, a Luxembourger who goes by the handle Nebuchadnezzar, had been testing a Samsung Exynos 5 Octa when he discovered that although he thought he was running the chip's GPU at 532MHz, it only hit that clock speed on two benchmarks he used for testing: AnTuTu and GLBenchmark. For all other apps, it ran at 480MHz – a much better speed for battery-life testing.

The Exynos 5 Octa, by the way, is so named because it has four high-performance ARM Cortex-A15 cores and four low-power ARM Cortex-A7 cores, all baked into a single 28-nanometer die. It comes in two versions: the 5410, which contains an Imagination Technologies PowerVR SGX544MP3 GPU, and the 5420, which uses an ARM Mali-T628 MP6 GPU. Nebuchadnezzar was testing a 5410.

Anand Lal Shimpi and Brian Klug over at the ever-interesting deep-tech site AnandTech were tipped to Nebuchadnezzar's discovery, and since they are both proud owners of the international version of the Samsung Galaxy S 4 powered by an Exynos 5410, they decided to see if they could replicate his findings.

They could – and with a few additions and clarifications. For example, the GLBenchmark v.2.5.1 did indeed run at 532MHz, but its latest v.2.7.0 incarnation – GLBenchmark having been subsumed into GFXBench along with DXBenchmark – was throttled to 480MHz.

Samsung hasn't published megahertzage for its GPU, but Shimpi and Klug said that their sources tell them it runs at 480MHz – which in fact is what they discovered its clock rate to be when running any games, "even the most demanding titles." But when running GLBenchmark 2.5.1, AnTuTu, or Quadrant – benchmarks that reviewers and product testers might naturally use to rate a products – they ran at 532MHz.

Although Nebuchadnezzar had only reported on GPU behavior, Shimpi and Klug checked out what the CPU was doing when running GLBenchmark v.2.5.1 and GFXBench v.2.7.0. To their surprise, they discovered that when running v.2.5.1, the four powerhouse Cortex-A15 cores were pinned at their top speed of 1.2GHz no matter what load the benchmark put upon them. When running v.2.7.0, however, the Exynos 5 Octa switched over to its less-powerful Cortex-A7 cores.

"A quick check across AnTuTu, Linpack, Benchmark Pi, and Quadrant reveals the same behavior," they write. The CPUs were gunned to their highest possible power capabilities when the benchmarks were running.

Digging into the Galaxy S 4's operating system support files, they came upon one with the name TwDVFSApp.apk, and since DVFS is short for dynamic voltage and frequency scaling (and, The Reg opines, "Tw" might be shorthand for "tweaking"), they opened it up in a hex editor and – behold! – in it were a list of what appeared for all the world to be a series of strings that allowed for top performance for some apps and not others, and a group-identification string with a rather incriminating name.

"The string 'BenchmarkBooster' is a particularly telling one," they write.

The gun may not be belching great clouds of damning smoke, but there's more than a mere wisp emanating from its barrel. As the AnandTech duo put it, "This seems to be purely an optimization to produce repeatable (and high) results in CPU tests, and deliver the highest possible GPU performance benchmarks."

If Nebuchadnezzar, Shimpi, and Klug are correct in their testing and analysis – and we have no reason to believe that they're not – there's only one possible conclusion: Samsung is cheating. And if they're cheating, there's a fair chance that others are, as well. But Samsung got caught.

Your Reg reporter has been around the technology-evaluation block enough times to remember – as Shimpi and Klug discuss in the conclusion to their article – when benchmark manipulation was rampant in the PC industry. As the director of a product-testing lab in the 90s, such cheating was the bane of my 9-to-5 existence.

Well, here it comes again – both fairly blatantly, as in Samsung's CPU and GPU rigging, or in a more slippery fashion, as in the use of an Intel-specific compiler in a test that enabled Chipzilla's Clover Trail+ platform to outperform ARM processors.

Today is different from the 90s, however. In those far-away days, speeds and specs were important even to consumers, while in today's shiny-shiny world, the average fandroid or fanboi couldn't care less about gigatexels or TMUs. "Experience" rules the checkbooks of the marketplace, not benchmark scores.

But deceit is still not right. Having experienced Samsung's chicanery directly, let's give our cheater-finders the last word on this sorry state of affairs.

Shimpi and Klug: "Just because we’ve seen things like this happen in the past however doesn't mean they should happen now."

Nebuchadnezzar: "Oh hell Samsung, shame on you!" ®

Bootnote

Here's a li'l fairness v. bias test we suggest you might find personally illuminating. Read the story above one more time, except each time you see the word "Samsung", substitute "Apple".

Then ask yourself: "Is my response any different?"

High performance access to file storage

More from The Register

next story
Report: Apple seeking to raise iPhone 6 price by a HUNDRED BUCKS
'Well, that 5c experiment didn't go so well – let's try the other direction'
Video games make you NASTY AND VIOLENT
Especially if you are bad at them and keep losing
Microsoft lobs pre-release Windows Phone 8.1 at devs who dare
App makers can load it before anyone else, but if they do they're stuck with it
Zucker punched: Google gobbles Facebook-wooed Titan Aerospace
Up, up and away in my beautiful balloon flying broadband-bot
Nvidia gamers hit trifecta with driver, optimizer, and mobile upgrades
Li'l Shield moves up to Android 4.4.2 KitKat, GameStream comes to notebooks
Gimme a high S5: Samsung Galaxy S5 puts substance over style
Biometrics and kid-friendly mode in back-to-basics blockbuster
AMD unveils Godzilla's graphics card – 'the world's fastest, period'
The Radeon R9 295X2: Water-cooled, 5,632 stream processors, 11.5TFLOPS
Sony battery recall as VAIO goes out with a bang, not a whimper
The perils of having Panasonic as a partner
NORKS' own smartmobe pegged as Chinese landfill Android
Fake kit in the hermit kingdom? That's just Kim Jong-un-believable!
prev story

Whitepapers

Mainstay ROI - Does application security pay?
In this whitepaper learn how you and your enterprise might benefit from better software security.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Mobile application security study
Download this report to see the alarming realities regarding the sheer number of applications vulnerable to attack, as well as the most common and easily addressable vulnerability errors.