Egad! Could Samsung be CHEATING in Galaxy benchmark tests?

What's this lipstick-marked 'BenchmarkBooster' we found in your system file, Sammy?

SANS - Survey on application security programs

Samsung has reportedly been cheating in benchmark tests, artificially boosting the scores of its latest and greatest system-on-chip, the Exynos 5 Octa, on those performance-ranking number generators so beloved by reviewers and product evaluators.

"Oh hell Samsung, shame on you!" wrote a Beyond3D forum member in a posting on one of that site's forums last month.

The poster, a Luxembourger who goes by the handle Nebuchadnezzar, had been testing a Samsung Exynos 5 Octa when he discovered that although he thought he was running the chip's GPU at 532MHz, it only hit that clock speed on two benchmarks he used for testing: AnTuTu and GLBenchmark. For all other apps, it ran at 480MHz – a much better speed for battery-life testing.

The Exynos 5 Octa, by the way, is so named because it has four high-performance ARM Cortex-A15 cores and four low-power ARM Cortex-A7 cores, all baked into a single 28-nanometer die. It comes in two versions: the 5410, which contains an Imagination Technologies PowerVR SGX544MP3 GPU, and the 5420, which uses an ARM Mali-T628 MP6 GPU. Nebuchadnezzar was testing a 5410.

Anand Lal Shimpi and Brian Klug over at the ever-interesting deep-tech site AnandTech were tipped to Nebuchadnezzar's discovery, and since they are both proud owners of the international version of the Samsung Galaxy S 4 powered by an Exynos 5410, they decided to see if they could replicate his findings.

They could – and with a few additions and clarifications. For example, the GLBenchmark v.2.5.1 did indeed run at 532MHz, but its latest v.2.7.0 incarnation – GLBenchmark having been subsumed into GFXBench along with DXBenchmark – was throttled to 480MHz.

Samsung hasn't published megahertzage for its GPU, but Shimpi and Klug said that their sources tell them it runs at 480MHz – which in fact is what they discovered its clock rate to be when running any games, "even the most demanding titles." But when running GLBenchmark 2.5.1, AnTuTu, or Quadrant – benchmarks that reviewers and product testers might naturally use to rate a products – they ran at 532MHz.

Although Nebuchadnezzar had only reported on GPU behavior, Shimpi and Klug checked out what the CPU was doing when running GLBenchmark v.2.5.1 and GFXBench v.2.7.0. To their surprise, they discovered that when running v.2.5.1, the four powerhouse Cortex-A15 cores were pinned at their top speed of 1.2GHz no matter what load the benchmark put upon them. When running v.2.7.0, however, the Exynos 5 Octa switched over to its less-powerful Cortex-A7 cores.

"A quick check across AnTuTu, Linpack, Benchmark Pi, and Quadrant reveals the same behavior," they write. The CPUs were gunned to their highest possible power capabilities when the benchmarks were running.

Digging into the Galaxy S 4's operating system support files, they came upon one with the name TwDVFSApp.apk, and since DVFS is short for dynamic voltage and frequency scaling (and, The Reg opines, "Tw" might be shorthand for "tweaking"), they opened it up in a hex editor and – behold! – in it were a list of what appeared for all the world to be a series of strings that allowed for top performance for some apps and not others, and a group-identification string with a rather incriminating name.

"The string 'BenchmarkBooster' is a particularly telling one," they write.

The gun may not be belching great clouds of damning smoke, but there's more than a mere wisp emanating from its barrel. As the AnandTech duo put it, "This seems to be purely an optimization to produce repeatable (and high) results in CPU tests, and deliver the highest possible GPU performance benchmarks."

If Nebuchadnezzar, Shimpi, and Klug are correct in their testing and analysis – and we have no reason to believe that they're not – there's only one possible conclusion: Samsung is cheating. And if they're cheating, there's a fair chance that others are, as well. But Samsung got caught.

Your Reg reporter has been around the technology-evaluation block enough times to remember – as Shimpi and Klug discuss in the conclusion to their article – when benchmark manipulation was rampant in the PC industry. As the director of a product-testing lab in the 90s, such cheating was the bane of my 9-to-5 existence.

Well, here it comes again – both fairly blatantly, as in Samsung's CPU and GPU rigging, or in a more slippery fashion, as in the use of an Intel-specific compiler in a test that enabled Chipzilla's Clover Trail+ platform to outperform ARM processors.

Today is different from the 90s, however. In those far-away days, speeds and specs were important even to consumers, while in today's shiny-shiny world, the average fandroid or fanboi couldn't care less about gigatexels or TMUs. "Experience" rules the checkbooks of the marketplace, not benchmark scores.

But deceit is still not right. Having experienced Samsung's chicanery directly, let's give our cheater-finders the last word on this sorry state of affairs.

Shimpi and Klug: "Just because we’ve seen things like this happen in the past however doesn't mean they should happen now."

Nebuchadnezzar: "Oh hell Samsung, shame on you!" ®


Here's a li'l fairness v. bias test we suggest you might find personally illuminating. Read the story above one more time, except each time you see the word "Samsung", substitute "Apple".

Then ask yourself: "Is my response any different?"

SANS - Survey on application security programs

More from The Register

next story
WTF happened to Pac-Man?
In his thirties and still afraid of ghosts
Reg man builds smart home rig, gains SUPREME CONTROL of DOMAIN – Pics
LightwaveRF and Arduino: Bright ideas for dim DIYers
Leaked pics show EMBIGGENED iPhone 6 screen
Fat-fingered fanbois rejoice over Chinternet snaps
Microsoft signs Motorola to Android patent pact – no, not THAT Motorola
The part that Google never got will play ball with Redmond
Apple patent LOCKS drivers out of their OWN PHONES
I'm sorry Dave, I'm afraid I can't let you text that
Happy 25th birthday, Game Boy!
Monochrome handset ushered in modern mobile gaming era
Slip your finger in this ring and unlock your backdoor, phone, etc
Take a look at this new NFC jewellery – why, what were you thinking of?
Rounded corners? Pah! Amazon's '3D phone has eye-tracking tech'
Now THAT'S what we call a proper new feature
Zucker punched: Google gobbles Facebook-wooed Titan Aerospace
Up, up and away in my beautiful balloon flying broadband-bot
prev story


Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
SANS - Survey on application security programs
In this whitepaper learn about the state of application security programs and practices of 488 surveyed respondents, and discover how mature and effective these programs are.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.