Egad! Could Samsung be cheating in Galaxy benchmark tests?
What's this lipstick-marked 'BenchmarkBooster' we found in your system file, Sammy?
Samsung has reportedly been cheating in benchmark tests, artificially boosting the scores of its latest and greatest system-on-chip, the Exynos 5 Octa, on those performance-ranking number generators so beloved by reviewers and product evaluators.
The poster, a Luxembourger who goes by the handle Nebuchadnezzar, had been testing a Samsung Exynos 5 Octa when he discovered that although he thought he was running the chip's GPU at 532MHz, it only hit that clock speed on two benchmarks he used for testing: AnTuTu and GLBenchmark. For all other apps, it ran at 480MHz – a much better speed for battery-life testing.
The Exynos 5 Octa, by the way, is so named because it has four high-performance ARM Cortex-A15 cores and four low-power ARM Cortex-A7 cores, all baked into a single 28-nanometer die. It comes in two versions: the 5410, which contains an Imagination Technologies PowerVR SGX544MP3 GPU, and the 5420, which uses an ARM Mali-T628 MP6 GPU. Nebuchadnezzar was testing a 5410.
Anand Lal Shimpi and Brian Klug over at the ever-interesting deep-tech site AnandTech were tipped to Nebuchadnezzar's discovery, and since they are both proud owners of the international version of the Samsung Galaxy S 4 powered by an Exynos 5410, they decided to see if they could replicate his findings.
They could – and with a few additions and clarifications. For example, the GLBenchmark v.2.5.1 did indeed run at 532MHz, but its latest v.2.7.0 incarnation – GLBenchmark having been subsumed into GFXBench along with DXBenchmark – was throttled to 480MHz.
Samsung hasn't published megahertzage for its GPU, but Shimpi and Klug said that their sources tell them it runs at 480MHz – which in fact is what they discovered its clock rate to be when running any games, "even the most demanding titles." But when running GLBenchmark 2.5.1, AnTuTu, or Quadrant – benchmarks that reviewers and product testers might naturally use to rate a products – they ran at 532MHz.
Although Nebuchadnezzar had only reported on GPU behavior, Shimpi and Klug checked out what the CPU was doing when running GLBenchmark v.2.5.1 and GFXBench v.2.7.0. To their surprise, they discovered that when running v.2.5.1, the four powerhouse Cortex-A15 cores were pinned at their top speed of 1.2GHz no matter what load the benchmark put upon them. When running v.2.7.0, however, the Exynos 5 Octa switched over to its less-powerful Cortex-A7 cores.
"A quick check across AnTuTu, Linpack, Benchmark Pi, and Quadrant reveals the same behavior," they write. The CPUs were gunned to their highest possible power capabilities when the benchmarks were running.
Digging into the Galaxy S 4's operating system support files, they came upon one with the name TwDVFSApp.apk, and since DVFS is short for dynamic voltage and frequency scaling (and, The Reg opines, "Tw" might be shorthand for "tweaking"), they opened it up in a hex editor and – behold! – in it were a list of what appeared for all the world to be a series of strings that allowed for top performance for some apps and not others, and a group-identification string with a rather incriminating name.
"The string 'BenchmarkBooster' is a particularly telling one," they write.
The gun may not be belching great clouds of damning smoke, but there's more than a mere wisp emanating from its barrel. As the AnandTech duo put it, "This seems to be purely an optimization to produce repeatable (and high) results in CPU tests, and deliver the highest possible GPU performance benchmarks."
If Nebuchadnezzar, Shimpi, and Klug are correct in their testing and analysis – and we have no reason to believe that they're not – there's only one possible conclusion: Samsung is cheating. And if they're cheating, there's a fair chance that others are, as well. But Samsung got caught.
Your Reg reporter has been around the technology-evaluation block enough times to remember – as Shimpi and Klug discuss in the conclusion to their article – when benchmark manipulation was rampant in the PC industry. As the director of a product-testing lab in the 90s, such cheating was the bane of my 9-to-5 existence.
Well, here it comes again – both fairly blatantly, as in Samsung's CPU and GPU rigging, or in a more slippery fashion, as in the use of an Intel-specific compiler in a test that enabled Chipzilla's Clover Trail+ platform to outperform ARM processors.
Today is different from the 90s, however. In those far-away days, speeds and specs were important even to consumers, while in today's shiny-shiny world, the average fandroid or fanboi couldn't care less about gigatexels or TMUs. "Experience" rules the checkbooks of the marketplace, not benchmark scores.
But deceit is still not right. Having experienced Samsung's chicanery directly, let's give our cheater-finders the last word on this sorry state of affairs.
Shimpi and Klug: "Just because we’ve seen things like this happen in the past however doesn't mean they should happen now."
Nebuchadnezzar: "Oh hell Samsung, shame on you!" ®
Here's a li'l fairness v. bias test we suggest you might find personally illuminating. Read the story above one more time, except each time you see the word "Samsung", substitute "Apple".
Then ask yourself: "Is my response any different?"