Feeds

Analyst: Tests showing Intel smartphones beating ARM were rigged

Bad benchmark gave bad results

Internet Security Threat Report 2014

So does Intel's Atom-based Clover Trail+ platform really outperform ARM processors in smartphone benchmarks? Not so fast, says one analyst.

In June, The Reg reported analyst firm ABI Research's claim that it had pitted a Lenovo K900 smartphone based on Intel's Atom Z2580 processor against a brace of devices build around ARM system-on-chip (SoC) components and found that not only did the Intel part perform better, but it also drew less power.

But Jim McGregor of analyst firm Tirias Research smelled something fishy, and after investigating, he now says the surprise showing by Intel had less to do with the chip itself as it did with inconsistencies in the AnTuTu benchmark used to conduct the tests.

McGregor's first clue was that different versions of the benchmark produced wildly different results.

"Going from the 2.9.3 version of the AnTuTu benchmark to the 3.3 version, the overall AnTuTu scores increased 122 percent, and the RAM score increased 292 percent for the Intel processor, while the scores for the Samsung processor increased only 59 percent and 53 percent, respectively," McGregor wrote in a blog post at EE Times. "This was just from a change in the benchmark test, not the processors."

The discrepancy, it seems, stems from the fact that the binary executables of AnTuTu versions 2.9.4 and later are built for Intel chips using a different compiler than that used for other processor architectures.

Versions of AnTuTu for ARM chips are built using the open source GCC compiler. But beginning with version 2.9.4, AnTuTu for Intel is built using ICC, a proprietary optimizing compiler designed by Intel itself.

Working with AnTuTu and technology consulting firm BDTI, McGregor determined that the version of the benchmark built with ICC was allowing Intel processors to skip some of the instructions that make up the RAM performance test, leading to artificially inflated results.

AnTuTu released version 3.3.2 of the benchmark on Wednesday to address the problem, and according to McGregor, it negates Intel's artificial advantage. Intel's CPU and Overall scores are now about 20 per cent lower than they were with the previous build, and the RAM score is around 50 per cent lower.

"As a result of the revised scores, the AnTuTu benchmark is no longer an outlier from the other benchmarks and paints a competitive picture similar to the other benchmarks, with the Samsung Exynos 5 Octa processor outperforming the Intel Atom Z2580 processor," McGregor wrote on Friday.

He added that it's still hard to be sure if even those results are valid, because AnTuTu has not explained what changes were made to the Intel version of the benchmark, although the new version is still built with ICC.

What's more, he said, AnTuTu is planning to revise its testing standards in August, which will presumably result in completely different scores across all of the various platforms, due to the inherent complexities of SoC components – so get ready for another round of finger-pointing then.

Asked for comment, a spokesperson for ABI Research told El Reg, "Honestly, we feel [McGregor] totally missed the point. He is focusing on the benchmark and not the power performance we highlighted. It is too easy to pick on benchmarks."

The spokesperson said ABI Research expects to have more to say on the matter next week. ®

Choosing a cloud hosting partner with confidence

More from The Register

next story
PEAK APPLE: iOS 8 is least popular Cupertino mobile OS in all of HUMAN HISTORY
'Nerd release' finally staggers past 50 per cent adoption
Tim Cook: The classic iPod HAD to DIE, and this is WHY
Apple, er, couldn’t get the parts for HDD models
Apple spent just ONE DOLLAR beefing up the latest iPad Air 2
New iPads look a lot like the old one. There's a reason for that
Google Glassholes are UNDATEABLE – HP exec
You need an emotional connection, says touchy-feely MD... We can do that
Lawyers mobilise angry mob against Apple over alleged 2011 Macbook Pro crapness
We suffered 'random bouts of graphical distortion' - fanbois
Caterham Seven 160 review: The Raspberry Pi of motoring
Back to driving's basics with a joyously legal high
prev story

Whitepapers

Cloud and hybrid-cloud data protection for VMware
Learn how quick and easy it is to configure backups and perform restores for VMware environments.
Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Saudi Petroleum chooses Tegile storage solution
A storage solution that addresses company growth and performance for business-critical applications of caseware archive and search along with other key operational systems.
The hidden costs of self-signed SSL certificates
Exploring the true TCO for self-signed SSL certificates, including a side-by-side comparison of a self-signed architecture versus working with a third-party SSL vendor.