Feeds

Analyst: Tests showing Intel smartphones beating ARM were rigged

Bad benchmark gave bad results

5 things you didn’t know about cloud backup

So does Intel's Atom-based Clover Trail+ platform really outperform ARM processors in smartphone benchmarks? Not so fast, says one analyst.

In June, The Reg reported analyst firm ABI Research's claim that it had pitted a Lenovo K900 smartphone based on Intel's Atom Z2580 processor against a brace of devices build around ARM system-on-chip (SoC) components and found that not only did the Intel part perform better, but it also drew less power.

But Jim McGregor of analyst firm Tirias Research smelled something fishy, and after investigating, he now says the surprise showing by Intel had less to do with the chip itself as it did with inconsistencies in the AnTuTu benchmark used to conduct the tests.

McGregor's first clue was that different versions of the benchmark produced wildly different results.

"Going from the 2.9.3 version of the AnTuTu benchmark to the 3.3 version, the overall AnTuTu scores increased 122 percent, and the RAM score increased 292 percent for the Intel processor, while the scores for the Samsung processor increased only 59 percent and 53 percent, respectively," McGregor wrote in a blog post at EE Times. "This was just from a change in the benchmark test, not the processors."

The discrepancy, it seems, stems from the fact that the binary executables of AnTuTu versions 2.9.4 and later are built for Intel chips using a different compiler than that used for other processor architectures.

Versions of AnTuTu for ARM chips are built using the open source GCC compiler. But beginning with version 2.9.4, AnTuTu for Intel is built using ICC, a proprietary optimizing compiler designed by Intel itself.

Working with AnTuTu and technology consulting firm BDTI, McGregor determined that the version of the benchmark built with ICC was allowing Intel processors to skip some of the instructions that make up the RAM performance test, leading to artificially inflated results.

AnTuTu released version 3.3.2 of the benchmark on Wednesday to address the problem, and according to McGregor, it negates Intel's artificial advantage. Intel's CPU and Overall scores are now about 20 per cent lower than they were with the previous build, and the RAM score is around 50 per cent lower.

"As a result of the revised scores, the AnTuTu benchmark is no longer an outlier from the other benchmarks and paints a competitive picture similar to the other benchmarks, with the Samsung Exynos 5 Octa processor outperforming the Intel Atom Z2580 processor," McGregor wrote on Friday.

He added that it's still hard to be sure if even those results are valid, because AnTuTu has not explained what changes were made to the Intel version of the benchmark, although the new version is still built with ICC.

What's more, he said, AnTuTu is planning to revise its testing standards in August, which will presumably result in completely different scores across all of the various platforms, due to the inherent complexities of SoC components – so get ready for another round of finger-pointing then.

Asked for comment, a spokesperson for ABI Research told El Reg, "Honestly, we feel [McGregor] totally missed the point. He is focusing on the benchmark and not the power performance we highlighted. It is too easy to pick on benchmarks."

The spokesperson said ABI Research expects to have more to say on the matter next week. ®

The essential guide to IT transformation

More from The Register

next story
So, Apple won't sell cheap kit? Prepare the iOS garden wall WRECKING BALL
It can throw the low cost race if it looks to the cloud
Samsung Gear S: Quick, LAUNCH IT – before Apple straps on iWatch
Full specs for wrist-mounted device here ... but who'll buy it?
Apple promises to lift Curse of the Drained iPhone 5 Battery
Have you tried turning it off and...? Never mind, here's a replacement
Now that's FIRE WIRE: HP recalls 6 MILLION burn-risk laptop cables
Right in the middle of Burning Mains Man week
HUGE iPAD? Maybe. HUGE ADVERTS? That's for SURE
Noo! Hand not big enough! Don't look at meee!
AMD unveils 'single purpose' graphics card for PC gamers and NO ONE else
Chip maker claims the Radeon R9 285 is 'best in its class'
prev story

Whitepapers

Endpoint data privacy in the cloud is easier than you think
Innovations in encryption and storage resolve issues of data privacy and key requirements for companies to look for in a solution.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Advanced data protection for your virtualized environments
Find a natural fit for optimizing protection for the often resource-constrained data protection process found in virtual environments.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.