Feeds

Analyst: Tests showing Intel smartphones beating ARM were rigged

Bad benchmark gave bad results

Secure remote control for conventional and virtual desktops

So does Intel's Atom-based Clover Trail+ platform really outperform ARM processors in smartphone benchmarks? Not so fast, says one analyst.

In June, The Reg reported analyst firm ABI Research's claim that it had pitted a Lenovo K900 smartphone based on Intel's Atom Z2580 processor against a brace of devices build around ARM system-on-chip (SoC) components and found that not only did the Intel part perform better, but it also drew less power.

But Jim McGregor of analyst firm Tirias Research smelled something fishy, and after investigating, he now says the surprise showing by Intel had less to do with the chip itself as it did with inconsistencies in the AnTuTu benchmark used to conduct the tests.

McGregor's first clue was that different versions of the benchmark produced wildly different results.

"Going from the 2.9.3 version of the AnTuTu benchmark to the 3.3 version, the overall AnTuTu scores increased 122 percent, and the RAM score increased 292 percent for the Intel processor, while the scores for the Samsung processor increased only 59 percent and 53 percent, respectively," McGregor wrote in a blog post at EE Times. "This was just from a change in the benchmark test, not the processors."

The discrepancy, it seems, stems from the fact that the binary executables of AnTuTu versions 2.9.4 and later are built for Intel chips using a different compiler than that used for other processor architectures.

Versions of AnTuTu for ARM chips are built using the open source GCC compiler. But beginning with version 2.9.4, AnTuTu for Intel is built using ICC, a proprietary optimizing compiler designed by Intel itself.

Working with AnTuTu and technology consulting firm BDTI, McGregor determined that the version of the benchmark built with ICC was allowing Intel processors to skip some of the instructions that make up the RAM performance test, leading to artificially inflated results.

AnTuTu released version 3.3.2 of the benchmark on Wednesday to address the problem, and according to McGregor, it negates Intel's artificial advantage. Intel's CPU and Overall scores are now about 20 per cent lower than they were with the previous build, and the RAM score is around 50 per cent lower.

"As a result of the revised scores, the AnTuTu benchmark is no longer an outlier from the other benchmarks and paints a competitive picture similar to the other benchmarks, with the Samsung Exynos 5 Octa processor outperforming the Intel Atom Z2580 processor," McGregor wrote on Friday.

He added that it's still hard to be sure if even those results are valid, because AnTuTu has not explained what changes were made to the Intel version of the benchmark, although the new version is still built with ICC.

What's more, he said, AnTuTu is planning to revise its testing standards in August, which will presumably result in completely different scores across all of the various platforms, due to the inherent complexities of SoC components – so get ready for another round of finger-pointing then.

Asked for comment, a spokesperson for ABI Research told El Reg, "Honestly, we feel [McGregor] totally missed the point. He is focusing on the benchmark and not the power performance we highlighted. It is too easy to pick on benchmarks."

The spokesperson said ABI Research expects to have more to say on the matter next week. ®

Choosing a cloud hosting partner with confidence

More from The Register

next story
Ford's B-Max: Fiesta-based runaround that goes THUNK
... when you close the slidey doors, that is ...
All aboard the Poo Bus! Ding ding, route Number Two departing
Only another three days of pooing and I can have a ride!
Official: European members prefer to fondle Apple iPads
Only 7 of 50 parliamentarians plump for Samsung Galaxy S
Fujitsu CTO: We'll be 3D-printing tech execs in 15 years
Fleshy techie disses network neutrality, helmet-less motorcyclists
Space Commanders rebel as Elite:Dangerous kills offline mode
Frontier cops an epic kicking in its own forums ahead of December revival
prev story

Whitepapers

Choosing cloud Backup services
Demystify how you can address your data protection needs in your small- to medium-sized business and select the best online backup service to meet your needs.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
10 threats to successful enterprise endpoint backup
10 threats to a successful backup including issues with BYOD, slow backups and ineffective security.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Getting ahead of the compliance curve
Learn about new services that make it easy to discover and manage certificates across the enterprise and how to get ahead of the compliance curve.