Benchmark smartphone drama: We wouldn't call it cheating, says Huawei, but look, everyone's at it
So is that a yes or a no?
Updated Huawei has addressed the issue of tweaking a phone's performance to improve its benchmark scores, after being caught redhanded.
The practice isn't new, and has been carried out by a number of phone makers in the past. It works like this: To boost a system's benchmark score, the phone detects whether a popular benchmark is running and ramps up the clock frequency.
"Newer devices come with a benchmark detection mechanism that enables a much higher power limit for the SoC with far more generous thermal headroom. Ultimately, on certain whitelisted applications, the device performs super high compared to what a user might expect from other similar non-whitelisted titles. This consumes power, pushes the efficiency of the unit down, and reduces battery life," AnandTech's Andrei Frumusanu and Ian Cutress explained.
With benchmark detection on, the power draw of an Honor Play increased from 4.39W to 8.57W.
The AnandTech staffers stumbled across the practice after recent benchmarks carried out on Huawei and Honor phones using Huawei's Kirin 970 SoC didn't quite match benchmark results from earlier in the year, which were slower. At the start of the year, Huawei said the slower than expected results were the result of a firmware "bug". The revised results showed the Kirin 970 as markedly slower than phones using Qualcomm, Samsung and Apple chips - particularly in graphics tests.
Samsung denies benchmark cheating, despite evidenceREAD MORE
In an interview with the publication, Huawei didn't outright admit the practice, but explained in a roundabout way that China vendors need to do it to stay competitive.
"Others do the same testing, get high scores, and Huawei cannot stay silent," Wang Chenglu, president of software at Huawei’s Consumer Business Group, told the site.
Samsung was found to be "enhancing" the performance of its phones when it detected a benchmark app in 2013 - again by the sleuths at Anandtech - but denied "cheating". Samsung's explanation was both similar and similarly devoid of moral judgement: everyone else does it, the Korean giant said.
HTC was also found to be inflating its scores and, along with Samsung, was delisted by UL (formerly Futuremark), which maintains the benchmark, back in 2014.
But this was long ago - four years is an eternity in smartphones. Benchmarks have diminished in importance as the market and the technology matured. Older devices run very well, and so the necessity of "future-proofing" your investment by buying the fastest phone possible is no longer justifiable, unless you have a specific interest, like gaming. The delays you encounter on a 2018 smartphone are far more likely to be the result of rotten network service.
In a statement, Huawei told us today:
Huawei always prioritizes the user experience rather than pursuing high benchmark scores – especially since there isn’t a direct connection between smartphone benchmarks and user experiences. Huawei smartphones use advanced technologies such as AI to optimize the performance of hardware, including the CPU, GPU and NPU.
When someone launches a photography app or plays a graphically-intensive game, Huawei’s intelligent software creates a smooth and stable user experience by applying the full capabilities of the hardware, while simultaneously managing the device’s temperature and power efficiency. For applications that aren’t as power intensive like browsing the web, it will only allocate the resources necessary to deliver the performance that’s needed.
In normal benchmarking scenarios, once Huawei’s software recognizes a benchmarking application, it intelligently adapts to 'Performance Mode' and delivers optimum performance. Huawei is planning to provide users with access to 'Performance Mode' so they can use the maximum power of their device when they need to.
Huawei – as the industry leader – is willing to work with partners to find the best benchmarking standards that can accurately evaluate the user experience.
Updated to add
UL has now delisted four Huawei devices as a result of the digital slight of hand.
"This kind of detection and optimization is forbidden by our rules for manufacturers. We contacted Huawei with our findings, and we are happy to say that they have pledged to implement a more transparent approach in a future update," it said in the statement.
Sponsored: Becoming a Pragmatic Security Leader