Free ESG report : Seamless data management with Avere FXT

On the heels of the release of Apple's iOS 7, the researchers at Pfeiffer Consulting – a firm whose tagline is "Quantifying the Intangible" – sought to answer one simple question: "How good is it really?"

To Pfeiffer, competition in the smartphone market is about software, not hardware.

"Take any recent top-of-the-line smartphone, and you are likely to get a well-designed, fast, pleasant to use bit of hardware: fluid operation, responsive interaction, fast graphics," their newly released research report (PDF) argues.

"The difference of user experience, therefore, stems almost exclusively from the operating system, the user interface design, the application integration, the overall coherence," they conclude – and so the Pfeiffer Report Benchmark Project set out to test those factors with Apple's iOS 6 and iOS 7, Windows Phone 8, Blackberry 10, and Samsung's implementation of Android.

Why Samsung's Android? Simple: "We have chosen Samsung's implementation of Android," they write, "since, given the overwhelming market share of Samsung in the smartphone market, it is clearly the most widely used version of Android currently distributed."

The Pfeifferians rated each operating system on four criteria:

• Cognitive load: the sum of elements with which users must familiarize themselves to use the operating system spontaneously and intuitively
• Efficiency: how efficiently key features and services are integrated for ease of operation
• Customization: how many options an operating system provides to allow a user to tailor it to their personal tastes
• User-experience friction: Pfeiffer describes this as "the bad stuff, the aspects of a device that can annoy you in a niggling way, or, in extreme cases, drive you crazy"

The company scored each of these attributes using what it identifies as "objective criteria for comparing, discussing and measuring the user experience of digital devices." A bit of information about their process can be found here, but as a many-year veteran of the product-testing game, your humble Reg reporter can assure you that objectivity in such matters is an elusive goal, to say the least.

To cut to the chase, the Pfeiffer report concludes that, yes, iOS 7 really is the best smartphone operating system available today. "The combined results of the four different benchmarks and evaluations give iOS 7 a clear advantage in terms of overall user experience," they write, "taking into account the context defined for these benchmarks: day-to-day user experience of an average, non-technical user."

It's not looking good for Microsoft's Windows Phone 8, is it?

Samsung's Android outscored iOS 7 in only one of Pfeiffer's four categories: customization. "Android has the longest list of customization options of all mobile operating systems surveyed," they write, "but lacks some of the accessibility options present in iOS." In that same category, Windows Phone 8 stunk up the place, scoring a mere 2 out of 10 possible points.

In addition, its lack of customization contributed to WinPho's poor showing in user-experience friction, as well. "The user interface in general is not conceived to deal efficiently with the dozens and dozens of apps smartphone users want," Pfeiffer writes. "Add to that the lack of some core user interface features and you get an idea of Microsoft's challenges to make Windows Phone truly competitive."

An 'average, non-technical user' would be cognitively overloaded by Samsung's Android, says Pfeiffer
(click to enlarge)

As poorly as Windows Phone 8 did in "UXF", Samsung's Android did even worse in its cognitive load rating. "Samsung's version of Android has by far the highest cognitive load of all the mobile operating systems surveyed here," the report found, citing "no less than 104 apps and widgets, more than 4 times more than iOS 7, and a significantly higher number of icons and user interface elements."

In a nutshell, the report contends, "What separates the Android user experience from iOS 7 is not functionality, but feature-bloat and sloppy user interface design. In terms of user experience, less IS more." ®

Free ESG report : Seamless data management with Avere FXT