Feeds

It's EE vs Vodafone: 'How good is my signal' study descends into network bunfight

No, YOUR results are biased!

Designing a Defense for Mobile Applications

Vodafone has implied that a mobile network test by RootMetrics was not "impartial" and included dodgy data. Vodafone came bottom in the testing. EE, which came top in the testing, has reacted with outrage at Vodafone's assertion.

Vodafone says: “We’d love to give a fully detailed response, but believe the way Root Metrics carried out its testing does not appear to follow standard industry practices or is fully impartial, while it also incorporates data some of which may well be over six months old.

The evidence we have seen of how Root Metrics conducted some of its tests leads us to believe that they were carried out in an inconsistent manner... We cannot take the results of this report seriously and neither should our customers.”

EE has responded in turn by saying:

Independent tests are, by their very nature, independent. This means that no operator knows where tests are being carried out or when, or influences the methodology or results. All operators can purchase the test results – though some choose not to.

The tests took place over the second half of 2013 and covered 1,000 indoor locations, with testers driving a total of 23,000 miles between locations.

RootMetrics claim to test from a consumer's point of view. This means it doesn't use specialised test equipment: instead it uses unmodified, shop-bought phones.

The firm says it buys the top-selling handset on each network but this means it is not always using the same model of phones as customers do. Even if a phone looks the same and is ostensibly the same model, there are possibly hundreds of different versions of each model, sometimes having different chipsets, radio circuitry and often customised radio interface layers.

The data gathered by RootMetrics uses an overlay of crowdsourced information from people who've downloaded the RootMetrics testing app, although this is only available on Android because, as we've said before, Apple doesn't give the necessary radio signal API to developers.

Bill Moore, CEO of RootMetrics, explained to The Register that it had decided that this reflects the user experience more accurately and gives a “best case” scenario. Testing organisations, however, would point to using equipment capable of analysing the radio signal as being more repeatable and accurate.

So who got what?

In the tests, which looked at reliability, speed, mobile internet and call reliability, EE came top in all categories and Vodafone bottom – except in speed where Vodafone was third and Three took fourth place. RootMetrics don't give any absolute figures for their results, just an indexed figure. You can, however, drill down on a cell by cell basis.

The scores were EE top with 84.6, Three second with 73.5, O2 third with 66.5 and Vodafone last with 52.4.

EE is at pains to point out that the methodology is very much of RootMetrics' choosing, and that the telecoms firm buys the rights to use the data for adverts and the raw data to look for areas where it has coverage issues. EE adds that it buys data from a variety of survey organisations, not just RootMetrics.

Even without knowing the weighting between the components of the score you can understand that EE will come top, but what's missing is the weighting between 2G, 3G and 4G, where a bias towards 4G might very much favour EE – even though a minority of UK mobile subscribers actually use 4G.

Handset testing on the rise

RootMetrics isn't the only company to use handset testing. UK company OpenSignal uses crowdsourced data. Anyone can download its Android – or unusually, iOS app – which will report the information back. A glance at its website shows that EE has the strongest signal in Bristol (the historic home of Orange) and Borehamwood (where T-Mobile grew up), and also in Slough, home of O2. In Newbury – home of Vodafone – it's O2 which has the best coverage.

Maidenhead is Three's base and at least it has the best coverage in its own back yard. In Paddington, which houses the London offices of both EE and Vodafone, it's EE which does best according to OpenSignal.

OpenSignal's Samuel Johnston says that he's not surprised by the results. "For 2G/3G we see Three performing the best nationally, with EE coming top for LTE. When combining the technologies we see EE performing best of the UK networks but it's less clear cut for us than it is for RootMetrics who have EE coming top of every major UK city - while we see, for instance, both Three and Vodafone outperforming EE in Manchester.

”The fact that RootMetrics combine 3G and LTE in an oblique way makes it hard for consumers to get informed about how networks will work for them,” continued Johnston, “especially for the majority of consumers who are still 3G-only."

+Comment

Drilling down through all the methodologies, claims and counter-claims, it appears that because RootMetrics splits its score by 45 per cent call, 45 per cent data and 10 per cent text performance – without breaking out the technologies – the companies with the biggest 4G footprint do best. Because it carried out its testing at a time when EE had, for regulatory reasons, been running 4G for a while, and Vodafone's 4G offering was relatively new, the headline figures may have skewed the numbers in favour of EE and away from Vodafone.

There is, however, an atmosphere of Vodafone complaining too much and EE crowing too loudly. That's quite likely to come back and haunt both of them in the future.

You can pull apart any testing methodology – particularly one with results you don't like – but given that only O2 has a regulatory obligation for 4G coverage, more testing is always a good thing. Ultimately testing is what will push operators to improve, particularly in buildings and rural areas where the costs and revenue don't always make it an obvious thing to do.

Politics aside, the RootMetrics results seem to be a pretty good wet finger to the wind for finding out when – and with what kit – RootMetrics did its testing. ®

HP ProLiant Gen8: Integrated lifecycle automation

More from The Register

next story
Scotland's BIG question: Will independence cost me my broadband?
They can take our lives, but they'll never take our SPECTRUM
Auntie remains MYSTIFIED by that weekend BBC iPlayer and website outage
Still doing 'forensics' on the caching layer – Beeb digi wonk
Bring back error correction, say Danish 'net boffins
We don't need no steenkin' TCP/IP retransmission and the congestion it causes
NBN Co adds apartments to FTTP rollout
Commercial trial locations to go live in September
Samsung Z Tizen OS mobe is post-phoned – this time for good?
Russian launch for Sammy's non-droid knocked back
Telstra to KILL 2G network by end of 2016
GSM now stands for Grave-Seeking-Mobile network
Seeking LTE expert to insert small cells into BT customers' places
Is this the first step to a FON-a-like 4G network?
What FTC lawsuit? T-Mobile US touts 10GB, $100 family-of-4 plan
Folks 'could use that money for more important things' says CEO Legere
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Securing Web Applications Made Simple and Scalable
Learn how automated security testing can provide a simple and scalable way to protect your web applications.