It's EE vs Vodafone: 'How good is my signal' study descends into network bunfight

No, YOUR results are biased!

Boost IT visibility and business value

Vodafone has implied that a mobile network test by RootMetrics was not "impartial" and included dodgy data. Vodafone came bottom in the testing. EE, which came top in the testing, has reacted with outrage at Vodafone's assertion.

Vodafone says: “We’d love to give a fully detailed response, but believe the way Root Metrics carried out its testing does not appear to follow standard industry practices or is fully impartial, while it also incorporates data some of which may well be over six months old.

The evidence we have seen of how Root Metrics conducted some of its tests leads us to believe that they were carried out in an inconsistent manner... We cannot take the results of this report seriously and neither should our customers.”

EE has responded in turn by saying:

Independent tests are, by their very nature, independent. This means that no operator knows where tests are being carried out or when, or influences the methodology or results. All operators can purchase the test results – though some choose not to.

The tests took place over the second half of 2013 and covered 1,000 indoor locations, with testers driving a total of 23,000 miles between locations.

RootMetrics claim to test from a consumer's point of view. This means it doesn't use specialised test equipment: instead it uses unmodified, shop-bought phones.

The firm says it buys the top-selling handset on each network but this means it is not always using the same model of phones as customers do. Even if a phone looks the same and is ostensibly the same model, there are possibly hundreds of different versions of each model, sometimes having different chipsets, radio circuitry and often customised radio interface layers.

The data gathered by RootMetrics uses an overlay of crowdsourced information from people who've downloaded the RootMetrics testing app, although this is only available on Android because, as we've said before, Apple doesn't give the necessary radio signal API to developers.

Bill Moore, CEO of RootMetrics, explained to The Register that it had decided that this reflects the user experience more accurately and gives a “best case” scenario. Testing organisations, however, would point to using equipment capable of analysing the radio signal as being more repeatable and accurate.

So who got what?

In the tests, which looked at reliability, speed, mobile internet and call reliability, EE came top in all categories and Vodafone bottom – except in speed where Vodafone was third and Three took fourth place. RootMetrics don't give any absolute figures for their results, just an indexed figure. You can, however, drill down on a cell by cell basis.

The scores were EE top with 84.6, Three second with 73.5, O2 third with 66.5 and Vodafone last with 52.4.

EE is at pains to point out that the methodology is very much of RootMetrics' choosing, and that the telecoms firm buys the rights to use the data for adverts and the raw data to look for areas where it has coverage issues. EE adds that it buys data from a variety of survey organisations, not just RootMetrics.

Even without knowing the weighting between the components of the score you can understand that EE will come top, but what's missing is the weighting between 2G, 3G and 4G, where a bias towards 4G might very much favour EE – even though a minority of UK mobile subscribers actually use 4G.

Handset testing on the rise

RootMetrics isn't the only company to use handset testing. UK company OpenSignal uses crowdsourced data. Anyone can download its Android – or unusually, iOS app – which will report the information back. A glance at its website shows that EE has the strongest signal in Bristol (the historic home of Orange) and Borehamwood (where T-Mobile grew up), and also in Slough, home of O2. In Newbury – home of Vodafone – it's O2 which has the best coverage.

Maidenhead is Three's base and at least it has the best coverage in its own back yard. In Paddington, which houses the London offices of both EE and Vodafone, it's EE which does best according to OpenSignal.

OpenSignal's Samuel Johnston says that he's not surprised by the results. "For 2G/3G we see Three performing the best nationally, with EE coming top for LTE. When combining the technologies we see EE performing best of the UK networks but it's less clear cut for us than it is for RootMetrics who have EE coming top of every major UK city - while we see, for instance, both Three and Vodafone outperforming EE in Manchester.

”The fact that RootMetrics combine 3G and LTE in an oblique way makes it hard for consumers to get informed about how networks will work for them,” continued Johnston, “especially for the majority of consumers who are still 3G-only."


Drilling down through all the methodologies, claims and counter-claims, it appears that because RootMetrics splits its score by 45 per cent call, 45 per cent data and 10 per cent text performance – without breaking out the technologies – the companies with the biggest 4G footprint do best. Because it carried out its testing at a time when EE had, for regulatory reasons, been running 4G for a while, and Vodafone's 4G offering was relatively new, the headline figures may have skewed the numbers in favour of EE and away from Vodafone.

There is, however, an atmosphere of Vodafone complaining too much and EE crowing too loudly. That's quite likely to come back and haunt both of them in the future.

You can pull apart any testing methodology – particularly one with results you don't like – but given that only O2 has a regulatory obligation for 4G coverage, more testing is always a good thing. Ultimately testing is what will push operators to improve, particularly in buildings and rural areas where the costs and revenue don't always make it an obvious thing to do.

Politics aside, the RootMetrics results seem to be a pretty good wet finger to the wind for finding out when – and with what kit – RootMetrics did its testing. ®

Secure remote control for conventional and virtual desktops

More from The Register

next story
UK fuzz want PINCODES on ALL mobile phones
Met Police calls for mandatory passwords on all new mobes
Netflix swallows yet another bitter pill, inks peering deal with TWC
Net neutrality crusader once again pays up for priority access
Fifteen zero days found in hacker router comp romp
Four routers rooted in SOHOpelessly Broken challenge
EE: STILL Blighty's best mobe network, says 'Frappucino' Moore
Fresh round of network stats fisticuffs possibly on the cards here
New Sprint CEO says he will lower axe on staff – but prices come first
'Very disruptive' new rates to be revealed next week
US TV stations bowl sueball directly at FCC's spectrum mega-sale
Broadcasters upset about coverage and cost as they shift up and down the dials
Canadian ISP Shaw falls over with 'routing' sickness
How sure are you of cloud computing now?
UK mobile coverage is BETTER than EVER, networks tell Ofcom
Regulator swallows this line and parrots it back out at us. What are they playing at?
What's the nature of your emergency, Vodafone?
Oh, you've dialled the wrong number for ad fibs, rules ASA
EE network whacked by 'PDP authentication failure' blunder
Carrier is 'aware' of cockup, working on a fix NOW
prev story


Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 10 endpoint backup mistakes
Avoid the ten endpoint backup mistakes to ensure that your critical corporate data is protected and end user productivity is improved.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Rethinking backup and recovery in the modern data center
Combining intelligence, operational analytics, and automation to enable efficient, data-driven IT organizations using the HP ABR approach.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.