It's EE vs Vodafone: 'How good is my signal' study descends into network bunfight

Vodafone has implied that a mobile network test by RootMetrics was not "impartial" and included dodgy data. Vodafone came bottom in the testing. EE, which came top in the testing, has reacted with outrage at Vodafone's assertion.

Vodafone says: “We’d love to give a fully detailed response, but believe the way Root Metrics carried out its testing does not appear to follow standard industry practices or is fully impartial, while it also incorporates data some of which may well be over six months old.

The evidence we have seen of how Root Metrics conducted some of its tests leads us to believe that they were carried out in an inconsistent manner... We cannot take the results of this report seriously and neither should our customers.”

EE has responded in turn by saying:

Independent tests are, by their very nature, independent. This means that no operator knows where tests are being carried out or when, or influences the methodology or results. All operators can purchase the test results – though some choose not to.

The tests took place over the second half of 2013 and covered 1,000 indoor locations, with testers driving a total of 23,000 miles between locations.

RootMetrics claim to test from a consumer's point of view. This means it doesn't use specialised test equipment: instead it uses unmodified, shop-bought phones.

The firm says it buys the top-selling handset on each network but this means it is not always using the same model of phones as customers do. Even if a phone looks the same and is ostensibly the same model, there are possibly hundreds of different versions of each model, sometimes having different chipsets, radio circuitry and often customised radio interface layers.

The data gathered by RootMetrics uses an overlay of crowdsourced information from people who've downloaded the RootMetrics testing app, although this is only available on Android because, as we've said before, Apple doesn't give the necessary radio signal API to developers.

Bill Moore, CEO of RootMetrics, explained to The Register that it had decided that this reflects the user experience more accurately and gives a “best case” scenario. Testing organisations, however, would point to using equipment capable of analysing the radio signal as being more repeatable and accurate.

So who got what?

In the tests, which looked at reliability, speed, mobile internet and call reliability, EE came top in all categories and Vodafone bottom – except in speed where Vodafone was third and Three took fourth place. RootMetrics don't give any absolute figures for their results, just an indexed figure. You can, however, drill down on a cell by cell basis.

The scores were EE top with 84.6, Three second with 73.5, O2 third with 66.5 and Vodafone last with 52.4.

EE is at pains to point out that the methodology is very much of RootMetrics' choosing, and that the telecoms firm buys the rights to use the data for adverts and the raw data to look for areas where it has coverage issues. EE adds that it buys data from a variety of survey organisations, not just RootMetrics.

Even without knowing the weighting between the components of the score you can understand that EE will come top, but what's missing is the weighting between 2G, 3G and 4G, where a bias towards 4G might very much favour EE – even though a minority of UK mobile subscribers actually use 4G.

Handset testing on the rise

RootMetrics isn't the only company to use handset testing. UK company OpenSignal uses crowdsourced data. Anyone can download its Android – or unusually, iOS app – which will report the information back. A glance at its website shows that EE has the strongest signal in Bristol (the historic home of Orange) and Borehamwood (where T-Mobile grew up), and also in Slough, home of O2. In Newbury – home of Vodafone – it's O2 which has the best coverage.

Maidenhead is Three's base and at least it has the best coverage in its own back yard. In Paddington, which houses the London offices of both EE and Vodafone, it's EE which does best according to OpenSignal.

OpenSignal's Samuel Johnston says that he's not surprised by the results. "For 2G/3G we see Three performing the best nationally, with EE coming top for LTE. When combining the technologies we see EE performing best of the UK networks but it's less clear cut for us than it is for RootMetrics who have EE coming top of every major UK city - while we see, for instance, both Three and Vodafone outperforming EE in Manchester.

”The fact that RootMetrics combine 3G and LTE in an oblique way makes it hard for consumers to get informed about how networks will work for them,” continued Johnston, “especially for the majority of consumers who are still 3G-only."

+Comment

Drilling down through all the methodologies, claims and counter-claims, it appears that because RootMetrics splits its score by 45 per cent call, 45 per cent data and 10 per cent text performance – without breaking out the technologies – the companies with the biggest 4G footprint do best. Because it carried out its testing at a time when EE had, for regulatory reasons, been running 4G for a while, and Vodafone's 4G offering was relatively new, the headline figures may have skewed the numbers in favour of EE and away from Vodafone.

There is, however, an atmosphere of Vodafone complaining too much and EE crowing too loudly. That's quite likely to come back and haunt both of them in the future.

You can pull apart any testing methodology – particularly one with results you don't like – but given that only O2 has a regulatory obligation for 4G coverage, more testing is always a good thing. Ultimately testing is what will push operators to improve, particularly in buildings and rural areas where the costs and revenue don't always make it an obvious thing to do.

Politics aside, the RootMetrics results seem to be a pretty good wet finger to the wind for finding out when – and with what kit – RootMetrics did its testing. ®