Original URL: https://www.theregister.com/2014/08/21/mobile_coverage_testing_part_1_monopoly_crawl/
Roll up for El Reg's 3G/4G MONOPOLY DATA PUB CRAWL
Part one: Measuring how many bars you get in London
Posted in Networks, 21st August 2014 09:02 GMT
Special Report In January I was in the pub with Richard, an old school friend I hadn’t seen for 30 years and whom I was keen to impress. Then he asked one of those seemingly simple questions. “Simon,” he said. “You’ll know which 4G network I should be on”.
Just 10 words but a whole heap of trouble. And like any good pub conversation, it led to stuff happening. Not long afterwards I met US coverage experts GWS at Mobile World Congress. I saw a way to answer Richard’s question. All I needed was to ask them for some help.
GWS delivered magnificently and we set about a project for which the company – if they had been billing The Register – would have charged approaching $100,000, although the kit costs at least that much.
Global Wireless Solutions Inc, or GWS to its friends, has an extensive team of testers who drive around significant parts of the US in a fleet of eighty Chrysler Grand Voyagers packed to the gills with Rohde & Schwarz test gear, checking mobile coverage for the US networks.
The bit of kit GWS is most excited by is the QualiPoc Freerider II from SwissQual. A custom backpack for testing mobile coverage, it liberates testing from the back of a van to measure coverage in locations where people actually use the things. We wanted something a little more comprehensive than just looking at Richard’s house and office, and so we looked for London landmarks instead.
I’d done something similar before. Twenty years ago when I was at What Mobile magazine, we looked at making phone calls from places which were featured on the Monopoly board. Journalist Derek Cohen wanted to know if the brand new Orange phone he had was any good. It turns out not only was Orange the 1994 winner, but that Orange used Derek’s route for testing for many years afterwards.
Twenty years on, the same question is still relevant. With more resources than Derek and his Yamaha Fazer 600, GWS set about deriving an authoritative answer. Part of the reason for using the QualiPoc Freerider was that it can measure in-building coverage, and we were discussing this in The Register's London office when tech wizard Murray had a flash of insight: “So it’s a pub crawl?”
Fabrice Tanoh and Drissa Coulibaly testing coverage at Marylebone Station
So here it is, Richard: The Register MObile Network OPeratOr anaLYsis (MONOPOLY) CoveRage And Weekday Legwork (CRAWL), a coverage survey, a board game and a pub crawl.
Two experienced GWS testers, Fabrice Tanoh and Drissa Coulibaly, flew over from Virginia and spent the last week of May and the first week of June trudging the streets of London, each of them equipped with the test kit. Over the time they made 4,511 voice calls and 22,915 file transfers. In doing so they ran up mobile data bills of £4,982 and quite a bit of time was spent explaining to the mobile networks that we were not fraudulently using the SIMs – in fact we were often paying up front, even on the contract SIMs, so that data didn’t get throttled or switched off. (The month after the testing, when one of the SIMs was returned to GWS’s PR man Colin Myer and a £1,500 bill had been paid, Vodafone asked for another £200 upfront)
How we defined our measurements
With vast amounts of data to trawl through, we needed a "non-engineer friendly" way to present it and one which would show the differences between the networks. It’s all very well saying “you could make a voice call on all of the networks all of the time”, but that doesn’t establish which one is best. Voice is only carried on the 2G and 3G networks, as none of the UK networks offer VoLTE.
Oxford Street Voice Data
Here is the processed summary data. In this case it’s for Oxford Street but we have similar metrics for all the 28 locations.
A total of 12 Samsung Galaxy S4 phones were used in parallel, with three devices being assigned to each operator as follows: Four were used for voice testing, four were used for 3G-only packet data testing and four were used for 4G-only data testing.
While there is a benefit to testing each operator's highest network technology (ie, 4G LTE), it has been found that the vast majority of users are still on the 3G network. Therefore, the performance of the 3G network must be taken into account.
For voice testing we receive metrics for how often the 3G signal drops back to 2G and show the number of calls made. The data gives us three categories of information: how easy it is to make a call, how "retainable" the call was in terms of not being dropped and how the sound quality is interpreted using Ec/Io. This is a signal-to-noise ratio.
In the data case, we have one device testing the 3G-only network and one testing the 4G-only network. This isn't exactly customer-experience testing, but it does give us a unique performance perspective of each network – and customer experience in terms of throughputs would likely be worse (since a real user would drop down to 2G or 3G, respectively).
Ec/Io is the ratio of a single user's intended signal versus the undesired interference and noise from all sources. Normally, GWS would use MOS (Mean Opinion Scoring) to assess voice quality.
MOS uses the Perceptual Objective Listening Quality Assessment (P.OLQA) algorithm to compare a source reference file against the recorded voice quality on the receiving end. Unfortunately, this requires the mobile device to call a dedicated answering station, which was not available for this test. In lieu of MOS, Ec/Io was correlated to voice quality using years of historic data provided by GWS.
Everything is always a balancing act and it’s one of those art-meets-science things to get right. There is a company called Optulink whose business is built around getting the balance right, which is just about to go into UK trials.
Simon shows how many bars he has
The "engineering way" of measuring signal strength uses dBm as a unit for each value (i.e. -30 dBm). Those values are averages for the whole location and not a percentage of time. The number of houses and hotels is determined by the successful call set up, successful call completion and voice quality percentages.
We measured Accessibility (being able to make a call), Retainability (being able to keep the call active to completion) and Quality (the clarity of the call). Accessibility and Retainability were weighted 50/50 into a Reliability score and the Quality was used separately as another ranking factor.
This is the mobile broadband info for Oxford Street
It's not just voice – we're looking at 3G and 4G too
For packet data testing, we need to get much more scientific than just running a couple of tests and showing the best speed. We ran a number of download tests and showed the harmonic mean of all download task throughputs. Only successfully terminated tasks are included in the harmonic mean task throughput calculation.
It is calculated by taking the sum of all application-level data transferred (kilobits) divided by the sum of ALL task durations (seconds). For both upload and download, the data transfer task was fetching or sending a 4MB file from or to the GWS server. The maximum time allowed for task completion was two minutes.
Most of the testing was done on foot with a little bit of drive-round testing using the Citroen C-zero electric car that we had on test at the time.
The equipment we used came from the Rohde and Schwarz company SwissQual, using the aforementioned QualiPoc Freerider II backpacks. Each pack – we had two – contains six Samsung Galaxy S4s which had been rooted to run custom software. This gave us full access to Layer 3 radio information. Getting the details such as network-level messaging and reasons why something has failed is essential to get an accurate look at the network's performance. This gives us greater insight into the reasoning behind each failure, as well as assurance that it was the network that caused the failures and not the device itself.
The Freerider weighs about 6kg, which includes an extended battery. A separate Android tablet controls the six phones through Bluetooth. Time synchronisation is controlled by the Android tablet and ensures reliability on all KPI calculations, so that networks can be accurately compared as like-for-like.
The system gives full logging and decoding of multiple protocol layers, 3GPP, Layer 3 and TCP/IP, with text decoding of Layer 3 and TCP/IP packets. The data collection is performed on the Samsung Galaxy S4s and then uploaded in scheduled bursts, during downtime, to the GWS servers in Virginia. There it hits the post-processing platform which provides a huge variety of post-processing options.
For The Register's MONOPOLY CRAWL test, the post-processing took the best part of eight weeks, with GWS carefully analysing the data and testing its validity by both looking at raw engineering data and applying statistical confidence models to the information.
I'll have an Ericsson RBS 6120 on Mayfair
This is vastly superior to testing methods that crowd-source the data or which use unmodified customer phones as there is little control or validity of the data. With access to the full radio data you can spot an anomaly and disregard it.
The 2014 MONOPOLY CRAWL test is far more comprehensive than the one What Mobile did in 1994.
We’ll be announcing winners for each of the Monopoly colour groups for voice calls on 22 August and for data transmission in the new look Weekend edition Register on 23 August, when we’ll announce the overall winners.
The Register's upcoming interactive map. Keep an eye out for the full map in our Weekend edition
On 29 August, we’ll be holding a live chat with Dr Paul Carter, the boss of GWS, to explain the decisions that were made in testing and to look at some of the interesting lessons learned, such as when mobile networks have better upload than download capabilities and who is sacrificing 3G to promote 4G.
Then, and only then, will my friend Richard know which mobile phone network he should be on. It has only taken six months for me to answer him. Maybe it’s time for another pint? Keep an eye out for the next instalment.
You can email questions ahead of the session to livechatquestions@theregister.co.uk or log on next Friday (29 August) to post them live. ®