4G is quicker than 3G, says Ofcom. Can't we get you on Mastermind, Sybil?

Methodology defended

Ofcom also argues that the testing is part of an ongoing series of tests. Coverage test results will be published in another report in the spring, and there will be ongoing speed tests to show improvements in the networks.

The coverage information is badly needed, and it’s lamentable that the way these tests were done was to ask the operators where to test. Ofcom argues that it was necessary because they could only test where 4G was available. It defends the methodology, saying:

• Each network was tested concurrently to ensure that environmental conditions were the same for each operator

• Identical handsets were used for each network

• SIMs were rotated between devices to eliminate any bias that may occur from variations in individual handset performance or location

• All of our testing took place while static to ensure repeatability

• Sixteen measurements for each metric were taken for each network at each test location. Handsets were rotated after four cycles to ensure that each handset spent the same amount of time at each point

• Undue contention was avoided by testing networks in parallel and ensuring that no concurrent tests were run on the same network

• All testing took place on each network at the same time (between 7am and 7pm, Monday to Friday) and at the same location, on the same equipment and under the same circumstances.

Ultimately any methodology can be picked apart point by point, but what matters is how it will affect the operators desire to improve the networks for next time around. It doesn’t succeed at that either.

One thing which made the testing substantially easier for Ofcom when compared with our Monopoly board testing is that Ofcom used SIMs supplied by the operator which did not have any data limits. We found that even with SIMs which had been on a contract for years our usage was throttled when we started hammering them in testing, so the operators will have had to set up the SIMs especially for Ofcom.

Even then, they ran into capping problems. Sensibly Ofcom benchmarked them against shop-bought SIMs to make sure there was nothing underhand, although the report notes that despite asking EE for a single speed SIM the one which was supplied was double speed.