You won't guess where European mobile data was rerouted for two hours. Oh. You can. Yes, it was China Telecom
BGP leaks are common but don’t usually take hours to fix...
Comment Yet another large interweb routing blunder has prompted internet engineers to stress the need for additional security at the network's foundational layer, and again raised eyebrows at the behavior of China Telecom.
On June 6, more than 70,000 BGP routes were leaked from Swiss colocation company Safe Host to China Telecom in Frankfurt, Germany, which then announced them on the global internet. This resulted in a massive rerouting of internet traffic via China Telecom systems in Europe, disrupting connectivity for netizens: a lot of data that should have gone to European cellular networks was instead piped to China Telecom-controlled boxes.
BGP leaks are common – they happen every hour of every day – though the size of this one and particularly the fact it lasted for two hours, rather than seconds or minutes, has prompted more calls for ISPs to join an industry program that adds security checks to the routing system.
The fact that China Telecom, which peers with Safe House, was again at the center of the problem – with traffic destined for European netizens routed through its network – has also made internet engineers suspicious, although they have been careful not to make any accusations without evidence.
"China Telecom, a major international carrier, has still implemented neither the basic routing safeguards necessary both to prevent propagation of routing leaks nor the processes and procedures necessary to detect and remediate them in a timely manner when they inevitably occur," noted Oracle Internet Intelligence's (OII) director of internet analysis Doug Madory in a report. "Two hours is a long time for a routing leak of this magnitude to stay in circulation, degrading global communications."
Here's Madory's summary of the leak:
Swiss data center colocation company Safe Host (AS21217) leaked over 70,000 routes to China Telecom (AS4134) in Frankfurt, Germany. China Telecom then announced these routes on to the global internet redirecting large amounts of internet traffic destined for some of the largest European mobile networks through China Telecom’s network. Some of the most impacted European networks included Swisscom (AS3303) of Switzerland, KPN (AS1136) of Holland, and Bouygues Telecom (AS5410) and Numericable-SFR (AS21502) of France.
Back in November, Madory was diplomatic but ultimately supportive of a report from the US Naval War College that noted "unusual and systematic hijacking patterns associated with China Telecom." Madory noted in response to that report that he had "expended a great deal of effort" to end traffic misdirection by China Telecom.
The issue of BGP leaks has grown in recent years with criminal and possibly state actors realizing their potential for grabbing internet traffic: something that can then be used for a variety of questionable purposes, including surveillance, disruption, and theft. In one case, it was used to steal millions of dollars of virtual currency.
An analysis of routing errors in 2017 revealed that 38 per cent of the 14,000 reported incidents were due to leaks or hijacks (as opposed to outages). Thanks to the way the internet works as an interconnected, largely autonomous mass of roughly 60,000 networks linking up with one another, it is impossible to know for sure how many of those incidents were malicious or planned, as opposed to a simple mistake.
It should be noted that the United States remains the number one source of BGP errors – in large part thanks to the number of networks in America – but when BGP leaks have been flagged as potentially suspicious there has been a persistent connection to Chinese and Russian operators, strongly suggesting that there is an underlying degree of manipulation going on with the internet's infrastructure.
With constant and reliable internet access a growing necessity, any disruption to routing tables is a problem. And, of course, internet engineers have been pushing for the adoption of several fixes for years but have been hampered by the autonomous nature of the internet.
One key industry group called Mutually Agreed Norms for Routing Security (MANRS) has four main recommendations: two technical and two cultural for fixing the problem. The two technical approaches are filtering and anti-spoofing, which basically check announcements from other network operators to see if they are legitimate and remove any that aren't; and the cultural fixes are coordination and global validation – which encourage operators to talk more to one another and work together to flag and remove any suspicious looking BGP changes.
The internet industry has been slowly adopting that approach, resulting in a 10 per cent drop of incidents in 2018 over 2017 and a significant drop (17 per cent) in the number of networks responsible for routing errors. But it still isn't sufficient and, as with the domain name system, engineers stress that there needs to be a network-wide improvement in security.
The problem of course, as with everything to do with the internet's infrastructure, is that there is a cost associated with adding new checks to an existing system and that cost comes with little immediate benefit to those investing in it.
One of the ironies with BGP leaks is that they tend not to impact the network that made the mistake but rather the networks that connect to them. There is also no clear way of knowing which network operators are doing a good job and which aren't, so there is little financial driver to do a good job.
Make BGP great again, er, no, for the first time: NIST backs internet route security brainwaveREAD MORE
MANRS hopes to be able to come up with a way to show that some network operators are providing greater security online as a way to drive consumer choice – where users sign up with one company over another thanks to their perceived better security – but the truth is that it remains a pipe dream.
The industry is increasingly focused on the problem, however, making it notable that a company as large as China Telecom has still not implemented filtering and anti-spoofing measures when almost every other company its size has.
There is real benefit to a state actor having access to internet traffic routed through their systems – just ask the NSA which went to great lengths to make sure the US government was tapped into the global internet backbone: an effort that Edward Snowden ultimately blew up.
Currently China and Russia, almost certainly the United States, and occasionally Brazil and Iran, are relying on the fuzziness of the global network to give them plausible deniability in online surveillance efforts. But as more operators adopt protective security measures, unusual incidents will stand out more.
The BGP leak this month was likely a simple mistake but China Telecom appears to have made the most of it. And that has sparked internet engineers to again press their colleagues to adopt better security measures on this critical underlying internet infrastructure. ®