ISP Level 3 goes TITSUP after giganto traffic routing blunder
Explanations spread way faster than Level 3 users' packets
ISP Level 3's customers have been left without internet access since this morning, after the provider seems to have leaked routes to a Tier 1 transit provider in Malaysia.
An incident report from CloudFlare said that while "the Tier 1 transit provider of the ISP leaking routes appears to have stopped accepting these announcements," some routing changes may still be re-occurring.
Taking a stab at explaining route leaking, amalcon at Hacker News offered:
Basically what's happened here is that Telecom Malaysia told one of Level3's networks (AS3549 - Global Crossing) that it was capable of delivering traffic to... anywhere on the Internet. Global Crossing apparently didn't have the usual sanity checks in place.
Because of how BGP works, once [Level 3] decided the route seemed reasonable, it immediately proceeded to dump huge amounts of traffic on the tiny Telecom Malaysia.
This isn't incorrect behavior on [Level 3]'s part (accepting the routes was incorrect, but sending the traffic was correct once they were accepted). This is way more traffic than Telecom Malaysia was prepared to handle.
Every tier 1 network gets rid of traffic as soon as possible (because it reduces their costs), completely ignoring performance or whether a route seems sensible. Telecom Malaysia therefore claimed to [be] the most attractive route from anywhere near Malaysia to most of the world.
Traffic should now be back to normal, CloudFlare says, but many on Twitter find that this is not the case.
The complaints began earlier this morning and seemed to affect users across the globe. Plenty of debate continues as to the exact cause and nature of the outage.
Tweets from Level 3 suggest it is dealing with the problem but that the issues are ongoing.
A Level 3 spokeswoman told The Register: "Level 3's network, alongside some other ISPs', experienced service disruptions affecting customers in EMEA and globally on June 12th. Maintaining a reliable, high-performing network for our customers is our top priority. Our network technicians worked to resolve the issue, and services have been widely restored."
"The root cause has been isolated and linked back to an Asian provider issue; we are working closely together to address this and prevent it from recurring." ®