Google routing blunder sent Japan's Internet dark on Friday

Another big BGP blunder

Last Friday, someone in Google fat-thumbed a border gateway protocol (BGP) advertisement and sent Japanese Internet traffic into a black hole.

The trouble began when The Chocolate Factory “leaked” a big route table to Verizon, the result of which was traffic from Japanese giants like NTT and KDDI was sent to Google on the expectation it would be treated as transit.

Since Google doesn't provide transit services, as BGP Mon explains, that traffic either filled a link beyond its capacity, or hit an access control list, and disappeared.

The outage in Japan only lasted a couple of hours, but was so severe that Japan Times reports the country's Internal Affairs and Communications ministries want carriers to report on what went wrong.

BGP Mon dissects what went wrong here, reporting that more than 135,000 prefixes on the Google-Verizon path were announced when they shouldn't have been.

Since it leaked what the monitors call “a full table” to Verizon, the fat-thumb error also provided a “peek into what Google's peering relationships look like and how their peers traffic engineer towards Google”.

For example, BGP Mon explains how the mistake hit ISP Jastel (Jasmine Telecom) in Thailand:

“If we take a closer look at the AS paths involved starting at the right side, we see the prefix was announced by 45629 (Jastel) as expected. Since Jastel peers with Google (15169) that’s the next AS we see. The next AS in the path is 701 (Verizon) and this is where it is getting interesting as Verizon has now started to provide transit for Jastel via Google.

“Verizon (701) then announced that to several of it’s customers, some of them very large such as KPN (286) and Orange (5511). So by just looking at four example paths we can see it hit large networks in Europe, Latin America, the US, and India (9498 Airtel).”

BGP is the Internet's protocol for distributing routing information between networks. A BGP advertisement shouts out to the rest of the internet to announce things like “if you give me traffic for Verizon, it will reach its destination”.

Designed for a more trusting (and much smaller) Internet, BGP's most serious shortcoming is that it's up to network admins to check and filter information in route advertisements.

As BGP Mon notes, BGP leaks are “a great risk to the Internet's stability”, and both sides of an advertisement should be filtering them before accepting them.

Previous BGP incidents have sent YouTube traffic to Pakistan, blackholed Chinese traffic, made Belarus the default route for more traffic than it could handle, and redirected Level 3's traffic to Malaysia.

There are various proposals to tweak BGP to stop this sort of thing happening, but as is so often the case, implementation is lagging far behind requirement. ®

Sponsored: The Joy and Pain of Buying IT - Have Your Say


Biting the hand that feeds IT © 1998–2017