Original URL: https://www.theregister.com/2012/11/06/google_outage_asia/
Hardware failure melts down Chocolate Factory
CloudFlare engineers sort out Google's problem
Posted in Networks, 6th November 2012 23:40 GMT
Google users in Southeast Asia had an outage last night after an Indonesian ISP had a bit of a meltdown.
Engineers at content-delivery provider CloudFlare noticed the outage at around 2:24 UTC on Tuesday after their own Google Apps services went offline. The outage was primarily an Asian problem and the company estimates between 3 and 5 per cent of the total internet population was affected.
CloudFlare network engineer Tom Paseka said on the company blog that an analysis of the problem showed that the company was unable to reach Google's public DNS server, and saw that packets of data were being rerouted via Indonesian ISP Moratel, and that this was causing the outage.
"This wasn't malicious at all," Paseka told The Register. "It was a hardware failure that caused the initial problem, and then a software failure following on. Moratel were aware of the router failure but not the problem with the uplink."
Network engineers are a chummy lot, staying in contact around the world. Paseka called an engineer he knows at Moratel and between them they got the problem sorted out and the service back online within around 30 minutes.
Google didn't even need to be involved in the fix, although Paseka said they were nice enough to get in contact after the event to say thank you. The Chocolate Factory had no official statement on the matter, but sources familiar with the affair said that the issue did highlight some problems with the current trust-based systems of routing. ®