Unknown fault darkens Australia’s Internet

Updated: a Dodo routed the net to a brief extinction

By Richard Chirgwin, 23rd February 2012

Customers of major Australian ISPs are stranded with no connection to the outside world this afternoon, taken out of action by an as-yet-undiagnosed routing fault.

As is typical of a country in which too many services are dependent on too few international links, The Register can directly confirm that the outage affected customers of Telstra, Optus and iiNet.

The severity of the outage seems to vary depending on carrier. iiNet customers at the time of writing have intermittent access to local addresses, but no international access. At 2.15pm AEST – 11.15 in WA – this author was able to view iiNet’s status page, where the company had logged a fault.

Its fault notice I908406 merely stated that “Customers trying to access international websites and services may experience delays and timeouts. iiNet Engineers are working to restore the services as quickly as possible.”

The iiNet notice attributed the problem to an “upstream routing issue”.

Telstra users seem to fare little better: they can get to Telstra’s site, but not much further (for example, its Bigpond.com – where network status is posted – is not accessible over Telstra mobile broadband).

Telstra took to Twitter to advise users when its services returned – which is just about as good as anyone can expect in the era of customer self-service, since a user without Internet access is also without Twitter.

Optus’ service status page had not logged any faults at 2.30pm, even though some Optus mobile Internet customers were unable to access even Optus content.

Update: An Optus spokesperson later stated to The Register that he was advised that Optus was not affected by this outage.

At the time of writing, iiNet was unable to comment to The Register, Telstra’s “media hotline” was playing a “what we’re doing over Christmas” message, and Optus was seeking information from its network personnel.

Posts to Australia's AusNOG (Australian network operators' group) mailing list attributed the problem to a Telstra international cable, however at the time of writing, The Register has been unable to confirm this. ®

Second Update: It later emerged that a faulty router at retailer Dodo caused the problem.

Author's Note: I included Optus in the list of ISPs affected because it is among he ISP accounts I have access to (via an Android 3G phone with a 2G data plan). With WiFi off and 3G on, that device was unable to contact any Websites at the same time as the major outage. I will be happy to provide a further update if Optus identifies another cause for this outage. ®

What went wrong: This post from the Australian Network Operators' Group mailing list this morning (February 24) provides an explanation of what happened.

The author, Aaron Swayn - a consultant for Ethan Group - says that the fault lay initially with retail ISP Dodo, which advertised 390,000 IP address prefixes to Telstra.

Whatever the cause - Dodo has blamed a "hardware fault" - the company was saying "here is the Internet". Perhaps because Dodo is a small retailer, Telstra had never expected it to advertise that many addresses, and so did not impose a limit on what it would accept from Dodo.

"Telstra then advertised the prefixes to Reach", Swayne says, which did notice the problem "due to 'max-prefixes' being breached"; and when Reach noticed, it shut down BGP from Telstra until the problem was fixed.

Swayn has asked The Register to make it clear that his note to Ausnog should be regarded as his personal opinion only, and that he was not speaking on behalf of his employer, Ethan Group. ®

Steps to Take Before Choosing a Business Continuity Partner