Canadian ISP Shaw falls over with 'routing' sickness
How sure are you of cloud computing now?
Updated Canadian ISP Shaw is experiencing a "nationwide outage affecting internet browsing", according to the polite phone robot put in place to discourage affected punters from overwhelming support systems.
The issue appears to be multi-faceted. Your Canadian vultures have personally encountered numerous "rolling" outages over the course of the day, and the issue continues to to evolve.
The first hint of an issue occurred when we were unable to RDP into any of our systems in our Edmonton test lab, which is on a Shaw network. Despite lack of RDP access, FTP, VPN and other protocols worked just fine to our boxen.
We were able to RDP into a Calgary datacenter for a period of several hours while we could not RDP into the Edmonton lab, and then that Calgary site too was inaccessible.
At various times Office 365 has been unavailable to both your northern vultures and various readers on a Shaw network. HTTPS-based access to my personal Exchange server has been unavailable all day and Teamviewer access to any Shaw-based systems is sketchy at best.
At some points we have lost all access to individual Shaw-based networks while others remain unaffected. The lengths of outages appear to be as random as the individual network segments affected, and network segments that had been "cured" have gone down again spontaneously over the course of the day.
Potential routing issues have been experienced by some customers. Please visit http://t.co/ka0CGaUb8c for more info. Thanks. ^KN— Shaw Help (@Shawhelp) August 12, 2014
Calling Shaw's Network Operations Center to lodge a support ticket turned up a description from the support tech that it is "a routing issue".
We here at Vulture North feel that the problem has to be little bit more than "a routing issue", given the number of networks in which specific protocols are affected, but not others. In addition to any routing issues, we're wondering if something has gone wrong with the deep packet inspection gear that Shaw uses.
How else could we be seeing issues like "all RDP access to a given network is denied, regardless of port number", but all VPN traffic and http traffic is fine? Or instances in which Office 365 and Exchange HTTPS traffic is suppressed, but regular HTTPS traffic to other websites carries on just fine?
We asked Shaw for comment, and will update this story when we know more. ®
Updated to add
We've drilled into what happened, right here: the internet hit an arbitrary limit of more than 512,000 routes, which broke a lot of kit worldwide.