UK ISPs crippled by undersea cable snap
Many quickly re-routed but slowcoaches O2, Be slapped
Magic Quadrant for Enterprise Backup/Recovery
Updated A major internet routing outage struck UK telcos over the bank holiday weekend - knackering access to the World of Warcraft website*, the BBC, Amazon, Facebook and other sites for more than 24 hours.
It's understood that a submarine cable carrying web traffic snapped between Blighty and the Netherlands, causing headaches for ISPs. Some Reg readers affected by the downtime had suspected that the culprit was undersea fibre optic cabling that had been mistakenly sliced.
Rumours that some considered to be outlandish on O2's customer forum suggested that a submarine may have hit a line in the Atlantic, causing internet connection issues throughout Europe on Sunday.
Meanwhile, officially, O2 told its customers:
Dear members, the recent routing issues causing unreachable destinations from our network are caused by a major fibre breakdown.
A number of Reg readers wondered if global internet connectivity outfit Level 3 had been responsible for the outage. However, the company confirmed to us that it too had been affected as a customer of networking biz Interoute, whose sub-sea cable was understood to be at fault.
Level 3 said in a statement to us:
We do have some leased capacity from Interoute which was impacted by the cable cut. Upon notification of the failure on the Interoute Network, Level 3 took immediate action to re-route our customers onto our own network.
UK telcos such as BSkyB and BT were apparently relatively swift at responding to the problem by routing their networks in order to keep disruption to a minimum.
However, Telefonica - which owns O2 and Be - came under fire from its punters for what many considered to be a slow response in tackling the outage.
One frustrated customer said on O2's forum just yesterday:
Almost 30 hours to get so much as any acknowledgement on any of the O2 status pages. A mind numbingly inept display by O2.
While another complained:
It's been 32 hrs since I went down, still no update from 02 or any sort of a time scale, according to net BT and Sky had this fixed hours ago.
El Reg has asked O2 and Interoute to comment on this story, but neither had responded at time of publication. We'll update the piece if O2 or Interoute do furnish us with statements. ®
* WoW was still down "for maintenance" at the time of publication.
Updated to add
Interoute's chief technology officer Matthew Finnie has been in touch to confirm that on 27 August at 02.57am GMT, the undersea cable broke approximately 120km off the UK shore and 100km off the Netherlands shore.
"A specialist cable marine crew and vessel have been engaged," he said. "The timeline for a full system fix is currently pending. The cause of the fibre break has not yet been fully determined. However, current information suggests that the culprit was a ship's anchor.
"Approximately 400Gbps of services have been successfully re-routed and handed back into active service since the break occurred. In the next 48 hours, Interoute aims to have 75 to 80 per cent of the total unprotected services back up and running.
"None of Interoute's protected services, e.g. Internet, corporate connectivity, hosting or video have been affected. We apologise for any inconvenience caused by this cable break."
COMMENTS
Okay, tell me that it's not just me.
The point of running huge data centres with failover hardware, and the point of IP routing is basically so that, should something happen, the dead bits die and everyone is automatically shifted onto the non-dead bits.
Assuming that the capacity of your average ISP isn't being constantly tested 24/7 on all of its lines (which kinda makes the word "redundancy" not applicable), why should any ISP have to take any physical action for this kind of thing to be noticed, monitored, and then failed-over to other routes? I can understand warnings, and notifications, of it happening but why does it require manual intervention? Why are there half-a-dozen routing protocols professing automatic best-route selection if nobody uses them? Why can I set up a simple fail-over over several routes in about two lines of "ip" commands and yet big ISP's, datacenters and network providers can't do the equivalent on their expensive Cisco equipment?
I completely understand "capacity problems because we're running on one-line-less than normal" but "takes 30 hours for someone to work out what to do and find another hole to plug the cable into?".
Do we really have an Internet where manual intervention is required to provide the routing that's required when something fails?
neither had responded at time of publication
> El Reg has asked O2 and Interoute to comment on this story, but neither had responded at time of publication. We'll update the piece if O2 or Interoute do furnish us with statements.
They're not receiving emails at the moment.
IT infrastructure monitoring strategies
Agentless Backup is Not a Myth
Steps to Take Before Choosing a Business Continuity Partner
Requirements Checklist for Choosing a Cloud Backup and Recovery Service Provider
Data control in the cloud
