BT wholesale kit crash silences thousands of customers
In Slough, no one can hear you scream
Magic Quadrant for Enterprise Backup/Recovery
A hardware failure downed thousands of punters connected to BT's network via some of its smaller Openreach providers on Wednesday evening.
Reports via Plusnet, AAISP.net and others suggested that the outage affected many of their customers for around half a day.
Initially a major fibre cut near Slough was blamed, but it was later reported that a hardware fault with a 21CN router was the culprit.
"We still have a substantial number of broadband subscribers who were disconnected around 5.30pm this evening and who are still unable to get back online," wrote Plusnet in one of a series of updates about the outage on its website on Wednesday night (30 March).
"Some ambiguity has emerged as to the true cause of the problem, which may not actually be due to a fibre break in Slough as previously reported.
"One thing we are confident about though, is the fact that the problem is limited to a subset of customers provisioned on BT's IPStream Connect or WBC wholesale networks."
Plusnet recently shifted an unnamed number of its subscribers over to those networks, and as a result many of its customers were hit by the downtime.
BT confirmed to Plusnet that it had fixed the problem in the early hours of Thursday morning (31 March). However, it didn't offer any further detail on what had gone wrong.
The Register asked the company to tell us why so many of BT's Openreach providers' punters lost connection to its networks. At time of writing, it hadn't got back to us with a response.
In not entirely unrelated news, the outage occurred on the eve of Ofcom once again calling on BT to make price cuts for ISPs signed up to its Openreach service. ®
COMMENTS
The same mistake time and again
"What I fail to understand is how one failure caused a complete loss of service for some customers"
For *some* customers. i.e. the entire network didn't fail, like a good network is designed to do. That's not an indication of a single point of failure.
The same thing comes up time and time again whenever there is any sort of outage - there are levels of resilience. The rest of BTs network carried on as normal, the Internet carried on as normal. If you want a five or six 9's level of service, you need to be paying more than £9.99 a month for it.
what a night for my first night shift!
The outage went as far as bristol on some networks and drown to brighton, we had customers all over the south whose private links went down with this outage.
Supposedly a faulty 21cn router in the docklands caused all this mess.
bt reported it fixed at 1.15am but it was flakey at best til 3.30
Around 8hours of downtime after all is said and done.
im starting to realise a&a were pretty much on the money with their previous comments about 21cn, they need to get SOME resilience in the network if one faulty router can cause problems as far wide from brighton to slough and bournemouth to bristol, thats surely one he'll of a design flaw?
I know it's complicated but
I know it's complicated but it would be helpful if Kelly's article showed some evidence of clue about the different roles played in this picture by the allegedly independent parts of BT.
Grossly oversimplified:
BT Openreach: provider of connection between punter and BTwholesale kit in the exchange (or indeed other providers' kit in the exchange, but this picture relates to BTwholesale).
BT Wholesale: historically incompetent provider of connectivity between exchange kit and ISP kit, traditionally using badly designed and/or undersized networks and charging outrageous wholesale prices to "smaller" ISPs, whilst offering a different product family to parent BT Retail.
BT Retail: overpriced and underperforming broadband provider, default choice of the ill-informed.
BT Sheffield: Plusnet, a diversionary "smaller ISP" which allows BT to undercut other BTwholesale-based ISPs whilst still raking in the money from the ill-informed on BT-branded tariffs and while Ofcon stand idly by.
IT infrastructure monitoring strategies
Agentless Backup is Not a Myth
Steps to Take Before Choosing a Business Continuity Partner
Requirements Checklist for Choosing a Cloud Backup and Recovery Service Provider
Data control in the cloud
