A hamfisted worker at colo provider Telecity shut down "several" ISPs and their customers across the country when he started pulling plugs at one of its datacentres late last night.

The hapless wire-scrapper then proceeded to make matters worse, by trying to fix the mess himself.

This company, which uses the Datahop ethernet extension service to link Redbus HEX to Telecity HEX, was alerted last night at 10:08pm when all its ADSL2+ customers vanished off the Internet. Alarmed night support staff contacted Telecity, which in turn contacted Datahop.

The report by AAISP on the support blog takes up the tale:

"The good news is we know what happened, and it will be sorted in a few minutes," wrote the engineer when things started to work, about two hours later. "We have also had a good chance to review the fallback arrangements and work out why they are not quite as they should be. That is something we will be going through tomorrow. However, this really is something we could not have expected to happen."

The "fault" wasn't so much a fault, apparently, as a complete farce, when someone working inside Telecity started pulling plugs.

AAISP's blogger contacted him, and reports: "The man from Telecity, who says 'sorry that was my fault', and who is described as an 'idiot' by the man from Datahop, managed to knock out at least 5 datahop ports, so affecting a number of ISPs."

Datahop staff then contacted Telecity and told them to make him stop, and banned him from touching anything.

It seems he didn't stop. He had several loose cables, so (as the blogger reports) "He then proceeded to plug them back in randomly. So, we have an active switch port that is not going where it should be." On a managed switch, the port matters.

So if you find your logs reporting a four-hour glitch last night into London data centres, or any data centres using Datahop to link to Telehouse or Telecity, that's what happened. ®

Related stories

• Pictures BT's Mayfair exchange downed by burglary (11 September 2008)
• Claranet blames data centre for downtime (4 August 2008)
• Police seek two for C&W network robbery (17 July 2008)
• BT opens wallet to send fibre to the home (15 July 2008)
• Updated Thieves take out Cable & Wireless centre (10 July 2008)
• Police launch hunt for bogus bobbies (7 December 2007)
• Updated Data centre looted in multimillion pound overnight heist (7 December 2007)
• Level 3 floored by robbery (1 November 2006)