'Idiot' pulls cables, downs ISPs at Telecity

A hamfisted worker at colo provider Telecity shut down "several" ISPs and their customers across the country when he started pulling plugs at one of its datacentres late last night.

The hapless wire-scrapper then proceeded to make matters worse, by trying to fix the mess himself.

El Reg hasn't been able to discover the identities of the other internet companies affected, but one ISP has blogged about the incident.

This company, which uses the Datahop ethernet extension service to link Redbus HEX to Telecity HEX, was alerted last night at 10:08pm when all its ADSL2+ customers vanished off the Internet. Alarmed night support staff contacted Telecity, which in turn contacted Datahop.

The report by AAISP on the support blog takes up the tale:

"The good news is we know what happened, and it will be sorted in a few minutes," wrote the engineer when things started to work, about two hours later. "We have also had a good chance to review the fallback arrangements and work out why they are not quite as they should be. That is something we will be going through tomorrow. However, this really is something we could not have expected to happen."

The "fault" wasn't so much a fault, apparently, as a complete farce, when someone working inside Telecity started pulling plugs.

AAISP's blogger contacted him, and reports: "The man from Telecity, who says 'sorry that was my fault', and who is described as an 'idiot' by the man from Datahop, managed to knock out at least 5 datahop ports, so affecting a number of ISPs."

Datahop staff then contacted Telecity and told them to make him stop, and banned him from touching anything.

It seems he didn't stop. He had several loose cables, so (as the blogger reports) "He then proceeded to plug them back in randomly. So, we have an active switch port that is not going where it should be." On a managed switch, the port matters.

So if you find your logs reporting a four-hour glitch last night into London data centres, or any data centres using Datahop to link to Telehouse or Telecity, that's what happened. ®