Telecity fix nixed: Borked UK internet hub 'had no UPS protection'

Telecity cancelled its plans for a second shot at fixing the stricken power supply in its Docklands internet hub at Sovereign House.

Engineers for the data centre operator were due to try fixing the broken system again on Wednesday night/Thursday morning, sources told The Register.

However, The Reg has learned, these plans were then called off. Telecity told customers yesterday there would be no intrusive work last night.

The second effort followed an earlier attempt between 1-3am GMT on Wednesday November 18 that produced a second power cut. Instead, Telecity – whose Sovereign House appears to handle 10 per cent of the UK's internet traffic, judging by the dip in traffic recorded by LINX during the original outage – has drawn up a plan to fully restore the UPS protection of the power feeds.

That plan, however, can’t be implemented immediately because it depends on the delivery of new parts that must be tested before being installed. In the meantime, customers' systems are running on mains power without the protection of UPS.

The lack of UPS coverage leaves them vulnerable to power spikes and any losses in mains power.

Customers with technical expertise and scale have re-routed traffic that would have normally gone through Sovereign House to other datacenter facilities. Those lacking the skills or size to make such changes, many being small and mid-market firms, are stuck – and potentially vulnerable.

Ideally, Sovereign House should be drawing power from something like a dedicated diesel generator set to mitigate against any potential loss from the grid. However, it seems, Telecity has not taken this step.

The company, meanwhile, is beginning to draw fire over the failure of its backup – power was interrupted, knocking off services on Tuesday at about 2pm. The outage choked VoIP and web hosting services, as well as parts of AWS whose traffic was routed via Sovereign House.

One customer made it clear they would expect answers from Telecity about why its additional power feeds failed to protect their equipment. A source told The Reg customers had paid Telecity for the use of dual, fully independent power suppliers to avoid outages but the fact their service was down indicated they hadn't actually got what they'd paid for.

El Reg could not reach Telecity for comment despite repeated phonecalls and emails.

Are you or your firm affected by the Telecity outage? Contact us: news@theregister.co.uk or phone the London newsdesk on +44 (0)20 3770 3147. ®