Fire safety gaffe knocks out Webfusion data centre
False alarm triggers mass server shutdown
Steps to Take Before Choosing a Business Continuity Partner
A bungling fire safety contractor caused a complete shutdown at hosting firm Webfusion's data centre today, crippling thousands of websites.
The firm's domain name arm, 123-Reg, was also temporarily offline.
"An external third party was carrying out routine maintenance in our data centre, and testing our systems for fire prevention," Webfusion said in a system status update.
"Unfortunately, due to human error, our fire prevention systems were in fact triggered.
"As a result of this, and acting as the system should in the event of a real fire, all of our servers were sent in to a safe mode whereby they went offline.
"Safety is our biggest concern, hence the system is configured to react in this way to avoid a major incident and permanent data loss."
The firm's £2.5m Leeds data centre, designed by IBM, opened in February last year. It disappeared from the internet at around 12pm, and customers are still reporting connectivity problems.
Webfusion said: "We deeply regret any problems this may have caused you, and assure you we are doing our utmost to return to normal service levels as quickly as we possibly can." ®
Update 4.40pm
Webfusion got in touch to say everything is now up and running again.
COMMENTS
Disaster tolerance? Transparent alternate site failover?
I suppose not, doing things properly is so unfashionable these days.
Security guards can also shut down data centers
At a previous job, a security guard (who was an untrained rent a cop from some third world country) tried to silence an alarm on an exit door -- by pressing the emergency power kill button next to the door. It was a large, red, well labeled button under a clear plastic cover; and it shut down all of the power in the data center except for the emergency lights.
The security guard couldn't read the sign that said "EMERGENCY POWER SHUT OFF" because it was in English rather than whatever third world language he spoke.
Moon on a stick
So you want failover and DR and all that malarky, but you're going to a budget hosting provider like Webfusion?
If you're with them, accept the fact that you *don't* have these services available - certainly not without additional spend.
In my experience it's the ones who spend the least that complain the most - the people who were spending a proper sum for decent service in the first place already had and subsequently enjoyed the protection their spend gave them.
IT infrastructure monitoring strategies
Requirements Checklist for Choosing a Cloud Backup and Recovery Service Provider
Cloud based data management
Enabling efficient data center monitoring
Agentless Backup is Not a Myth
