Heart Internet in 22-hour TITSUP after data centre power stuffup
One of the 'most efficient and resilient data centres in Europe'
Nottingham-based web hosting company Heart Internet has gone TITSUP* due to a power outage at what it claims is "one of the most efficient and resilient data centres in Europe".
Aggrieved customers have been in touch with The Register to claim Heart had initially blamed the issue on a DDoS attack, before the company acknowledged there had been a power issue. It has been down for more than 22 hours at the time of publication.
A blog post from the business admitted that "On Wednesday 10th February, we suffered an interruption in power to our data centre facility during emergency works to fix a fault. This has caused disruption to customers’ services and our teams are working to resolve remaining issues."
The post continued, stating that the company wanted to be "open and transparent with our customers and give you the causes of the fault", and then immediately claimed: "Our UK-based data centre facility is one of the most efficient and resilient data centres in Europe."
Apparently, a nine-minute power loss affected a single data centre hall which had two UPSes feeding into a load transfer module, which showed a fault on the primary power supply and was running on its backup.
The business claims its teams "followed the guidelines and contacted the manufacturer and an expert in this piece of hardware was sent to our facility to investigate and fix" the issue.
As the fault code indicated an issue with the voltage monitor, it was recognised that primary power had been automatically shut down "even though the power supply itself is likely fine."
The engineer on site then "assisted with the fitting of the replacement part" and "a safety mechanism in the device triggered incorrectly," taking down the entire data centre. Who was actually responsible, however, the company did not disclose.
One Heart customer told The Register: "I cannot even log in to report problems! One customer in Derby and Leicester has lost all email since yesterday for about 30 staff."
Another testified that they had 12 small businesses whose websites and emails were host through Heart: "They have always been faultless in the past, but nearly 24 hours to recover a data centre is unacceptable. For our own company emails we have had to fall back on Gmail, but our clients are, not surprisingly, unimpressed. Many of them are not IT-literate, so they don't care where the fault lies - by default it lies with us."
"Currently they are in a shambles. No tickets being answered, no way to call support at all. Our own ecommerce site has been titsup since yesterday 2:30 –ish and we are losing money," another customer told us.
A Heart spokesman told The Register: "The majority of customers are up and running already and we are working as hard as possible to ensure that the remaining customers’ websites are back as soon as possible. Heart Internet will take care of customers and reimburse them fairly where appropriate. Our immediate focus is to get the few customers who don’t yet have connectivity back online."
The spokesman confirmed the DDoS attack but said it only affected "a few servers at the time and further investigations revealed it was not the main cause of the outage." She added: "We removed the mention of it from Heart Internet’s status page to avoid any possible confusion for customers."
For those affected, Heart's System Status Page is here. ®
* Suffered a Total Inability To Support Usual Performance