UK web host 123-Reg goes TITSUP, customer servers evaporate
One customer says 'rogue script' deleted all sites
UK hosting and domains provider 123-Reg has been struck by a weekend TITSUP (Total Inability To Support Usual Performance) that knocked an unspecified number of VPS (virtual private server) customers offline.
The company posted a status message saying that the unspecified issues arose on April 16.
123-reg customer, software company INNmaster, contacted The Register directing our attention to its post on the topic, claiming a rogue script had deleted customer sites.
“At 7am on Saturday morning someone at our hosting provider ran a script that had a catastrophic error in it. The result was that the script deleted the servers and websites of all their customers!”
INNmaster's post says that company has implemented a disaster recovery plan, running up a backup server in a different location, but expresses sympathy for anybody without a proper backup.
The Register has contacted a support e-mail address at 123-reg to ask whether INNmaster's account of events is accurate.
However, more serious is the apparent silence from 123-reg. The company's Twitter account has stuck to variations on “we're working on the problem” since the outage began, with the last Tweet at the time of writing promising an e-mail to affected customers:
Update 5: Customers affected by VPS issues will be sent an email from 123-reg this afternoon. Further updates: https://t.co/37s8eWkhJ7— 123-reg (@123reg) April 17, 2016
We await further information. Other affected 123-Reg customers can feel free to contact us with their experiences, here. ®
Update: 123-Reg has responded via e-mail: "In relation to the issues affecting the performance of our VPS product over the weekend, we have been dealing with customers directly through our support centre and also senior level staff. We will continue to do so. Our number one priority is to return services to normal.
"We have been working with an extended team of experts who have been working into the night to restore as much as we can as quickly as possible.
123-reg would like to apologise to those customers who have been affected." ®
Sponsored: DevOps and continuous delivery