Feeds

Hundreds of websites go titsup in Prime Hosting disk meltdown

UK biz brought up servers using months-old backups

Bridging the IT gap between rising business demands and ageing tools

Hundreds of UK-hosted websites and email accounts fell offline when a disk array failed at web biz Prime Hosting. As many as 860 customers are still waiting for a fix more than 48 hours after the storage unit went titsup.

The downtime at the Manchester-based hosting reseller began at 5am on 31 July, and two days later some sites are still down.

James Smith, executive director at parent company M247, explained that the fault lay with the hardware: "The problems were caused by the failure of three hard drives in a RAID6 array, therefore losing parity."

After one blew, the rebuild process staggered into problems, he said:

Attempts were made to rebuild the array due to one failed drive, two other drives then started to exhibit a high rate of media errors during the array rebuild.

The rebuild process failed at 15 per cent complete, a few minutes after the rebuild failed a second hard drive went in to a “missing” state and would not rejoin the array.

At this point, the array was severely degraded and could not tolerate any further failures. Unfortunately the third drive with high media errors then also went in to a failed state during efforts to take a more recent replicated copy of data. The replication of data had 790GB of data to sync, it managed 150GB before failing.

Several customers were upset to find their restored websites had reverted to a backup that was months old. A Reg reader said his site, email and databases had slipped back to a version from three months ago.

Smith assured The Reg that data had not been lost, and this resurrection of old files was just part of the fix process. The outdated data was used to speed up efforts to bring the virtual machines hosting services back online, he explained:

Efforts then started to bring virtual machines online, initially using the outdated replicated data. Services were restored quickly with outdated replicated data from secondary storage area network.

Using the outdated replicated data enabled us to bring service online quicker than individual account restores from archived backups. More recent individual account backups are currently being restored to bring customer data completely up to date.

Outage at Prime Hosting tweets, credit screengrab Twitter

Prime Hosting report outage on Twitter: Reversions to backups from April were only temporary

Prime Hosting apologised to punters, and said that it has had a team working solidly for 36 hours without sleep in order to minimise the impact. It also promised to compensate all affected customers with one month's service credit.

Smith emphasised that although Prime Hosting operates under the M247 banner, the problem solely affected Prime Hosting customers and sprang from hardware managed by Prime Hosting. ®

The Essential Guide to IT Transformation

More from The Register

next story
Scotland's BIG question: Will independence cost me my broadband?
They can take our lives, but they'll never take our SPECTRUM
Auntie remains MYSTIFIED by that weekend BBC iPlayer and website outage
Still doing 'forensics' on the caching layer – Beeb digi wonk
Bring back error correction, say Danish 'net boffins
We don't need no steenkin' TCP/IP retransmission and the congestion it causes
NBN Co adds apartments to FTTP rollout
Commercial trial locations to go live in September
GoTenna: How does this 'magic' work?
An ideal product if you believe the Earth is flat
Samsung Z Tizen OS mobe is post-phoned – this time for good?
Russian launch for Sammy's non-droid knocked back
Telstra to KILL 2G network by end of 2016
GSM now stands for Grave-Seeking-Mobile network
Seeking LTE expert to insert small cells into BT customers' places
Is this the first step to a FON-a-like 4G network?
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Securing Web Applications Made Simple and Scalable
Learn how automated security testing can provide a simple and scalable way to protect your web applications.