Feeds

Hundreds of websites go titsup in Prime Hosting disk meltdown

UK biz brought up servers using months-old backups

Eight steps to building an HP BladeSystem

Hundreds of UK-hosted websites and email accounts fell offline when a disk array failed at web biz Prime Hosting. As many as 860 customers are still waiting for a fix more than 48 hours after the storage unit went titsup.

The downtime at the Manchester-based hosting reseller began at 5am on 31 July, and two days later some sites are still down.

James Smith, executive director at parent company M247, explained that the fault lay with the hardware: "The problems were caused by the failure of three hard drives in a RAID6 array, therefore losing parity."

After one blew, the rebuild process staggered into problems, he said:

Attempts were made to rebuild the array due to one failed drive, two other drives then started to exhibit a high rate of media errors during the array rebuild.

The rebuild process failed at 15 per cent complete, a few minutes after the rebuild failed a second hard drive went in to a “missing” state and would not rejoin the array.

At this point, the array was severely degraded and could not tolerate any further failures. Unfortunately the third drive with high media errors then also went in to a failed state during efforts to take a more recent replicated copy of data. The replication of data had 790GB of data to sync, it managed 150GB before failing.

Several customers were upset to find their restored websites had reverted to a backup that was months old. A Reg reader said his site, email and databases had slipped back to a version from three months ago.

Smith assured The Reg that data had not been lost, and this resurrection of old files was just part of the fix process. The outdated data was used to speed up efforts to bring the virtual machines hosting services back online, he explained:

Efforts then started to bring virtual machines online, initially using the outdated replicated data. Services were restored quickly with outdated replicated data from secondary storage area network.

Using the outdated replicated data enabled us to bring service online quicker than individual account restores from archived backups. More recent individual account backups are currently being restored to bring customer data completely up to date.

Outage at Prime Hosting tweets, credit screengrab Twitter

Prime Hosting report outage on Twitter: Reversions to backups from April were only temporary

Prime Hosting apologised to punters, and said that it has had a team working solidly for 36 hours without sleep in order to minimise the impact. It also promised to compensate all affected customers with one month's service credit.

Smith emphasised that although Prime Hosting operates under the M247 banner, the problem solely affected Prime Hosting customers and sprang from hardware managed by Prime Hosting. ®

The Power of One Brief: Top reasons to choose HP BladeSystem

More from The Register

next story
Google Nest, ARM, Samsung pull out Thread to strangle ZigBee
But there's a flaw in Google's IP-based IoT system
Orange spent weekend spamming customers with TXTs
Zero, not infinity, is the Magic Number customers want
Want to beat Verizon's slow Netflix? Get a VPN
Exec finds stream speed climbs when smuggled out
US freemium mobile network eyes up Europe
FreedomPop touts 'free' calls, texts and data
'Two-speed internet' storm turns FCC.gov into zero-speed website
Deadline for comments on net neutrality shake-up extended to Friday
GoTenna: How does this 'magic' work?
An ideal product if you believe the Earth is flat
NBN Co execs: No FTTN product until 2015
Faster? Not yet. Cheaper? No data
prev story

Whitepapers

Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.