Feeds

Hundreds of websites go titsup in Prime Hosting disk meltdown

UK biz brought up servers using months-old backups

3 Big data security analytics techniques

Hundreds of UK-hosted websites and email accounts fell offline when a disk array failed at web biz Prime Hosting. As many as 860 customers are still waiting for a fix more than 48 hours after the storage unit went titsup.

The downtime at the Manchester-based hosting reseller began at 5am on 31 July, and two days later some sites are still down.

James Smith, executive director at parent company M247, explained that the fault lay with the hardware: "The problems were caused by the failure of three hard drives in a RAID6 array, therefore losing parity."

After one blew, the rebuild process staggered into problems, he said:

Attempts were made to rebuild the array due to one failed drive, two other drives then started to exhibit a high rate of media errors during the array rebuild.

The rebuild process failed at 15 per cent complete, a few minutes after the rebuild failed a second hard drive went in to a “missing” state and would not rejoin the array.

At this point, the array was severely degraded and could not tolerate any further failures. Unfortunately the third drive with high media errors then also went in to a failed state during efforts to take a more recent replicated copy of data. The replication of data had 790GB of data to sync, it managed 150GB before failing.

Several customers were upset to find their restored websites had reverted to a backup that was months old. A Reg reader said his site, email and databases had slipped back to a version from three months ago.

Smith assured The Reg that data had not been lost, and this resurrection of old files was just part of the fix process. The outdated data was used to speed up efforts to bring the virtual machines hosting services back online, he explained:

Efforts then started to bring virtual machines online, initially using the outdated replicated data. Services were restored quickly with outdated replicated data from secondary storage area network.

Using the outdated replicated data enabled us to bring service online quicker than individual account restores from archived backups. More recent individual account backups are currently being restored to bring customer data completely up to date.

Outage at Prime Hosting tweets, credit screengrab Twitter

Prime Hosting report outage on Twitter: Reversions to backups from April were only temporary

Prime Hosting apologised to punters, and said that it has had a team working solidly for 36 hours without sleep in order to minimise the impact. It also promised to compensate all affected customers with one month's service credit.

Smith emphasised that although Prime Hosting operates under the M247 banner, the problem solely affected Prime Hosting customers and sprang from hardware managed by Prime Hosting. ®

3 Big data security analytics techniques

More from The Register

next story
A black box for your SUITCASE: Now your lost luggage can phone home – quite literally
Breakfast in London, lunch in NYC, and your clothes in Peru
Broadband Secretary of SHEEP sensationally quits Cabinet
Maria Miller finally resigns over expenses row
Skype pimps pro-level broadcast service
Playing Cat and Mouse with the media
EE dismisses DATA-BURNING glitch with Orange Mail app
Bug quietly slurps PAYG credit - yet EE denies it exists
Like Google, Comcast might roll its own mobile voice network
Says anything's possible if regulators approve merger with Time Warner
Turnbull leaves Australia's broadband blackspots in the dark
New Statement of Expectations to NBN Co offers get-out clauses for blackspot builds
Facebook claims 100 MEEELLION active users in India
Who needs China when you've got the next billion in your sights?
Facebook splats in-app chat, whacks brats into crack yakety-yak app
Jibber-jabbering addicts turfed out just as Zuck warned
prev story

Whitepapers

Designing a defence for mobile apps
In this whitepaper learn the various considerations for defending mobile applications; from the mobile application architecture itself to the myriad testing technologies needed to properly assess mobile applications risk.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.