Feeds

Hundreds of websites go titsup in Prime Hosting disk meltdown

UK biz brought up servers using months-old backups

Top 5 reasons to deploy VMware with Tegile

Hundreds of UK-hosted websites and email accounts fell offline when a disk array failed at web biz Prime Hosting. As many as 860 customers are still waiting for a fix more than 48 hours after the storage unit went titsup.

The downtime at the Manchester-based hosting reseller began at 5am on 31 July, and two days later some sites are still down.

James Smith, executive director at parent company M247, explained that the fault lay with the hardware: "The problems were caused by the failure of three hard drives in a RAID6 array, therefore losing parity."

After one blew, the rebuild process staggered into problems, he said:

Attempts were made to rebuild the array due to one failed drive, two other drives then started to exhibit a high rate of media errors during the array rebuild.

The rebuild process failed at 15 per cent complete, a few minutes after the rebuild failed a second hard drive went in to a “missing” state and would not rejoin the array.

At this point, the array was severely degraded and could not tolerate any further failures. Unfortunately the third drive with high media errors then also went in to a failed state during efforts to take a more recent replicated copy of data. The replication of data had 790GB of data to sync, it managed 150GB before failing.

Several customers were upset to find their restored websites had reverted to a backup that was months old. A Reg reader said his site, email and databases had slipped back to a version from three months ago.

Smith assured The Reg that data had not been lost, and this resurrection of old files was just part of the fix process. The outdated data was used to speed up efforts to bring the virtual machines hosting services back online, he explained:

Efforts then started to bring virtual machines online, initially using the outdated replicated data. Services were restored quickly with outdated replicated data from secondary storage area network.

Using the outdated replicated data enabled us to bring service online quicker than individual account restores from archived backups. More recent individual account backups are currently being restored to bring customer data completely up to date.

Outage at Prime Hosting tweets, credit screengrab Twitter

Prime Hosting report outage on Twitter: Reversions to backups from April were only temporary

Prime Hosting apologised to punters, and said that it has had a team working solidly for 36 hours without sleep in order to minimise the impact. It also promised to compensate all affected customers with one month's service credit.

Smith emphasised that although Prime Hosting operates under the M247 banner, the problem solely affected Prime Hosting customers and sprang from hardware managed by Prime Hosting. ®

Secure remote control for conventional and virtual desktops

More from The Register

next story
Download alert: Nearly ALL top 100 Android, iOS paid apps hacked
Attack of the Clones? Yeah, but much, much scarier – report
Broadband sellers in the UK are UP TO no good, says Which?
Speedy network claims only apply to 10% of customers
Virgin Media struck dumb by NATIONWIDE packet loss balls-up
Turning it off and on again fixes glitch 12 HOURS LATER
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
Fujitsu CTO: We'll be 3D-printing tech execs in 15 years
Fleshy techie disses network neutrality, helmet-less motorcyclists
Facebook, working on Facebook at Work, works on Facebook. At Work
You don't want your cat or drunk pics at the office
Soz, web devs: Google snatches its Wallet off the table
Killing off web service in 3 months... but app-happy bonkers are fine
prev story

Whitepapers

Choosing cloud Backup services
Demystify how you can address your data protection needs in your small- to medium-sized business and select the best online backup service to meet your needs.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
10 threats to successful enterprise endpoint backup
10 threats to a successful backup including issues with BYOD, slow backups and ineffective security.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Internet Security Threat Report 2014
An overview and analysis of the year in global threat activity: identify, analyze, and provide commentary on emerging trends in the dynamic threat landscape.