Feeds

Hundreds of websites go titsup in Prime Hosting disk meltdown

UK biz brought up servers using months-old backups

Build a business case: developing custom apps

Hundreds of UK-hosted websites and email accounts fell offline when a disk array failed at web biz Prime Hosting. As many as 860 customers are still waiting for a fix more than 48 hours after the storage unit went titsup.

The downtime at the Manchester-based hosting reseller began at 5am on 31 July, and two days later some sites are still down.

James Smith, executive director at parent company M247, explained that the fault lay with the hardware: "The problems were caused by the failure of three hard drives in a RAID6 array, therefore losing parity."

After one blew, the rebuild process staggered into problems, he said:

Attempts were made to rebuild the array due to one failed drive, two other drives then started to exhibit a high rate of media errors during the array rebuild.

The rebuild process failed at 15 per cent complete, a few minutes after the rebuild failed a second hard drive went in to a “missing” state and would not rejoin the array.

At this point, the array was severely degraded and could not tolerate any further failures. Unfortunately the third drive with high media errors then also went in to a failed state during efforts to take a more recent replicated copy of data. The replication of data had 790GB of data to sync, it managed 150GB before failing.

Several customers were upset to find their restored websites had reverted to a backup that was months old. A Reg reader said his site, email and databases had slipped back to a version from three months ago.

Smith assured The Reg that data had not been lost, and this resurrection of old files was just part of the fix process. The outdated data was used to speed up efforts to bring the virtual machines hosting services back online, he explained:

Efforts then started to bring virtual machines online, initially using the outdated replicated data. Services were restored quickly with outdated replicated data from secondary storage area network.

Using the outdated replicated data enabled us to bring service online quicker than individual account restores from archived backups. More recent individual account backups are currently being restored to bring customer data completely up to date.

Outage at Prime Hosting tweets, credit screengrab Twitter

Prime Hosting report outage on Twitter: Reversions to backups from April were only temporary

Prime Hosting apologised to punters, and said that it has had a team working solidly for 36 hours without sleep in order to minimise the impact. It also promised to compensate all affected customers with one month's service credit.

Smith emphasised that although Prime Hosting operates under the M247 banner, the problem solely affected Prime Hosting customers and sprang from hardware managed by Prime Hosting. ®

Secure remote control for conventional and virtual desktops

More from The Register

next story
UK fuzz want PINCODES on ALL mobile phones
Met Police calls for mandatory passwords on all new mobes
Canadian ISP Shaw falls over with 'routing' sickness
How sure are you of cloud computing now?
Don't call it throttling: Ericsson 'priority' tech gives users their own slice of spectrum
Actually it's a nifty trick - at least you'll pay for what you get
Three floats Jolla in Hong Kong: Says Sailfish is '3rd option'
Network throws hat into ring with Linux-powered handsets
Fifteen zero days found in hacker router comp romp
Four routers rooted in SOHOpelessly Broken challenge
New Sprint CEO says he will lower axe on staff – but prices come first
'Very disruptive' new rates to be revealed next week
PwC says US biz lagging in Internet of Things
Grass is greener in Asia, say the sensors
Ofcom sees RISE OF THE MACHINE-to-machine cell comms
Study spots 9% growth in IoT m2m mobile data connections
O2 vs Vodafone: Mobe firms grab for GCHQ, gov.uk security badge
No, the spooks love US best, say rival firms
Ancient pager tech SMS: It works, it's fab, but wow, get a load of that incoming SPAM
Networks' main issue: they don't know how it works, says expert
prev story

Whitepapers

Endpoint data privacy in the cloud is easier than you think
Innovations in encryption and storage resolve issues of data privacy and key requirements for companies to look for in a solution.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Solving today's distributed Big Data backup challenges
Enable IT efficiency and allow a firm to access and reuse corporate information for competitive advantage, ultimately changing business outcomes.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.