Feeds

Engineer accidentally deletes cloud

Off-demand computing - First Amazon, now FlexiScale

Protecting against web application threats using SSL

Another large cloud is on the fritz.

Following last month's much-discussed Amazon S3 outage, most (if not all) of XCalibre's FlexiScale cloud went dark on Tuesday, and nearly two days later, the UK-based hosting outfit has yet to restore service.

According to XCalibre CEO Tony Lucas, the outage has affected "a vast majority" of businesses relying on FlexiScale for on-demand storage, processing, and/or network bandwidth.

Lucas won't say how many cloud-happy outfits depend on his cloud, which went live in October. But he expects some of them will be back up and running this evening UK time. "And other customers will start coming back online from there on," he says. "But we're not sure how long that will take."

As Lucas explained in an email to customers - posted to the Web by CNet - the outage occurred when an XCalibre engineer accidentally deleted one of FlexiScale's main storage volumes. "The problem was caused by human error," Lucas told us. "We've been having some capacity issues - FlexiScale has been growing at about 30 per cent a month in terms of usage. We've been adding capacity and adding capacity and we were in the process of adding even more, when one of the engineers who was tidying things up on the disk architecture made a mistake."

XCalibre immediately took its entire disk structure offline - in order to "preserve the integrity of the data on the system." And with storage down, FlexiScale's processing and networking services are dark as well. But the platform should be returned to normal, once engineers restore the lost data to a brand new disk structure.

"After consulting with our storage vendor it was agreed the most sensible option would be to copy the entire volume to a new disk structure (still maintaining its integrity and structure), from where we could re-mount it correctly," Lucas' email reads. "Unfortunately, due to its size, we didn't have spare capacity on the platform to create a complete duplicate of it."

To Lucas' credit, he makes no excuses for outage - and hasn't tried to hide it. "This shouldn't have happened, shouldn't have been allowed to happen, but it happened. We'll be putting things into place that will prevent this from happening in the future."

Specifically, XCalibre will soon distribute its architecture across multiple data centers. "So, if something like this were to happen again, customers could fall over to an other data centers," Lucas says. But a second data center isn't due to open until January.

Yes, Lucas says, it would have been nice to have a second data center in place back in October, but, well, funds were tight. "We were a privately funded company," he says. "We were resource constrained. We will complete another round of funding in the next few weeks."

And Lucas is confident that cloudy outfits like his will eventually iron out all these issues. "One of my pet projects is interoperability and portability between cloud computing devices - basically, the ability to move from one to another if something goes wrong," he says. "But I didn't think I'd make such a good example of why that's important."

So, XCalibre's problems could be solved if Google would just make "a more loving cloud." ®

Choosing a cloud hosting partner with confidence

More from The Register

next story
Wanna keep your data for 1,000 YEARS? No? Hard luck, HDS wants you to anyway
Combine Blu-ray and M-DISC and you get this monster
Google+ GOING, GOING ... ? Newbie Gmailers no longer forced into mandatory ID slurp
Mountain View distances itself from lame 'network thingy'
US boffins demo 'twisted radio' mux
OAM takes wireless signals to 32 Gbps
Apple flops out 2FA for iCloud in bid to stop future nude selfie leaks
Millions of 4chan users howl with laughter as Cupertino slams stable door
Students playing with impressive racks? Yes, it's cluster comp time
The most comprehensive coverage the world has ever seen. Ever
Run little spreadsheet, run! IBM's Watson is coming to gobble you up
Big Blue's big super's big appetite for big data in big clouds for big analytics
Seagate's triple-headed Cerberus could SAVE the DISK WORLD
... and possibly bring us even more HAMR time. Yay!
prev story

Whitepapers

Secure remote control for conventional and virtual desktops
Balancing user privacy and privileged access, in accordance with compliance frameworks and legislation. Evaluating any potential remote control choice.
WIN a very cool portable ZX Spectrum
Win a one-off portable Spectrum built by legendary hardware hacker Ben Heck
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
The next step in data security
With recent increased privacy concerns and computers becoming more powerful, the chance of hackers being able to crack smaller-sized RSA keys increases.