Feeds

Engineer accidentally deletes cloud

Off-demand computing - First Amazon, now FlexiScale

Intelligent flash storage arrays

Another large cloud is on the fritz.

Following last month's much-discussed Amazon S3 outage, most (if not all) of XCalibre's FlexiScale cloud went dark on Tuesday, and nearly two days later, the UK-based hosting outfit has yet to restore service.

According to XCalibre CEO Tony Lucas, the outage has affected "a vast majority" of businesses relying on FlexiScale for on-demand storage, processing, and/or network bandwidth.

Lucas won't say how many cloud-happy outfits depend on his cloud, which went live in October. But he expects some of them will be back up and running this evening UK time. "And other customers will start coming back online from there on," he says. "But we're not sure how long that will take."

As Lucas explained in an email to customers - posted to the Web by CNet - the outage occurred when an XCalibre engineer accidentally deleted one of FlexiScale's main storage volumes. "The problem was caused by human error," Lucas told us. "We've been having some capacity issues - FlexiScale has been growing at about 30 per cent a month in terms of usage. We've been adding capacity and adding capacity and we were in the process of adding even more, when one of the engineers who was tidying things up on the disk architecture made a mistake."

XCalibre immediately took its entire disk structure offline - in order to "preserve the integrity of the data on the system." And with storage down, FlexiScale's processing and networking services are dark as well. But the platform should be returned to normal, once engineers restore the lost data to a brand new disk structure.

"After consulting with our storage vendor it was agreed the most sensible option would be to copy the entire volume to a new disk structure (still maintaining its integrity and structure), from where we could re-mount it correctly," Lucas' email reads. "Unfortunately, due to its size, we didn't have spare capacity on the platform to create a complete duplicate of it."

To Lucas' credit, he makes no excuses for outage - and hasn't tried to hide it. "This shouldn't have happened, shouldn't have been allowed to happen, but it happened. We'll be putting things into place that will prevent this from happening in the future."

Specifically, XCalibre will soon distribute its architecture across multiple data centers. "So, if something like this were to happen again, customers could fall over to an other data centers," Lucas says. But a second data center isn't due to open until January.

Yes, Lucas says, it would have been nice to have a second data center in place back in October, but, well, funds were tight. "We were a privately funded company," he says. "We were resource constrained. We will complete another round of funding in the next few weeks."

And Lucas is confident that cloudy outfits like his will eventually iron out all these issues. "One of my pet projects is interoperability and portability between cloud computing devices - basically, the ability to move from one to another if something goes wrong," he says. "But I didn't think I'd make such a good example of why that's important."

So, XCalibre's problems could be solved if Google would just make "a more loving cloud." ®

Secure remote control for conventional and virtual desktops

More from The Register

next story
BOFH: WHERE did this 'fax-enabled' printer UPGRADE come from?
Don't worry about that cable, it's part of the config
Azure TITSUP caused by INFINITE LOOP
Fat fingered geo-block kept Aussies in the dark
You think the CLOUD's insecure? It's BETTER than UK.GOV's DATA CENTRES
We don't even know where some of them ARE – Maude
Want to STUFF Facebook with blatant ADVERTISING? Fine! But you must PAY
Pony up or push off, Zuck tells social marketeers
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
Oi, Europe! Tell US feds to GTFO of our servers, say Microsoft and pals
By writing a really angry letter about how it's harming our cloud business, ta
prev story

Whitepapers

Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
Getting started with customer-focused identity management
Learn why identity is a fundamental requirement to digital growth, and how without it there is no way to identify and engage customers in a meaningful way.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Simplify SSL certificate management across the enterprise
Simple steps to take control of SSL across the enterprise, and recommendations for a management platform for full visibility and single-point of control for these Certificates.
Top 5 reasons to deploy VMware with Tegile
Data demand and the rise of virtualization is challenging IT teams to deliver storage performance, scalability and capacity that can keep up, while maximizing efficiency.