Feeds

Engineer accidentally deletes cloud

Off-demand computing - First Amazon, now FlexiScale

Beginner's guide to SSL certificates

Another large cloud is on the fritz.

Following last month's much-discussed Amazon S3 outage, most (if not all) of XCalibre's FlexiScale cloud went dark on Tuesday, and nearly two days later, the UK-based hosting outfit has yet to restore service.

According to XCalibre CEO Tony Lucas, the outage has affected "a vast majority" of businesses relying on FlexiScale for on-demand storage, processing, and/or network bandwidth.

Lucas won't say how many cloud-happy outfits depend on his cloud, which went live in October. But he expects some of them will be back up and running this evening UK time. "And other customers will start coming back online from there on," he says. "But we're not sure how long that will take."

As Lucas explained in an email to customers - posted to the Web by CNet - the outage occurred when an XCalibre engineer accidentally deleted one of FlexiScale's main storage volumes. "The problem was caused by human error," Lucas told us. "We've been having some capacity issues - FlexiScale has been growing at about 30 per cent a month in terms of usage. We've been adding capacity and adding capacity and we were in the process of adding even more, when one of the engineers who was tidying things up on the disk architecture made a mistake."

XCalibre immediately took its entire disk structure offline - in order to "preserve the integrity of the data on the system." And with storage down, FlexiScale's processing and networking services are dark as well. But the platform should be returned to normal, once engineers restore the lost data to a brand new disk structure.

"After consulting with our storage vendor it was agreed the most sensible option would be to copy the entire volume to a new disk structure (still maintaining its integrity and structure), from where we could re-mount it correctly," Lucas' email reads. "Unfortunately, due to its size, we didn't have spare capacity on the platform to create a complete duplicate of it."

To Lucas' credit, he makes no excuses for outage - and hasn't tried to hide it. "This shouldn't have happened, shouldn't have been allowed to happen, but it happened. We'll be putting things into place that will prevent this from happening in the future."

Specifically, XCalibre will soon distribute its architecture across multiple data centers. "So, if something like this were to happen again, customers could fall over to an other data centers," Lucas says. But a second data center isn't due to open until January.

Yes, Lucas says, it would have been nice to have a second data center in place back in October, but, well, funds were tight. "We were a privately funded company," he says. "We were resource constrained. We will complete another round of funding in the next few weeks."

And Lucas is confident that cloudy outfits like his will eventually iron out all these issues. "One of my pet projects is interoperability and portability between cloud computing devices - basically, the ability to move from one to another if something goes wrong," he says. "But I didn't think I'd make such a good example of why that's important."

So, XCalibre's problems could be solved if Google would just make "a more loving cloud." ®

Beginner's guide to SSL certificates

More from The Register

next story
Azure TITSUP caused by INFINITE LOOP
Fat fingered geo-block kept Aussies in the dark
NASA launches new climate model at SC14
75 days of supercomputing later ...
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
You think the CLOUD's insecure? It's BETTER than UK.GOV's DATA CENTRES
We don't even know where some of them ARE – Maude
DEATH by COMMENTS: WordPress XSS vuln is BIGGEST for YEARS
Trio of XSS turns attackers into admins
Cloud unicorns are extinct so DiData cloud mess was YOUR fault
Applications need to be built to handle TITSUP incidents
BOFH: WHERE did this 'fax-enabled' printer UPGRADE come from?
Don't worry about that cable, it's part of the config
Astro-boffins start opening universe simulation data
Got a supercomputer? Want to simulate a universe? Here you go
prev story

Whitepapers

Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
Getting started with customer-focused identity management
Learn why identity is a fundamental requirement to digital growth, and how without it there is no way to identify and engage customers in a meaningful way.
How to determine if cloud backup is right for your servers
Two key factors, technical feasibility and TCO economics, that backup and IT operations managers should consider when assessing cloud backup.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Internet Security Threat Report 2014
An overview and analysis of the year in global threat activity: identify, analyze, and provide commentary on emerging trends in the dynamic threat landscape.