SaaS data loss: The problem you didn’t know you had

Another large cloud is on the fritz.

Following last month's much-discussed Amazon S3 outage, most (if not all) of XCalibre's FlexiScale cloud went dark on Tuesday, and nearly two days later, the UK-based hosting outfit has yet to restore service.

Lucas won't say how many cloud-happy outfits depend on his cloud, which went live in October. But he expects some of them will be back up and running this evening UK time. "And other customers will start coming back online from there on," he says. "But we're not sure how long that will take."

As Lucas explained in an email to customers - posted to the Web by CNet - the outage occurred when an XCalibre engineer accidentally deleted one of FlexiScale's main storage volumes. "The problem was caused by human error," Lucas told us. "We've been having some capacity issues - FlexiScale has been growing at about 30 per cent a month in terms of usage. We've been adding capacity and adding capacity and we were in the process of adding even more, when one of the engineers who was tidying things up on the disk architecture made a mistake."

XCalibre immediately took its entire disk structure offline - in order to "preserve the integrity of the data on the system." And with storage down, FlexiScale's processing and networking services are dark as well. But the platform should be returned to normal, once engineers restore the lost data to a brand new disk structure.

"After consulting with our storage vendor it was agreed the most sensible option would be to copy the entire volume to a new disk structure (still maintaining its integrity and structure), from where we could re-mount it correctly," Lucas' email reads. "Unfortunately, due to its size, we didn't have spare capacity on the platform to create a complete duplicate of it."

To Lucas' credit, he makes no excuses for outage - and hasn't tried to hide it. "This shouldn't have happened, shouldn't have been allowed to happen, but it happened. We'll be putting things into place that will prevent this from happening in the future."

Specifically, XCalibre will soon distribute its architecture across multiple data centers. "So, if something like this were to happen again, customers could fall over to an other data centers," Lucas says. But a second data center isn't due to open until January.

Yes, Lucas says, it would have been nice to have a second data center in place back in October, but, well, funds were tight. "We were a privately funded company," he says. "We were resource constrained. We will complete another round of funding in the next few weeks."

And Lucas is confident that cloudy outfits like his will eventually iron out all these issues. "One of my pet projects is interoperability and portability between cloud computing devices - basically, the ability to move from one to another if something goes wrong," he says. "But I didn't think I'd make such a good example of why that's important."

So, XCalibre's problems could be solved if Google would just make "a more loving cloud." ®

Steps to Take Before Choosing a Business Continuity Partner