Feeds

Engineer accidentally deletes cloud

Off-demand computing - First Amazon, now FlexiScale

Internet Security Threat Report 2014

Another large cloud is on the fritz.

Following last month's much-discussed Amazon S3 outage, most (if not all) of XCalibre's FlexiScale cloud went dark on Tuesday, and nearly two days later, the UK-based hosting outfit has yet to restore service.

According to XCalibre CEO Tony Lucas, the outage has affected "a vast majority" of businesses relying on FlexiScale for on-demand storage, processing, and/or network bandwidth.

Lucas won't say how many cloud-happy outfits depend on his cloud, which went live in October. But he expects some of them will be back up and running this evening UK time. "And other customers will start coming back online from there on," he says. "But we're not sure how long that will take."

As Lucas explained in an email to customers - posted to the Web by CNet - the outage occurred when an XCalibre engineer accidentally deleted one of FlexiScale's main storage volumes. "The problem was caused by human error," Lucas told us. "We've been having some capacity issues - FlexiScale has been growing at about 30 per cent a month in terms of usage. We've been adding capacity and adding capacity and we were in the process of adding even more, when one of the engineers who was tidying things up on the disk architecture made a mistake."

XCalibre immediately took its entire disk structure offline - in order to "preserve the integrity of the data on the system." And with storage down, FlexiScale's processing and networking services are dark as well. But the platform should be returned to normal, once engineers restore the lost data to a brand new disk structure.

"After consulting with our storage vendor it was agreed the most sensible option would be to copy the entire volume to a new disk structure (still maintaining its integrity and structure), from where we could re-mount it correctly," Lucas' email reads. "Unfortunately, due to its size, we didn't have spare capacity on the platform to create a complete duplicate of it."

To Lucas' credit, he makes no excuses for outage - and hasn't tried to hide it. "This shouldn't have happened, shouldn't have been allowed to happen, but it happened. We'll be putting things into place that will prevent this from happening in the future."

Specifically, XCalibre will soon distribute its architecture across multiple data centers. "So, if something like this were to happen again, customers could fall over to an other data centers," Lucas says. But a second data center isn't due to open until January.

Yes, Lucas says, it would have been nice to have a second data center in place back in October, but, well, funds were tight. "We were a privately funded company," he says. "We were resource constrained. We will complete another round of funding in the next few weeks."

And Lucas is confident that cloudy outfits like his will eventually iron out all these issues. "One of my pet projects is interoperability and portability between cloud computing devices - basically, the ability to move from one to another if something goes wrong," he says. "But I didn't think I'd make such a good example of why that's important."

So, XCalibre's problems could be solved if Google would just make "a more loving cloud." ®

Beginner's guide to SSL certificates

More from The Register

next story
Docker's app containers are coming to Windows Server, says Microsoft
MS chases app deployment speeds already enjoyed by Linux devs
'Hmm, why CAN'T I run a water pipe through that rack of media servers?'
Leaving Las Vegas for Armenia kludging and Dubai dune bashing
'Urika': Cray unveils new 1,500-core big data crunching monster
6TB of DRAM, 38TB of SSD flash and 120TB of disk storage
Facebook slurps 'paste sites' for STOLEN passwords, sprinkles on hash and salt
Zuck's ad empire DOESN'T see details in plain text. Phew!
SDI wars: WTF is software defined infrastructure?
This time we play for ALL the marbles
Windows 10: Forget Cloudobile, put Security and Privacy First
But - dammit - It would be insane to say 'don't collect, because NSA'
Oracle hires former SAP exec for cloudy push
'We know Larry said cloud was gibberish, and insane, and idiotic, but...'
Symantec backs out of Backup Exec: Plans to can appliance in Jan
Will still provide support to existing customers
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
Win a year’s supply of chocolate
There is no techie angle to this competition so we're not going to pretend there is, but everyone loves chocolate so who cares.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Intelligent flash storage arrays
Tegile Intelligent Storage Arrays with IntelliFlash helps IT boost storage utilization and effciency while delivering unmatched storage savings and performance.