Feeds

Amazon gets 'F' for communication amidst cloud outage

CTO's distributed computing pal analyzes EC2 failure

  • alert
  • submit to reddit

The essential guide to IT transformation

'500,000' volumes affected

According to RightScale's extrapolations, about 500,000 EBS volumes were affected.

"After Amazon managed to contain the problems to one zone, it took a very long time to get the EBS machinery under control and to recover all the volumes. Given the extrapolated number of volumes it would not be surprising that an event of this scale exceeded the design parameters and was never tested (or able to be tested). I’m not sure there is any system of comparable scale in operation anywhere," von Eicken says.

"I do want to state that while 'something large' clearly failed, namely the EBS system as a whole, the real big failure is that multiple availability zones were affected for ~3 hours."

In addition, von Eicken says he's "uncomfortable" with the performance of Amazon RDS (Relational Database Service), which serves up MySQL and at least one other database via EC2. RDS also experienced problems during the outage. "Some databases that were replicated across multiple availability zones did not fail-over properly," he says. "It evidently took more than 12 hours to recover a number of the multi-[availability zone] databases."

According to von Eicken, Amazon has "made it difficult" to backup their database outside of RDS, and this leaves them "no choice but to wait for someone at Amazon to work on their database. This lock-in is one reason many of our customers prefer to use our MySQL master-slave setup or to architect their own."

Even RightScale, he says, was confused by Amazon's terse status messages. "In hindsight we should have intentionally failed-over our master database which was operating in the 'impacted availability zone' early on at a time where we could minimize downtime." he says. "We were lucky that it didn’t get affected until about 12 hours after the start of the outage, but we didn’t connect one and one. A clear message from Amazon that more and more volumes were continuing to fail in the zone would have been really helpful."

von Eicken has called on Amazon to make a long list of improvements to its communication policies. These include giving actual percentages of volumes affected by an outage, listing the names of each availability zone affected, and sending individual status updates to particular users.

"Use email (or other means) to tell us what the status of our instances and volumes is," he says. "I don’t mean things I can get myself like cpu load or such, but information like 'the following volumes (by id) are currently recovering and should be available within the next hour, the following volumes will require manual intervention at a later time, …' That allows users to plan and choose where to put their efforts."

But he also wants a blog post from the company. Which is still yet to be seen. As of 1pm Pacific time on Monday, Amazon says that a "vast majority" of volumes have been recovered and that EBS is now operating normally for all APIs on these volumes, but some volumes have not been recovered, and the company says it is working to contact the customers involved.

Thorsten von Eicken was a professor in Cornell's computer science department when Werner Vogels did research there, and the two have coauthored distributed-computing research. RightScale began as a web front-end for Amazon Web Services when the service was only accessible from a command line, but it has sense morphed into a management service for several other infrastructure clouds as well. ®

Boost IT visibility and business value

More from The Register

next story
Pay to play: The hidden cost of software defined everything
Enter credit card details if you want that system you bought to actually be useful
HP busts out new ProLiant Gen9 servers
Think those are cool? Wait till you get a load of our racks
Shoot-em-up: Sony Online Entertainment hit by 'large scale DDoS attack'
Games disrupted as firm struggles to control network
Silicon Valley jolted by magnitude 6.1 quake – its biggest in 25 years
Did the earth move for you at VMworld – oh, OK. It just did. A lot
VMware's high-wire balancing act: EVO might drag us ALL down
Get it right, EMC, or there'll be STORAGE CIVIL WAR. Mark my words
Forrester says it's time to give up on physical storage arrays
The physical/virtual storage tipping point may just have arrived
prev story

Whitepapers

Top 10 endpoint backup mistakes
Avoid the ten endpoint backup mistakes to ensure that your critical corporate data is protected and end user productivity is improved.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Backing up distributed data
Eliminating the redundant use of bandwidth and storage capacity and application consolidation in the modern data center.
The essential guide to IT transformation
ServiceNow discusses three IT transformations that can help CIOs automate IT services to transform IT and the enterprise
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.