Feeds

How a tiny leap-day miscalculation trashed Microsoft Azure

Redmond drills into cause of eight-hour outage

Mobile application security vulnerability report

As soon as Microsoft's cloudy platform Azure crashed to Earth, and stayed there for eight hours, on 29 February, every developer who has ever had to handle dates immediately figured it was a leap-day bug.

Now the software biz behemoth has put its hands up and admitted in a detailed dissection of the blunder how a calendar glitch trashed its server farm. It's also a handy guide to setting up your own wholesale-sized cloud platform.

The mega-crash stemmed from the handling of messages passed between a virtual machine running a client's application and the underlying host operating system running on each of the Azure servers.

These messages are encrypted using a public-private key pair taken from a "transfer certificate" generated within the VM. This security measure allows the host OS and the VM to trust their communications channel, through which stuff such as SSL certificates and diagnostic health checks are sent.

A transfer certificate is valid for a year from its creation date. A certificate created by an agent in a VM on 29 February 2012 will expire on 29 February 2013, a date that simply doesn't exist. This crashed the certificate validation process, bringing the start up of the VM to a halt.

The host OS tries to restart a crashed app VM every 25 minutes, which would have been fruitless in this case. After a series of failures, the host OS declares the hardware to be at fault and reports the server as knackered. Automatic systems that manage the clusters of servers try to self-heal the cloud by restarting the VMs on other boxes - in this scenario it caused them to also fall over, cascading the gaffe into a full-blown outage.

It didn't help that, at the time, new versions of the cloud's platform software were being rolled out, which required the generation of new albeit broken transfer certificates. Once enough servers are reported faulty in a cluster, the whole set is put on red alert - halting self-healing and software updates to minimise the damage.

Microsoft engineers confessed that in a rush to roll out the fix to the servers, they hit incompatibility problems within their own code, which knocked out services again.

You can read the full cock-up, blow by blow, here. Microsoft has said it will cough up service credits for customers walloped by the Azure outage. ®

Bridging the IT gap between rising business demands and ageing tools

More from The Register

next story
THUD! WD plonks down SIX TERABYTE 'consumer NAS' fatboy
Now that's a LOT of porn or pirated movies. Or, you know, other consumer stuff
EU's top data cops to meet Google, Microsoft et al over 'right to be forgotten'
Plan to hammer out 'coherent' guidelines. Good luck chaps!
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
FLAPE – the next BIG THING in storage
Find cold data with flash, transmit it from tape
Seagate chances ARM with NAS boxes for the SOHO crowd
There's an Atom-powered offering, too
prev story

Whitepapers

Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.