Feeds

HOT SWEATY RACKS blamed for Outlook.com, Hotmail MELTDOWN

Firmware cock-up cooked servers in data-centre oven

Designing a Defense for Mobile Applications

Microsoft has admitted a dodgy firmware upgrade cooked its servers and knocked its Hotmail and Outlook.com email services offline for 16 hours.

In a postmortem examination of the disaster, the Windows 8 giant said a software upgrade for its data centre equipment - an update that had worked successfully in the past - failed unexpectedly on the afternoon of Tuesday, 12 March. The cock-up resulted in a “rapid and substantial temperature spike” in the hall housing the websites' servers.

Microsoft did not say in which of its many data centres the machines are based. The company's online storage service Skydrive was also affected but brought back online faster than its web siblings.

The Hotmail and Outlook.com rack-mounted computers weren't the only systems getting hot under the collar: irate users vented on Twitter their displeasure at being unable to access their emails for nearly a full day.

Microsoft veep Arthur de Haan said the soaring mercury triggered unspecified “safeguards” in the data centre that shut down the sweating servers before the fault turned really nasty - albeit without switching to the backup:

These safeguards prevented access to mailboxes housed on these servers and also prevented any other pieces of our infrastructure to automatically failover and allow continued access. This area of the datacenter houses parts of the Hotmail.com, Outlook.com, and SkyDrive infrastructure, and so some people trying to access those services were impacted.

De Haan’s analysis continued in torturously overworked prose to explain that, in non-technical terms, some techies tried turning it off and back on again to fix it. The exec himself put it thus: “There was a mix of infrastructure software and human intervention that was needed to bring the core infrastructure back online."

His explanation was infused with standard-issue corporate-grade apologies and attempts to play down the impact of the outage. De Haan said the vast majority of Hotmail, Outlook.com and Skydrive users weren’t affected, adding that Microsoft “sincerely apologises" and takes any downtime “very seriously”.

The problem hit Hotmail accounts being migrated to Outlook.com and is the second such failure in less than two months. Microsoft’s goal is to transfer all 360 million Hotmail inboxes by the summer. Hotmail is still the world's largest free email service.

The fact that a regular firmware update had near-disastrous consequences should raise a few eyebrows: after all, cloud-based services are, by design, supposed to keep ticking even if they take a kicking without outsiders noticing. ®

The Power of One eBook: Top reasons to choose HP BladeSystem

More from The Register

next story
Apple fanbois SCREAM as update BRICKS their Macbook Airs
Ragegasm spills over as firmware upgrade kills machines
Attack of the clones: Oracle's latest Red Hat Linux lookalike arrives
Oracle's Linux boss says Larry's Linux isn't just for Oracle apps anymore
THUD! WD plonks down SIX TERABYTE 'consumer NAS' fatboy
Now that's a LOT of porn or pirated movies. Or, you know, other consumer stuff
EU's top data cops to meet Google, Microsoft et al over 'right to be forgotten'
Plan to hammer out 'coherent' guidelines. Good luck chaps!
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
FLAPE – the next BIG THING in storage
Find cold data with flash, transmit it from tape
prev story

Whitepapers

Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.