Feeds

MS blames lowly techie for Web blackout

Takes 22 hours to fix router config error

  • alert
  • submit to reddit

High performance access to file storage

Microsoft has blamed a lowly technician for a cock-up which almost completely blocked access to its Web sites for most users yesterday.

From the early hours of yesterday morning until late evening www.microsoft.com, msn.com, expedia.co.uk and msnbc.com were all unavailable. The software giant's Hotmail service was also inaccessible for many.

The problem, whose final resolution came some six hours after Microsoft promised a fix would be in place yesterday, was due to changes in Microsoft's domain name server network caused requests to access its Web sites to fail. A fix was eventually put in place when Microsoft removed the changes made to the configuration that were behind the problem.

In a statement, Microsoft admitted: "At 6:30 p.m. Tuesday (PST), a Microsoft technician made a configuration change to the routers on the edge of Microsoft's Domain Name Server network. The DNS servers are used to connect domain names with numeric IP addresses (eg. 207.46.230.219) of the various servers and networks that make up Microsoft's Web presence.

"The mistaken configuration change limited communication between DNS servers on the Internet and Microsoft's DNS servers. This limited communication caused many of Microsoft's sites to be unreachable (although they were actually still operational) to a large number of customers."

Microsoft has apologised to customers for the problem, which it denies is down to either a technology failure with its or anybody else's products nor down to lax security of its networks. This leaves as lowly techie, who we'd be most interesting in talking to, carrying the can for the whole sorry debacle, which hardly inspires much confidence in Microsoft's .Net vision for delivering software as a service over the Internet.

As Register readers pointed out, all four of Microsoft's domain name servers appear to be located on the same subnet - effectively putting all its eggs in one basket.

Russ Cooper, editor of security mailing list NTBugtraq, has laid into Microsoft's explanation and said "stupid people do make mistakes, but for a company the size of MS there's really no excuse for such a blunder."

Cooper said: "It seems the corporation [Microsoft] is not sufficiently aware of the importance of DNS, despite its role in both .NET. You think you might have a disaster recovery plan that gets invoked within an hour of confirmation that DNS is out... one that includes checking the router configuration for your DNS 'network'."

He added that the debacle showed that there was no vetting or management sign-off to changes in production networks at Redmond. ®

Related Stories

Microsoft brings web sites back into play
Microsoft confirms Web site blackout
DNS trouble made Microsoft, Yahoo! unavailable

SANS - Survey on application security programs

More from The Register

next story
Dropbox defends fantastically badly timed Condoleezza Rice appointment
'Nothing is going to change with Dr. Rice's appointment,' file sharer promises
Audio fans, prepare yourself for the Second Coming ... of Blu-ray
High Fidelity Pure Audio – is this what your ears have been waiting for?
MtGox chief Karpelès refuses to come to US for g-men's grilling
Bitcoin baron says he needs another lawyer for FinCEN chat
Record labels sue Pandora over vintage song royalties
Companies want payout on recordings made before 1972
Zucker punched: Google gobbles Facebook-wooed Titan Aerospace
Up, up and away in my beautiful balloon flying broadband-bot
Apple DOMINATES the Valley, rakes in more profit than Google, HP, Intel, Cisco COMBINED
Cook & Co. also pay more taxes than those four worthies PLUS eBay and Oracle
Number crunching suggests Yahoo! US is worth less than nothing
China and Japan holdings worth more than entire company
prev story

Whitepapers

SANS - Survey on application security programs
In this whitepaper learn about the state of application security programs and practices of 488 surveyed respondents, and discover how mature and effective these programs are.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Top three mobile application threats
Learn about three of the top mobile application security threats facing businesses today and recommendations on how to mitigate the risk.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.