Feeds

MS blames lowly techie for Web blackout

Takes 22 hours to fix router config error

  • alert
  • submit to reddit

Boost IT visibility and business value

Microsoft has blamed a lowly technician for a cock-up which almost completely blocked access to its Web sites for most users yesterday.

From the early hours of yesterday morning until late evening www.microsoft.com, msn.com, expedia.co.uk and msnbc.com were all unavailable. The software giant's Hotmail service was also inaccessible for many.

The problem, whose final resolution came some six hours after Microsoft promised a fix would be in place yesterday, was due to changes in Microsoft's domain name server network caused requests to access its Web sites to fail. A fix was eventually put in place when Microsoft removed the changes made to the configuration that were behind the problem.

In a statement, Microsoft admitted: "At 6:30 p.m. Tuesday (PST), a Microsoft technician made a configuration change to the routers on the edge of Microsoft's Domain Name Server network. The DNS servers are used to connect domain names with numeric IP addresses (eg. 207.46.230.219) of the various servers and networks that make up Microsoft's Web presence.

"The mistaken configuration change limited communication between DNS servers on the Internet and Microsoft's DNS servers. This limited communication caused many of Microsoft's sites to be unreachable (although they were actually still operational) to a large number of customers."

Microsoft has apologised to customers for the problem, which it denies is down to either a technology failure with its or anybody else's products nor down to lax security of its networks. This leaves as lowly techie, who we'd be most interesting in talking to, carrying the can for the whole sorry debacle, which hardly inspires much confidence in Microsoft's .Net vision for delivering software as a service over the Internet.

As Register readers pointed out, all four of Microsoft's domain name servers appear to be located on the same subnet - effectively putting all its eggs in one basket.

Russ Cooper, editor of security mailing list NTBugtraq, has laid into Microsoft's explanation and said "stupid people do make mistakes, but for a company the size of MS there's really no excuse for such a blunder."

Cooper said: "It seems the corporation [Microsoft] is not sufficiently aware of the importance of DNS, despite its role in both .NET. You think you might have a disaster recovery plan that gets invoked within an hour of confirmation that DNS is out... one that includes checking the router configuration for your DNS 'network'."

He added that the debacle showed that there was no vetting or management sign-off to changes in production networks at Redmond. ®

Related Stories

Microsoft brings web sites back into play
Microsoft confirms Web site blackout
DNS trouble made Microsoft, Yahoo! unavailable

Build a business case: developing custom apps

More from The Register

next story
BBC goes offline in MASSIVE COCKUP: Stephen Fry partly muzzled
Auntie tight-lipped as major outage rolls on
iPad? More like iFAD: We reveal why Apple fell into IBM's arms
But never fear fanbois, you're still lapping up iPhones, Macs
Nadella: Apps must run on ALL WINDOWS – PCs, slabs and mobes
Phone egg, meet desktop chicken - your mother
White? Male? You work in tech? Let us guess ... Twitter? We KNEW it!
Grim diversity numbers dumped alongside Facebook earnings
Microsoft: We're making ONE TRUE WINDOWS to rule us all
Enterprise, Windows still power firm's shaky money-maker
HP, Microsoft prove it again: Big Business doesn't create jobs
SMEs get lip service - what they need is dinner at the Club
ITC: Seagate and LSI can infringe Realtek patents because Realtek isn't in the US
Land of the (get off scot) free, when it's a foreign owner
Dude, you're getting a Dell – with BITCOIN: IT giant slurps cryptocash
1. Buy PC with Bitcoin. 2. Mine more coins. 3. Goto step 1
There's NOTHING on TV in Europe – American video DOMINATES
Even France's mega subsidies don't stop US content onslaught
prev story

Whitepapers

Top three mobile application threats
Prevent sensitive data leakage over insecure channels or stolen mobile devices.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
The Essential Guide to IT Transformation
ServiceNow discusses three IT transformations that can help CIO's automate IT services to transform IT and the enterprise.