Feeds

MS blames lowly techie for Web blackout

Takes 22 hours to fix router config error

  • alert
  • submit to reddit

New hybrid storage solutions

Microsoft has blamed a lowly technician for a cock-up which almost completely blocked access to its Web sites for most users yesterday.

From the early hours of yesterday morning until late evening www.microsoft.com, msn.com, expedia.co.uk and msnbc.com were all unavailable. The software giant's Hotmail service was also inaccessible for many.

The problem, whose final resolution came some six hours after Microsoft promised a fix would be in place yesterday, was due to changes in Microsoft's domain name server network caused requests to access its Web sites to fail. A fix was eventually put in place when Microsoft removed the changes made to the configuration that were behind the problem.

In a statement, Microsoft admitted: "At 6:30 p.m. Tuesday (PST), a Microsoft technician made a configuration change to the routers on the edge of Microsoft's Domain Name Server network. The DNS servers are used to connect domain names with numeric IP addresses (eg. 207.46.230.219) of the various servers and networks that make up Microsoft's Web presence.

"The mistaken configuration change limited communication between DNS servers on the Internet and Microsoft's DNS servers. This limited communication caused many of Microsoft's sites to be unreachable (although they were actually still operational) to a large number of customers."

Microsoft has apologised to customers for the problem, which it denies is down to either a technology failure with its or anybody else's products nor down to lax security of its networks. This leaves as lowly techie, who we'd be most interesting in talking to, carrying the can for the whole sorry debacle, which hardly inspires much confidence in Microsoft's .Net vision for delivering software as a service over the Internet.

As Register readers pointed out, all four of Microsoft's domain name servers appear to be located on the same subnet - effectively putting all its eggs in one basket.

Russ Cooper, editor of security mailing list NTBugtraq, has laid into Microsoft's explanation and said "stupid people do make mistakes, but for a company the size of MS there's really no excuse for such a blunder."

Cooper said: "It seems the corporation [Microsoft] is not sufficiently aware of the importance of DNS, despite its role in both .NET. You think you might have a disaster recovery plan that gets invoked within an hour of confirmation that DNS is out... one that includes checking the router configuration for your DNS 'network'."

He added that the debacle showed that there was no vetting or management sign-off to changes in production networks at Redmond. ®

Related Stories

Microsoft brings web sites back into play
Microsoft confirms Web site blackout
DNS trouble made Microsoft, Yahoo! unavailable

Security for virtualized datacentres

More from The Register

next story
Phones 4u slips into administration after EE cuts ties with Brit mobe retailer
More than 5,500 jobs could be axed if rescue mission fails
Driving with an Apple Watch could land you with a £100 FINE
Bad news for tech-addicted fanbois behind the wheel
Phones 4u website DIES as wounded mobe retailer struggles to stay above water
Founder blames 'ruthless network partners' for implosion
Sony says year's losses will be FOUR TIMES DEEPER than thought
Losses of more than $2 BILLION loom over troubled Japanese corp
Radio hams can encrypt, in emergencies, says Ofcom
Consultation promises new spectrum and hints at relaxed licence conditions
Big Content Australia just blew a big hole in its credibility
AHEDA's research on average content prices did not expose methodology, so appears less than rigourous
Special pleading against mass surveillance won't help anyone
Protecting journalists alone won't protect their sources
Bono: Apple will sort out monetising music where the labels failed
Remastered so hard it would be difficult or impossible to master it again
prev story

Whitepapers

Secure remote control for conventional and virtual desktops
Balancing user privacy and privileged access, in accordance with compliance frameworks and legislation. Evaluating any potential remote control choice.
WIN a very cool portable ZX Spectrum
Win a one-off portable Spectrum built by legendary hardware hacker Ben Heck
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
The next step in data security
With recent increased privacy concerns and computers becoming more powerful, the chance of hackers being able to crack smaller-sized RSA keys increases.