Feeds

Google says it's sorry for Monday's hours-long Gmail delays

Dual networking outage won't happen again, honest

Secure remote control for conventional and virtual desktops

Google apologized on Tuesday for a networking glitch that prevented emails from reaching many Gmail users' accounts for as much as two hours or even longer.

"The message delivery delays were triggered by a dual network failure," Gmail site reliability engineer Sabrina Farmer wrote in a blog post. "This is a very rare event in which two separate, redundant network paths both stop working at the same time."

Email delivery broke down at around 5:54am Pacific time on Monday, Farmer said, and the online ad giant didn't get on top of the problem until around 1pm the same day. The full message backlog wasn't cleared until around 4pm.

The service interruption only affected around 29 per cent of messages passing through Gmail, Farmer said, and of those, the typical message was only delayed by 2.6 seconds. But some messages were left hanging much longer, and in the worst cases – about 1.5 per cent of the total – they were delayed for more than two hours. In addition, some users who tried to download large attachments from their Gmail accounts experienced errors.

A contrite Farmer expressed Google's regrets. "We realize that our users rely on Gmail to be always available and always fast, and for several hours we didn't deliver," she wrote.

But to be fair, she said, even several hours' worth of spotty networking had negligible impact on Gmail's overall uptime stats. "Gmail remains well above 99.9% available," she wrote, "and we intend to keep it that way!"

Users weren't locked out of their Gmail accounts during the incident, and they were able to read email that had already been delivered and even send new messages of their own.

Gmail has certainly dealt with worse. In 2012, a misconfigured sync server triggered a system-wide Gmail outage and caused the Chrome browser to spontaneously crash at the same time. And just last month, all of Google's services mysteriously vanished from the net at once, causing not just Gmail but 40 per cent of internet traffic to go dark for a few minutes.

Still, Farmer says Gmail will be updating its network capacity and adjusting its infrastructure so that mail delivery will be "more resilient," even in the event of a dual network failure. What's more, the Chocolate Factory will rejigger its internal practices so that the next time something like this happens, its engineering teams will be quicker to respond.

Shame on you, Google. Shame on you. ®

Gartner critical capabilities for enterprise endpoint backup

More from The Register

next story
UK fuzz want PINCODES on ALL mobile phones
Met Police calls for mandatory passwords on all new mobes
Canadian ISP Shaw falls over with 'routing' sickness
How sure are you of cloud computing now?
Don't call it throttling: Ericsson 'priority' tech gives users their own slice of spectrum
Actually it's a nifty trick - at least you'll pay for what you get
Three floats Jolla in Hong Kong: Says Sailfish is '3rd option'
Network throws hat into ring with Linux-powered handsets
Fifteen zero days found in hacker router comp romp
Four routers rooted in SOHOpelessly Broken challenge
New Sprint CEO says he will lower axe on staff – but prices come first
'Very disruptive' new rates to be revealed next week
US TV stations bowl sueball directly at FCC's spectrum mega-sale
Broadcasters upset about coverage and cost as they shift up and down the dials
Trans-Pacific: Google spaffs cash on FAST undersea packet-flinging
One of 6 backers for new 60 Tbps cable to hook US to Japan
Tech city types developing 'Google Glass for the blind' app
An app and service where other people 'see' for you
UK mobile coverage is BETTER than EVER, networks tell Ofcom
Regulator swallows this line and parrots it back out at us. What are they playing at?
prev story

Whitepapers

5 things you didn’t know about cloud backup
IT departments are embracing cloud backup, but there’s a lot you need to know before choosing a service provider. Learn all the critical things you need to know.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.
Rethinking backup and recovery in the modern data center
Combining intelligence, operational analytics, and automation to enable efficient, data-driven IT organizations using the HP ABR approach.
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.