Feeds

Google says it's sorry for Monday's hours-long Gmail delays

Dual networking outage won't happen again, honest

Top three mobile application threats

Google apologized on Tuesday for a networking glitch that prevented emails from reaching many Gmail users' accounts for as much as two hours or even longer.

"The message delivery delays were triggered by a dual network failure," Gmail site reliability engineer Sabrina Farmer wrote in a blog post. "This is a very rare event in which two separate, redundant network paths both stop working at the same time."

Email delivery broke down at around 5:54am Pacific time on Monday, Farmer said, and the online ad giant didn't get on top of the problem until around 1pm the same day. The full message backlog wasn't cleared until around 4pm.

The service interruption only affected around 29 per cent of messages passing through Gmail, Farmer said, and of those, the typical message was only delayed by 2.6 seconds. But some messages were left hanging much longer, and in the worst cases – about 1.5 per cent of the total – they were delayed for more than two hours. In addition, some users who tried to download large attachments from their Gmail accounts experienced errors.

A contrite Farmer expressed Google's regrets. "We realize that our users rely on Gmail to be always available and always fast, and for several hours we didn't deliver," she wrote.

But to be fair, she said, even several hours' worth of spotty networking had negligible impact on Gmail's overall uptime stats. "Gmail remains well above 99.9% available," she wrote, "and we intend to keep it that way!"

Users weren't locked out of their Gmail accounts during the incident, and they were able to read email that had already been delivered and even send new messages of their own.

Gmail has certainly dealt with worse. In 2012, a misconfigured sync server triggered a system-wide Gmail outage and caused the Chrome browser to spontaneously crash at the same time. And just last month, all of Google's services mysteriously vanished from the net at once, causing not just Gmail but 40 per cent of internet traffic to go dark for a few minutes.

Still, Farmer says Gmail will be updating its network capacity and adjusting its infrastructure so that mail delivery will be "more resilient," even in the event of a dual network failure. What's more, the Chocolate Factory will rejigger its internal practices so that the next time something like this happens, its engineering teams will be quicker to respond.

Shame on you, Google. Shame on you. ®

Securing Web Applications Made Simple and Scalable

More from The Register

next story
Yorkshire cops fail to grasp principle behind BT Fon Wi-Fi network
'Prevent people that are passing by to hook up to your network', pleads plod
Auntie remains MYSTIFIED by that weekend BBC iPlayer and website outage
Still doing 'forensics' on the caching layer – Beeb digi wonk
Major problems beset UK ISP filth filters: But it's OK, nobody uses them
It's almost as though pr0n was actually rather popular
Microsoft unsheathes cheap Android-killer: Behold, the Lumia 530
Say it with us: I'm King of the Landfill-ill-ill-ill
All those new '5G standards'? Here's the science they rely on
Radio professor tells us how wireless will get faster in the real world
Apple orders huge MOUNTAIN of 80 MILLION 'Air' iPhone 6s
Bigger, harder trouser bulges foretold for fanbois
US freemium mobile network eyes up Europe
FreedomPop touts 'free' calls, texts and data
'Two-speed internet' storm turns FCC.gov into zero-speed website
Deadline for comments on net neutrality shake-up extended to Friday
Oh girl, you jus' didn't: Level 3 slaps Verizon in Netflix throttle blowup
Just hook us up to more 10Gbps ports, backbone biz yells in tit-for-tat spat
prev story

Whitepapers

Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.