The Register® — Biting the hand that feeds IT

Feeds

Worldwide Gmail crash was due to Google Sync bug

Engineer: How Oompah-Loompah config foul-up pulled down world's email

Email delivery: Hate phishing emails? You'll love DMARC

It was a Googler misconfiguring a sync server that took out Chrome and Gmail on Monday, an engineer has stated on dev forums.

The reason for Gmail's brief burnout on Monday has been winkled out, and it was connected to the rolling crashes suffered on the Chrome browser on the same day.

It was a human error in configuring a Chrome component that throttled traffic in Chrome and then on several other Google services, explained Chrome engineer Tim Steele in a post on the Google code forum.

The component controlled traffic for Chrome Sync - a service that allows users to synchronise their customised Chrome browser across all of their devices, giving them the same bookmarks, widgets, settings and browsing history.

A small change to its configuration settings meant the load-balancing component started to throttle traffic when it wasn't supposed to. And because the component is core to the infrastructure that many Google services depend on, it affected them too.

It was Chrome Sync users who were hit by the outage first.

Steele explained:

- Chrome Sync Server relies on a backend infrastructure component to enforce quotas on per-datatype sync traffic.
- That quota service experienced traffic problems today due to a faulty load balancing configuration change.
- That change was to a core piece of infrastructure that many services at Google depend on. This means other services may have been affected at the same time, leading to the confounding original title [the Gmail bug] of this bug.
- Because of the quota service failure, Chrome Sync servers reacted too conservatively by telling clients to throttle "all" data types, without accounting for the fact that not all client versions support all data types.

The crash is due to faulty logic responsible for handling "throttled" data types on the client when the data types are unrecognized.

There is WRONG in the cloud. ®

5 ways to reduce advertising network latency

Whitepapers

Microsoft’s Cloud OS
System Center Virtual Machine manager and how this product allows the level of virtualization abstraction to move from individual physical computers and clusters to unifying the whole Data Centre as an abstraction layer.
5 ways to prepare your advertising infrastructure for disaster
Being prepared allows your brand to greatly improve your advertising infrastructure performance and reliability that, in the end, will boost confidence in your brand.
Supercharge your infrastructure
Fusion­‐io has developed a shared storage solution that provides new performance management capabilities required to maximize flash utilization.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Avere FXT with FlashMove and FlashMirror
This ESG Lab validation report documents hands-on testing of the Avere FXT Series Edge Filer with the AOS 3.0 operating environment.

More from The Register

next story
Dedupe-dedupe, dedupe-dedupe-dedupe: Flashy clients crowd around Permabit diamond
3 of the top six flash vendors are casing the OEM dedupe tech, claims analyst
Disk-pushers, get reel: Even GOOGLE relies on tape
Prepare to be beaten by your old, cheap rival
Dragons' Den star's biz Outsourcery sends yet more millions up in smoke
Telly moneybags went into the cloud and still nobody's making any profit
Hong Kong's data centres stay high and dry amid Typhoon Usagi
180 km/h winds kill 25 in China, but the data centres keep humming
Microsoft lures punters to hybrid storage cloud with free storage arrays
Spend on Azure, get StorSimple box at the low, low price of $0
WD unveils new MyBook line: External drives now bigger... and CHEAP
Less than £0.04/GB, but it loses the Thunderbolt speed
VMware vSAN test pilots: Don't panic but there's a chance of DATA LOSS
AHCI SATA controller won't play nice with Virtzilla's robo-storage beta
prev story