SaaS data loss: The problem you didn’t know you had

The cooling system inside Google's Belgium data center has no chillers. It uses nothing but outside air - so-called "free-cooling" - to keep temperatures down. And if the Belgian air gets too hot, Google shifts the data center's compute loads to other facilities.

As we reported late last month, Google senior manager of engineering and architecture Vijay Gill alluded to this chiller-less setup during a cloud-happy tech conference in San Francisco. And our piece sparked a follow-up story from our friends at Data Center Knowledge.

The free-cooling idea is hardly unique - the likes of Microsoft and Yahoo! are also working to cut energy costs by using alternative cooling sources - but Google isn't even using chillers as a fall-back.

Google tends to operate its data centers at over 80 degrees Fahrenheit - well above the norm - and according to one former employee, Intel provides the Chocolate Factory with chips that are better able to withstand heat than garden variety Chipzilla processors. But it's unclear how Google's free cooling setup operates. The company did not immediately respond to our request for comment.

Speaking at Structure 09, the wonderfully-witty Vijay Gill seemed to indicate that when there's a temperature spike in the chiller-less data center, its top-secret infrastructure can respond without human intervention.

"You have to have integration with everything right from the chillers down all the way to the CPU," he said. "Sometimes, there's a temperature excursion, and you might want to do a quick load-shedding - a quick load-shedding to prevent a temperature excursion because, hey, you have a data center with no chillers. You want to move some load off. You want to cut some CPUs and some of the processes in RAM."

And he hinted that Google can (almost) instantly shift loads from one data center to another as if moving data between servers. Google likes to think of each data center as one big machine. The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines is the title of the now-famous paper.

"How do you manage the system and optimize it on a global level? That is the interesting part," Gill continued. "What we’ve got here [with Google] is massive - like hundreds of thousands of variable linear programming problems that need to run in quasi-real-time. When the temperature starts to excurse in a data center, you don’t have the luxury to sitting around for a half an hour...You have on the order of seconds."

But when asked if this technology is in place today, Gill responded in typical Google fashion. "I could not possibly comment on that," he said. Likewise, when The Reg contacted Google today about its chiller-less setup, the company did not immediately respond.

In a March interview with Data Center Knowledge, however, Google senior vice president of operations Urs Holzle indicated that the company uses manual tools for load shifts between data centers. "Teams regularly practice failing out of or routing around specific data centers as part of scheduled maintenance,” he said. "Sometimes, we need to build new tools when new classes of problems happen."

And if Google does have automated tools doing this sort of thing, you have to wonder how well they're working. Earlier this year, two much-discussed Gmail outages involved Google shifting loads between data centers. ®

Steps to Take Before Choosing a Business Continuity Partner