Feeds

Google Go boldly goes where no code has gone before

How to build all the Google stuff Google won't talk about

Remote control for virtualized desktops

Every Google data center has a Chubby, and Heroku wanted one too.

Like the rest of Google's much admired back-end infrastructure, Chubby is decidedly closed source, so Heroku men Keith Rarick and Blake Mizerany built their own. Although they didn't have source code, they did have one of those secret-spilling Google research papers. And they had Go, Google's open source programming language for building systems like Google's.

Running its myriad online services atop a famously distributed back-end, splitting tasks into tiny pieces and spreading them across a vast network of machines, Google needs a way of controlling access to those machines – and that's Chubby. According to a 2006 Google research paper (PDF), there's a least one Chubby instance in each of the company's data centers, coordinating server access for its GFS distributed file system, its BigTable database, and its epic number-crunching platform, MapReduce.

Heroku runs a similarly distributed infrastructure behind its eponymous web service – a means of building, hosting, and readily scaling Ruby on Rails applications – and this too requires the sort of server-juggling you get from Chubby. "When you have a lot of distributed processes – lots of things going on, a network spanning a lot of machines – coordinating things can be tricky," Heroku system architect Keith Rarick tells The Register.

"If a machine goes down or you're having network problems where some machines can't talk to each other or the network is slow or you're getting packet loss, there are so many ways things that can fail. It can be really, really hard to get all those things right if you're doing ad hoc coordination."

Google Chubby

Only known image of Google Chubby

Using an algorithm called Paxos, Rarick says, Chubby almost magically simplifies this process. But first, you have to build the thing. And for that, Rarick turned to Go. Conceived by a trio of Google heavyweights – Unix founding father Ken Thompson, fellow Bell Labs Unix developer Rob Pike, and Robert Griesemer, who worked on the Java HotSpot compiler – Go is specifically designed for distributed systems.

"We realized that the kind of software we build at Google is not always served well by the languages we had available," Rob Pike tells us. "Robert Griesemer, Ken Thompson, and myself decided to make a language that would be very good for writing the kinds of programs we write at Google."

Released as an experimental language in late 2009 and now in production at Google, Go includes built-in mechanisms for running concurrent tasks. Using lightweight processes known as "goroutines", you can readily juggle multiple tasks within the same operating system thread or spread them across disparate threads, and all this is automatically scheduled by the Go runtime. The inherently concurrent setup was ideal for Heroku's Chubby mimic, an open source project known as Doozer.

"Go is very good at letting you have multiple points of control in a single program, and coordinating the synchronization and communication between those different points of control," Rarick explains. "That's something that's really useful in a project like [Doozer]. We have a bunch of different processes on different machines trying to talk to each other, and you have to make sure they all have a consistent view of the world."

Go not only provides the concurrency Doozer requires, it keeps memory usage to a minimum. It offers garbage collection, but it also frees you from thread overload. "We don't have to worry about how many threads we're running. We aren't constantly saying 'Are we running out of resources?'" Rarick explains. "We can have lots of goroutines instead, and that's built into the language."

Other languages provide somewhat similar mechanisms for concurrency – such as Erlang and Scala – but Go is designed to provide maximum efficiency and control, as well. "It comes from a line of systems programming language like C and C++, so it gives you the ability to really control the performance characteristics," Rarick says. "When it comes time to measure things and make sure they run fast, you have the flexibility to really get in there and do what you need. And when you figure out why your program is being slow, you really have the control you need to fix it."

Keith Rarick

Keith Rarick

It's a unique combination. "C gives you control, but it's not good for concurrency. It doesn't even give you garbage collection," he says. "Go gives you concurrency and garbage collection, but it still gives you control over memory layout and resource use."

Google won't say how Go is used inside its mystery data centers. But Doozer – officially released earlier this month and set to go live at Heroku – is a suitable poster child for the fledgling language. It's no coincidence that Robert Griesemer, one of the original Go architects, also worked on Chubby.

Internet Security Threat Report 2014

More from The Register

next story
Download alert: Nearly ALL top 100 Android, iOS paid apps hacked
Attack of the Clones? Yeah, but much, much scarier – report
NSA SOURCE CODE LEAK: Information slurp tools to appear online
Now you can run your own intelligence agency
Microsoft: Your Linux Docker containers are now OURS to command
New tool lets admins wrangle Linux apps from Windows
First in line to order a Nexus 6? AT&T has a BRICK for you
Black Screen of Death plagues early Google-mobe batch
Microsoft adds video offering to Office 365. Oh NOES, you'll need Adobe Flash
Lovely presentations... but not on your Flash-hating mobe
You stupid BRICK! PCs running Avast AV can't handle Windows fixes
Fix issued, fingers pointed, forums in flames
prev story

Whitepapers

Seattle children’s accelerates Citrix login times by 500% with cross-tier insight
Seattle Children’s is a leading research hospital with a large and growing Citrix XenDesktop deployment. See how they used ExtraHop to accelerate launch times.
How to determine if cloud backup is right for your servers
Two key factors, technical feasibility and TCO economics, that backup and IT operations managers should consider when assessing cloud backup.
Getting started with customer-focused identity management
Learn why identity is a fundamental requirement to digital growth, and how without it there is no way to identify and engage customers in a meaningful way.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Beginner's guide to SSL certificates
De-mystify the technology involved and give you the information you need to make the best decision when considering your online security options.