Feeds

Google Go boldly goes where no code has gone before

How to build all the Google stuff Google won't talk about

Top 5 reasons to deploy VMware with Tegile

Every Google data center has a Chubby, and Heroku wanted one too.

Like the rest of Google's much admired back-end infrastructure, Chubby is decidedly closed source, so Heroku men Keith Rarick and Blake Mizerany built their own. Although they didn't have source code, they did have one of those secret-spilling Google research papers. And they had Go, Google's open source programming language for building systems like Google's.

Running its myriad online services atop a famously distributed back-end, splitting tasks into tiny pieces and spreading them across a vast network of machines, Google needs a way of controlling access to those machines – and that's Chubby. According to a 2006 Google research paper (PDF), there's a least one Chubby instance in each of the company's data centers, coordinating server access for its GFS distributed file system, its BigTable database, and its epic number-crunching platform, MapReduce.

Heroku runs a similarly distributed infrastructure behind its eponymous web service – a means of building, hosting, and readily scaling Ruby on Rails applications – and this too requires the sort of server-juggling you get from Chubby. "When you have a lot of distributed processes – lots of things going on, a network spanning a lot of machines – coordinating things can be tricky," Heroku system architect Keith Rarick tells The Register.

"If a machine goes down or you're having network problems where some machines can't talk to each other or the network is slow or you're getting packet loss, there are so many ways things that can fail. It can be really, really hard to get all those things right if you're doing ad hoc coordination."

Google Chubby

Only known image of Google Chubby

Using an algorithm called Paxos, Rarick says, Chubby almost magically simplifies this process. But first, you have to build the thing. And for that, Rarick turned to Go. Conceived by a trio of Google heavyweights – Unix founding father Ken Thompson, fellow Bell Labs Unix developer Rob Pike, and Robert Griesemer, who worked on the Java HotSpot compiler – Go is specifically designed for distributed systems.

"We realized that the kind of software we build at Google is not always served well by the languages we had available," Rob Pike tells us. "Robert Griesemer, Ken Thompson, and myself decided to make a language that would be very good for writing the kinds of programs we write at Google."

Released as an experimental language in late 2009 and now in production at Google, Go includes built-in mechanisms for running concurrent tasks. Using lightweight processes known as "goroutines", you can readily juggle multiple tasks within the same operating system thread or spread them across disparate threads, and all this is automatically scheduled by the Go runtime. The inherently concurrent setup was ideal for Heroku's Chubby mimic, an open source project known as Doozer.

"Go is very good at letting you have multiple points of control in a single program, and coordinating the synchronization and communication between those different points of control," Rarick explains. "That's something that's really useful in a project like [Doozer]. We have a bunch of different processes on different machines trying to talk to each other, and you have to make sure they all have a consistent view of the world."

Go not only provides the concurrency Doozer requires, it keeps memory usage to a minimum. It offers garbage collection, but it also frees you from thread overload. "We don't have to worry about how many threads we're running. We aren't constantly saying 'Are we running out of resources?'" Rarick explains. "We can have lots of goroutines instead, and that's built into the language."

Other languages provide somewhat similar mechanisms for concurrency – such as Erlang and Scala – but Go is designed to provide maximum efficiency and control, as well. "It comes from a line of systems programming language like C and C++, so it gives you the ability to really control the performance characteristics," Rarick says. "When it comes time to measure things and make sure they run fast, you have the flexibility to really get in there and do what you need. And when you figure out why your program is being slow, you really have the control you need to fix it."

Keith Rarick

Keith Rarick

It's a unique combination. "C gives you control, but it's not good for concurrency. It doesn't even give you garbage collection," he says. "Go gives you concurrency and garbage collection, but it still gives you control over memory layout and resource use."

Google won't say how Go is used inside its mystery data centers. But Doozer – officially released earlier this month and set to go live at Heroku – is a suitable poster child for the fledgling language. It's no coincidence that Robert Griesemer, one of the original Go architects, also worked on Chubby.

Providing a secure and efficient Helpdesk

More from The Register

next story
Preview redux: Microsoft ships new Windows 10 build with 7,000 changes
Latest bleeding-edge bits borrow Action Center from Windows Phone
Google opens Inbox – email for people too thick to handle email
Print this article out and give it to someone tech-y if you get stuck
Microsoft promises Windows 10 will mean two-factor auth for all
Sneak peek at security features Redmond's baking into new OS
UNIX greybeards threaten Debian fork over systemd plan
'Veteran Unix Admins' fear desktop emphasis is betraying open source
Google+ goes TITSUP. But WHO knew? How long? Anyone ... Hello ...
Wobbly Gmail, Contacts, Calendar on the other hand ...
DEATH by PowerPoint: Microsoft warns of 0-day attack hidden in slides
Might put out patch in update, might chuck it out sooner
Redmond top man Satya Nadella: 'Microsoft LOVES Linux'
Open-source 'love' fairly runneth over at cloud event
prev story

Whitepapers

Cloud and hybrid-cloud data protection for VMware
Learn how quick and easy it is to configure backups and perform restores for VMware environments.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.