Original URL: http://www.theregister.co.uk/2011/06/07/inside_google_app_engine/

App Engine: Google's deepest secrets as a service

The software scales. But will the Google rulebook?

By Cade Metz

Posted in Cloud, 7th June 2011 03:50 GMT

Google will never open source its back end. You'll never run the Google File System or Google MapReduce or Google BigTable on your own servers. Except on the rarest of occasions, the company won't even discuss the famously distributed software that underpins its sweeping collection of web services.

But if you like, you can still run your own applications atop GFS and MapReduce and BigTable. With Google App Engine – the "platform cloud" the company floated in the spring of 2008 – anyone can hoist code onto Google's live infrastructure. App Engine aims to share the company's distributed computing expertise by way of an online service.

"Google has built up this infrastructure – a lot of distributed software, internal processes – for building-out and scaling applications. We don't have the luxury of slowly ramping something up: when we launch something, we have to get it out there and scale it very, very quickly," Google App Engine product manager Sean Lynch recently told The Register.

"We decided we could take a lot of this infrastructure and expose it in a way that would let third-party developers use it – leverage the knowledge and techniques we have built up – to basically simplify the entire process of building their own web apps: building them, managing them once they're up there, and scaling them once they take off."

For some coders, it's an appealing proposition, not only because the Google back end has a reputation few others can match – a reputation fueled in part by the company's reluctance to discuss particulars – but also because App Engine completely removes the need to run your own infrastructure. As a platform cloud, App Engine goes several steps beyond an "infrastructure cloud" à la Amazon EC2. It doesn't give you raw virtual machines. It gives you APIs, and once you code to these APIs, the service takes care of the rest.

"When you're one guy trying to run a startup, you have to do absolutely everything. I've done everything from managing machines to writing code to marketing," says Jeff Schnitzer, a former senior engineer in EA's online games division who's now using App Engine to build an online-dating application. "That's one of the reasons why I've been drawn to App Engine. It eliminates entire classes of job descriptions, from sys admin to DBA."

The rub is that in order to benefit from this automation, you have to play by a strict Google rulebook. All applications must be built with Python, Java, or one of a handful of other supported languages, including Google's new-age Go programming language, which was just added to the list. And even within these languages, there are limits on the libraries and frameworks you can use, the way you handle data, and the duration of your processes.

Google is loosening some restrictions as the service matures. But inherently, App Engine requires a change of mindset. "You have to throw away a lot of what you know and basically write for Google's model: small instances, faster start, completely new data storage. It's a completely different beast," says Matt Cooper, a developer and entrepreneur who just recently started using the service. And because Google jealously guards the secrets of its infrastructure, anyone who builds an application atop App Engine will face additional hurdles if they ever decide to move the app elsewhere.

All of which makes Google App Engine a particularly fascinating case study. Much like in other markets, Google is promising you an added payoff if you seriously change the way you've done things in the past – and if you put a hefty amount of trust on its servers. Many have already embraced the proposition – App Engine serves more than a billion and a half page views a day, and 100,000 coders access the online console each month – but it's yet to be seen whether the service has a future in the mainstream.

App Engine Logo

Google App Engine

We know that Google has a knack for scaling web applications. "Google employees are very good about not talking about their proprietary [infrastructure], but they do give you qualitative feel for it," says Dwight Merriman, cofounder of MongoDB, the distributed database that seeks to solve many of the same scaling issues as the Google back end. What we don't know is whether the Google ethos can scale into the enterprise.

Google believes it can. Lynch calls App Engine a "long-term business", and later this year, the service will exit a three-year beta period, brandishing new enterprise-centric terms of service. Proof of the service's viability, the company believes, lies in the track record of the Google infrastructure. "This is a service that's a tremendous differentiator for us, " says Lynch. "We can see it being valuable for other people to use it as well."

Lazy developers not allowed

The restrictions that App Engine places on third-party coders are much like those that Google places on its own developers. Or so it seems. Recently, when we sat down with Google's Sean Lynch (who has since left the company) and senior App Engine product manager Gregory D'alesandre, they fielded most questions about the ins and outs of App Engine, but they drew the line at discussing the particulars of the Google infrastructure. Like Amazon – its cloud rival – Google is typically reticent to discuss its internal operations.

But by all accounts, the App Engine rulebook mimics Google's internal rulebook. "[Google App Engine] has the same limitations that even Google engineers complain about around their internal development and deployment," says Jason Hoffman, chief technology officer of Joyent, the San Francisco-based startup that offers both infrastructure-cloud and platform-cloud services. But these limits are in place for good reason.

Two years ago, at a conference in San Francisco, Google's Vijay Gill – who oversaw the company's network and data-center infrastructure at the time – briefly described the tight restrictions placed on Google's internal app developers. "Any idiot can build atoms together and then create this vast infrastructure. The question is: how do you actually get the applications to use the infrastructure? How do you distribute it? How do you optimize it? That's the hard part. To do that, you require an insane amount of force of will," Gill said.

"We have a set of primitives, if you would, that takes those collections of atoms - those data centers, those networks - that we've built, and then they abstract that entire infrastructure out as a set of services - some of the public ones are GFS obviously, BigTable, MapReduce." GFS is the company's distributed file system, BigTable its distributed database, MapReduce its distributed number-crunching platform.

Vijay Gill

Vijay Gill

That force of will is required to ensure that developers stick to the rulebook. "People are lazy. They say 'I don't want to design my applications into this confined space.' And it is a confined space. So you need a large force of will to get people to do that," Gill said.

In building for these "confined spaces", Google ensures that applications can readily scale, but it also allows application developers to operate without worrying about the particulars of the underlying infrastructure. "If we make a minor change to, say, disk storage to get a three per cent gain, and we roll that out to the GFS library, suddenly the entire base of applications stored on GFS sees that gain," Gill said.

App Engine brings this setup to the masses. "App Engine came out of the philosophy of internal development at Google. You don't want to have to worry about scale once scale hits you, because then it's too late. At Google, it's something we always plan for. You don't rewrite your application once it starts to scale. You write it so it can scale up front," D'alesandre says.

"But also – and this is something else that came from Google internally – people hate carrying pagers. No one likes to be on call for things. You try to build on services that you know someone else it taking care for you. That's where App Engine came from. All of the people building on top of it? We're essentially carrying the pagers for them."

But, yes, if you want Google to carry your pager, you too must write for those confined spaces.

Er, where's MySQL?

For the average developer, one of the biggest challenges is writing for App Engine's Datastore API. On App Engine, you can't write to the file system. You have to use the Datastore API, the Memcache API, or other services provided by Google, and the Datastore API – based on BigTable – uses a data model vastly different from the relational databases most coders are used to.

"It takes some wrestling from a design standpoint, because it tends to want you to design your back end in a de-normalized sort of way," says Will Merydith, who's using App Engine to build a gaming platform known as SuperKablamo. "Unlike a MySQL environment, where you can do all these really interesting joins, you can't really do that with Datastore."

But you can scale. And with the rise of open source databases such as MongoDB and HBase, which grew out of Google's BigTable research paper, many developers are already adjusting to the non-relational mindset. "Two or three years ago, people said: 'Moving off relational? That's a pretty high hurdle. I don't think that's going to happen'. But now it's definitely happening," says Dwight Merriman, the CEO and cofounder of 10gen, MongoDB's chief steward.

"There is a pretty big hurdle to clear," he says. "Everybody learned relational in school and there's a lot of tooling around it that already exists and there's a lot of legacy code that uses it. ... But what has helped people get over the hurdle is the scaling imperative. Computer architectures are changing, and in the cloud we need horizontal scaling. Really, there's no choice."

Sean Lynch believes that if you've used MongoDB or HBase, you'll take to the Datastore API rather easily. "While it is close to a couple of competitors out there in terms of functionality, it's not identical, so there are things that would be specific to App Engine when you're using something like the Datastore," he says. "But it wouldn't be a big conceptual leap if you were moving from MongoDB or HBase to App Engine. The model should feel very much the same."

But even if you're steeped in this sort of non-relational data model, App Engine takes some getting used to. You can't throw any old code onto App Engine. You can use Python and many languages that compile to Java byte code, including Java, JRuby, Jython, Scala, and Groovy. And now there's an SDK for Go. But that's it. And you can only use certain Python libraries and only certain Java byte code languages, libraries, and frameworks.

What's more, App Engine restricts you to a tight sandbox. Your application can only access other net machines through App Engine's URL fetch, email, and XMPP services. Other machines can only connect to your application via HTTP requests on the standard ports. And an application code only runs in response to a web request, a queued task, or a scheduled task. An app must return a response to a web request within 30 seconds, while tasks (queued or scheduled) have ten minutes to complete.

These restrictions are in place, Lynch says, so that applications can scale, but also for security reasons. "There are some limitations around how long the code will run, and you can't use sockets. If you're used to just dropping in some code to call some socket somewhere, that may not work," he says. "But this is a balancing act. Do you give as much flexibility as possible or do you bake in security from the get-go? It's much easier to build in the security from the base layer and build from there, than it is to try and add the security layer later.

"The other nice thing is that we've been able to scale up over the last couple of years with a very high confidence that things aren't affecting each other when they're scaling. When they're moving around, there are not ways you could touch someone else's memory."

He adds that, as this confidence builds, Google is "peeling back" some restrictions. With App Engine version 1.4.3, it added the ability to do concurrent requests with Java applications (concurrency for Python is on the way). And in version 1.5.0, it introduced "Backends" – long-running, high-memory instances. These have a 24-hour request deadline, and they can use between 128MB and 1GB of memory and a proportional amount of CPU power.

Dwight Merriman

Dwight Merriman

Previously, says developer Jeff Schnitzer, he couldn't run an in-memory index on the service. "You have these little instances that are limited to about 100MB of RAM, and you don't really have a lot of control over how many of them are, and you can't individually address them. For a whole variety of reasons, you just don't have access to large quantities of RAM," he explains. But with Backends, such an index is possible.

The added catch is that – at least for now – Backends are too expensive for Schnitzer's purposes. He has already moved his in-memory index to Amazon. For Schnitzer, at least, App Engine's limitations ensure that applications often spill over into other clouds – or local servers. "I think this is something that anyone who spends a lot of time with app engine realizes," he says. "App Engine isn't really a complete set of tools. You have to have parts of an app in App Engine and parts in other places."

An open source project in Google clothing

The Datastore API and other proprietary Google APIs present an additional problem. Because Google's infrastructure is closed source, you may be forced to rewrite your application if you ever move it off App Engine. "App Engine is one of the only platform-as-a-services out there that force you to use proprietary APIs," says Lucas Carlson, the founder of PHP Fog, a platform-as-a-service for PHP developers. "You do have to use their APIs very strictly."

Yes, databases like MongoDB and HBase are similar to BigTable, but there are still differences. If you move your application to one of these platforms, says Lynch, "you're not just going to take the WAR file and drop it in and have it run. There is going to be a little bit of moving things about. But the concepts shouldn't change, the basic way your app is laid out."

Dwight Merriman, who built MongoDB, goes a little further. "The APIs to the App Engine Datastore are different from the APIs to, say, HBase," he says. "The data models are similar, but you would have to recode. And in addition, you're probably using this whole API set that's App Engine-specific. So, if you write an App Engine app, it's going to be fairly difficult to port it off of App Engine. That's one of the problems of platform-as-a-service."

Lynch says he looks forward to the day when the non-relational world gets something like the JDBC (Java Database Connectivity) API , an API thats let you easily move between disparate databases. And such work is already underway, with projects like Django-norel.

But for App Engine users, the most important independent projects lie elsewhere. There are not one but two efforts to create an open source version of App Engine: App Scale and TyphoonAE. Since Google's underlying software is proprietary, these can never provide an exact replication, but each seeks to expose the same APIs so you can readily move entire applications from App Engine to other cloud services such as Amazon EC2 – or to your local servers. TyphoonAE replicates App Engine's Python side, while App Scale aims for Python and Java, and even Go.

The App Scale project was started at the University of California, Santa Barbara, by professors Chandra Krintz and Rich Wolski. Previously, Wolski had founded the Eucalyptus project, an effort to create an open source version of Amazon's cloud. "Eucalyptus has really taken off, building an open source community," Krintz tells us. "Taking that as inspiration, we decided to build an open source implementation of the APIs for Google App Engine as Eucalyptus did for Amazon's APIs. That way, we could have App Engine applications running on private clusters and experiment with new clusters and – this being a research platform – investigate with how cloud applications behave.

"We just implemented each API incrementally and, over time, developed a full-fledged distributed system such that you can now run your App Engine application either locally or on Google's resources and not know the difference. You don't have to change your application."

Krintz acknowledges that App Scale will never scale quite like App Engine. But she's confident it will eventually perform well enough for the enterprise. A new release is due later this summer. "With this next release, we're quite mature," says Krintz. "There are still bugs to be worked out, but it's pretty stable, and it's really scalable."

Chandra Krintz

Chandra Krintz

If the world is to run App Engine applications outside of App Engine, projects like this are the only option. Sean Lynch makes it clear that Google has no intention of offering its own "on-premise" version of App Engine, seeing limited value in platforms such as App Scale. "They may make deployment internally easier. You may be able to the same tool integration to get an app up and running," he says. "But there's still someone who's doing resource budgeting and making sure you have a capacity plan for the next year, worrying about versions and operating systems. If there's someone still worrying about that, you're not getting the benefit of a cloud-based platform-as-a-service."

When they started the App Scale project, after applying for a Google Research Award, Wolski and Krintz received seed money from the company. And on from the App Engine mailing list, when App Scale's engineers have questions about the public side of App Engine, Googlers are happy to help. After Google introduced the Go SDK for App Engine, an App Scale researcher asked for the SDK source code, and Google was happy to provide it. But the company had no interest in helping the open sourcers rebuild the Go incarnation of App Engine. And none too surprisingly, Google provides precious little information about the inner workings of App Engine.

"We had to infer everything," Krintz says. "They helped us in no way to understand any of the APIs. We had to emulate and run our own programs on their machines so that we could then implement the same functionality."

If it wasn't already obvious: Google believes everything should happen on the web.

The price of the platform

The question is a familiar one: will the rest of the world buy into Google belief system? Many developers already have. "I've been wanting a platform-as-a service for a while, with an eye towards not doing any more IT work, not installing servers and disk drives and all that. For me, those days are over," says developer Matt Cooper. He now builds web applications in Python atop App Engine – even though he's a former Microsoft system engineer with a history coding of in .NET, which serves as the basis for Microsoft' platform cloud: Windows Azure. Before using App Engine, Cooper had never coded in Python.

Robert Kluin also had no Python experience, and had spent years working with Microsoft SQL Server and MySQL. But he too moved to Python on App Engine. "The thing that caught my attention was the distributed nature. I've done consulting work for oil and gas companies, managing clusters of SQL servers and things like that, and it's always a pain in the butt. App Engine is handling those kinds of features for us," he says.

"It truly is a platform as a service. There is no low-level messing around setting up databases, configuring your application servers or memcaching or anything of that nature. You can simply use their SDK, their API, and you have all those types of things at your disposal."

Six months ago, Kluin says, he had a long list of complaints about the service. But since then, Google has addressed most all of them. The company recently introduced a high-replication version of its Datastore, he points out, and it's now the default. This adds latency for data writes, and it doesn't quite offer the same consistency. But it increases the number of Google data centers that keep real-time replicas of your data, providing additional fallbacks in the event of Datastore issues, including latency spikes.

Kluin acknowledges the restrictions that App Engine places on developers, but he doesn't see them as a big issue. "It takes some extra effort, but you can find solutions," he says. "And the restrictions are necessary."

But others argue that because of the restrictions, mainstream developers are unlikely to take to the service. "It works well for Python people for certain use cases," says PHP Fog's Lucas Carlson. "These are generally the type of people who don't mind working around things and really enjoy tinkering. People who use Python are by nature tinkerers, and while Python is a popular language, the overall number of Python people out there is smaller than other major languages."

And Carlson believes that Google's proprietary APIs can be a significant burden as well. "Lock-in can become a real issue, and I believe most enterprises will not make this gamble," he says. "When you program to Google's APIs, you can't run that anywhere else. It's not like you can change your mind and choose another provider down the road." Of course, that doesn't take into account independent projects like App Scale.

Many of those proprietary APIs are merely a means of hooking into other parts of Google's platform. There's an API, for instance, for authenticating Google accounts. Google is now offering App Engine as a means of building applications that dovetail with its own Google Apps suite, and for those exploring such applications, App Engine makes perfect sense. But this too is a limited audience.

Google acknowledges that some businesses have been reluctant to invest in the platform, but the company indicates this is mainly because the service is still in "preview" mode. Later this year, App Engine will officially graduate to "shipping" status, providing all paid users a 99.95 per cent uptime service level agreement, operational and developer support, billing via invoice, and terms of service designed specifically for businesses.

As part of this, Google is also changing the App Engine pricing structure. The company will still offer a free version of the service for those willing to stay under certain quotas, but for larger users it will charge for instance use rather than CPU use as it does now. In announcing the new pricing, Google said it was "easier to understand" and "in line with the value App Engine provides".

Jeff Schnitzer

Jeff Schnitzer

The service's new business suit will no doubt attract more companies to App Engine. The new pricing model is likely to raise prices in many cases – Google has said that the previous model would not allow it to sustain a business – but it's reasonable compared to other services out there. The trouble with the new model is that so many people have already built their applications for the old model – and now it's changing beneath them. Many have vehemently complained.

Developer Jeff Schnitzer takes a philosophical stance on the matter. "They've changed the whole pricing model, and there's a bunch of uncertainties in there," he says. "Basically, what they announced was: 'Pricing is going to change, but we really have no idea how much it's going to change'. I'm trying to withhold judgment until they get everything nailed down." But whatever the end result of the new model, Google's change highlights the unique nature of a platform cloud. If you code to Google's cloud, your application will automatically scale, and you'll automatically benefit from any improvements Google's makes to its own infrastructure. But, at least on some level, you're also beholden to Google.

The good news is that Google clearly realizes that it needs to better accomodate developers. The company has already made a few changes to its new pricing model, trying to appease unhappy devs, just as – with new enhancements like Backends and concurrent Java requests – it's easing the restrictions developers have complained about. Google won't open source its infrastructure. But it certainly wants you to use it. ®