App Engine: Google's deepest secrets as a service

An open source project in Google clothing

The Datastore API and other proprietary Google APIs present an additional problem. Because Google's infrastructure is closed source, you may be forced to rewrite your application if you ever move it off App Engine. "App Engine is one of the only platform-as-a-services out there that force you to use proprietary APIs," says Lucas Carlson, the founder of PHP Fog, a platform-as-a-service for PHP developers. "You do have to use their APIs very strictly."

Yes, databases like MongoDB and HBase are similar to BigTable, but there are still differences. If you move your application to one of these platforms, says Lynch, "you're not just going to take the WAR file and drop it in and have it run. There is going to be a little bit of moving things about. But the concepts shouldn't change, the basic way your app is laid out."

Dwight Merriman, who built MongoDB, goes a little further. "The APIs to the App Engine Datastore are different from the APIs to, say, HBase," he says. "The data models are similar, but you would have to recode. And in addition, you're probably using this whole API set that's App Engine-specific. So, if you write an App Engine app, it's going to be fairly difficult to port it off of App Engine. That's one of the problems of platform-as-a-service."

Lynch says he looks forward to the day when the non-relational world gets something like the JDBC (Java Database Connectivity) API , an API thats let you easily move between disparate databases. And such work is already underway, with projects like Django-norel.

But for App Engine users, the most important independent projects lie elsewhere. There are not one but two efforts to create an open source version of App Engine: App Scale and TyphoonAE. Since Google's underlying software is proprietary, these can never provide an exact replication, but each seeks to expose the same APIs so you can readily move entire applications from App Engine to other cloud services such as Amazon EC2 – or to your local servers. TyphoonAE replicates App Engine's Python side, while App Scale aims for Python and Java, and even Go.

The App Scale project was started at the University of California, Santa Barbara, by professors Chandra Krintz and Rich Wolski. Previously, Wolski had founded the Eucalyptus project, an effort to create an open source version of Amazon's cloud. "Eucalyptus has really taken off, building an open source community," Krintz tells us. "Taking that as inspiration, we decided to build an open source implementation of the APIs for Google App Engine as Eucalyptus did for Amazon's APIs. That way, we could have App Engine applications running on private clusters and experiment with new clusters and – this being a research platform – investigate with how cloud applications behave.

"We just implemented each API incrementally and, over time, developed a full-fledged distributed system such that you can now run your App Engine application either locally or on Google's resources and not know the difference. You don't have to change your application."

Krintz acknowledges that App Scale will never scale quite like App Engine. But she's confident it will eventually perform well enough for the enterprise. A new release is due later this summer. "With this next release, we're quite mature," says Krintz. "There are still bugs to be worked out, but it's pretty stable, and it's really scalable."

Chandra Krintz

If the world is to run App Engine applications outside of App Engine, projects like this are the only option. Sean Lynch makes it clear that Google has no intention of offering its own "on-premise" version of App Engine, seeing limited value in platforms such as App Scale. "They may make deployment internally easier. You may be able to the same tool integration to get an app up and running," he says. "But there's still someone who's doing resource budgeting and making sure you have a capacity plan for the next year, worrying about versions and operating systems. If there's someone still worrying about that, you're not getting the benefit of a cloud-based platform-as-a-service."

When they started the App Scale project, after applying for a Google Research Award, Wolski and Krintz received seed money from the company. And on from the App Engine mailing list, when App Scale's engineers have questions about the public side of App Engine, Googlers are happy to help. After Google introduced the Go SDK for App Engine, an App Scale researcher asked for the SDK source code, and Google was happy to provide it. But the company had no interest in helping the open sourcers rebuild the Go incarnation of App Engine. And none too surprisingly, Google provides precious little information about the inner workings of App Engine.

"We had to infer everything," Krintz says. "They helped us in no way to understand any of the APIs. We had to emulate and run our own programs on their machines so that we could then implement the same functionality."

If it wasn't already obvious: Google believes everything should happen on the web.