Feeds

App Engine: Google's deepest secrets as a service

The software scales. But will the Google rulebook?

Mobile application security vulnerability report

An open source project in Google clothing

The Datastore API and other proprietary Google APIs present an additional problem. Because Google's infrastructure is closed source, you may be forced to rewrite your application if you ever move it off App Engine. "App Engine is one of the only platform-as-a-services out there that force you to use proprietary APIs," says Lucas Carlson, the founder of PHP Fog, a platform-as-a-service for PHP developers. "You do have to use their APIs very strictly."

Yes, databases like MongoDB and HBase are similar to BigTable, but there are still differences. If you move your application to one of these platforms, says Lynch, "you're not just going to take the WAR file and drop it in and have it run. There is going to be a little bit of moving things about. But the concepts shouldn't change, the basic way your app is laid out."

Dwight Merriman, who built MongoDB, goes a little further. "The APIs to the App Engine Datastore are different from the APIs to, say, HBase," he says. "The data models are similar, but you would have to recode. And in addition, you're probably using this whole API set that's App Engine-specific. So, if you write an App Engine app, it's going to be fairly difficult to port it off of App Engine. That's one of the problems of platform-as-a-service."

Lynch says he looks forward to the day when the non-relational world gets something like the JDBC (Java Database Connectivity) API , an API thats let you easily move between disparate databases. And such work is already underway, with projects like Django-norel.

But for App Engine users, the most important independent projects lie elsewhere. There are not one but two efforts to create an open source version of App Engine: App Scale and TyphoonAE. Since Google's underlying software is proprietary, these can never provide an exact replication, but each seeks to expose the same APIs so you can readily move entire applications from App Engine to other cloud services such as Amazon EC2 – or to your local servers. TyphoonAE replicates App Engine's Python side, while App Scale aims for Python and Java, and even Go.

The App Scale project was started at the University of California, Santa Barbara, by professors Chandra Krintz and Rich Wolski. Previously, Wolski had founded the Eucalyptus project, an effort to create an open source version of Amazon's cloud. "Eucalyptus has really taken off, building an open source community," Krintz tells us. "Taking that as inspiration, we decided to build an open source implementation of the APIs for Google App Engine as Eucalyptus did for Amazon's APIs. That way, we could have App Engine applications running on private clusters and experiment with new clusters and – this being a research platform – investigate with how cloud applications behave.

"We just implemented each API incrementally and, over time, developed a full-fledged distributed system such that you can now run your App Engine application either locally or on Google's resources and not know the difference. You don't have to change your application."

Krintz acknowledges that App Scale will never scale quite like App Engine. But she's confident it will eventually perform well enough for the enterprise. A new release is due later this summer. "With this next release, we're quite mature," says Krintz. "There are still bugs to be worked out, but it's pretty stable, and it's really scalable."

Chandra Krintz

Chandra Krintz

If the world is to run App Engine applications outside of App Engine, projects like this are the only option. Sean Lynch makes it clear that Google has no intention of offering its own "on-premise" version of App Engine, seeing limited value in platforms such as App Scale. "They may make deployment internally easier. You may be able to the same tool integration to get an app up and running," he says. "But there's still someone who's doing resource budgeting and making sure you have a capacity plan for the next year, worrying about versions and operating systems. If there's someone still worrying about that, you're not getting the benefit of a cloud-based platform-as-a-service."

When they started the App Scale project, after applying for a Google Research Award, Wolski and Krintz received seed money from the company. And on from the App Engine mailing list, when App Scale's engineers have questions about the public side of App Engine, Googlers are happy to help. After Google introduced the Go SDK for App Engine, an App Scale researcher asked for the SDK source code, and Google was happy to provide it. But the company had no interest in helping the open sourcers rebuild the Go incarnation of App Engine. And none too surprisingly, Google provides precious little information about the inner workings of App Engine.

"We had to infer everything," Krintz says. "They helped us in no way to understand any of the APIs. We had to emulate and run our own programs on their machines so that we could then implement the same functionality."

If it wasn't already obvious: Google believes everything should happen on the web.

Bridging the IT gap between rising business demands and ageing tools

More from The Register

next story
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
EU's top data cops to meet Google, Microsoft et al over 'right to be forgotten'
Plan to hammer out 'coherent' guidelines. Good luck chaps!
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
FLAPE – the next BIG THING in storage
Find cold data with flash, transmit it from tape
Seagate chances ARM with NAS boxes for the SOHO crowd
There's an Atom-powered offering, too
Gartner: To the right, to the right – biz sync firms who've won in a box to the right...
Magic quadrant: Top marks for, er, completeness of vision, EMC
prev story

Whitepapers

Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.