Feeds

App Engine: Google's deepest secrets as a service

The software scales. But will the Google rulebook?

Intelligent flash storage arrays

An open source project in Google clothing

The Datastore API and other proprietary Google APIs present an additional problem. Because Google's infrastructure is closed source, you may be forced to rewrite your application if you ever move it off App Engine. "App Engine is one of the only platform-as-a-services out there that force you to use proprietary APIs," says Lucas Carlson, the founder of PHP Fog, a platform-as-a-service for PHP developers. "You do have to use their APIs very strictly."

Yes, databases like MongoDB and HBase are similar to BigTable, but there are still differences. If you move your application to one of these platforms, says Lynch, "you're not just going to take the WAR file and drop it in and have it run. There is going to be a little bit of moving things about. But the concepts shouldn't change, the basic way your app is laid out."

Dwight Merriman, who built MongoDB, goes a little further. "The APIs to the App Engine Datastore are different from the APIs to, say, HBase," he says. "The data models are similar, but you would have to recode. And in addition, you're probably using this whole API set that's App Engine-specific. So, if you write an App Engine app, it's going to be fairly difficult to port it off of App Engine. That's one of the problems of platform-as-a-service."

Lynch says he looks forward to the day when the non-relational world gets something like the JDBC (Java Database Connectivity) API , an API thats let you easily move between disparate databases. And such work is already underway, with projects like Django-norel.

But for App Engine users, the most important independent projects lie elsewhere. There are not one but two efforts to create an open source version of App Engine: App Scale and TyphoonAE. Since Google's underlying software is proprietary, these can never provide an exact replication, but each seeks to expose the same APIs so you can readily move entire applications from App Engine to other cloud services such as Amazon EC2 – or to your local servers. TyphoonAE replicates App Engine's Python side, while App Scale aims for Python and Java, and even Go.

The App Scale project was started at the University of California, Santa Barbara, by professors Chandra Krintz and Rich Wolski. Previously, Wolski had founded the Eucalyptus project, an effort to create an open source version of Amazon's cloud. "Eucalyptus has really taken off, building an open source community," Krintz tells us. "Taking that as inspiration, we decided to build an open source implementation of the APIs for Google App Engine as Eucalyptus did for Amazon's APIs. That way, we could have App Engine applications running on private clusters and experiment with new clusters and – this being a research platform – investigate with how cloud applications behave.

"We just implemented each API incrementally and, over time, developed a full-fledged distributed system such that you can now run your App Engine application either locally or on Google's resources and not know the difference. You don't have to change your application."

Krintz acknowledges that App Scale will never scale quite like App Engine. But she's confident it will eventually perform well enough for the enterprise. A new release is due later this summer. "With this next release, we're quite mature," says Krintz. "There are still bugs to be worked out, but it's pretty stable, and it's really scalable."

Chandra Krintz

Chandra Krintz

If the world is to run App Engine applications outside of App Engine, projects like this are the only option. Sean Lynch makes it clear that Google has no intention of offering its own "on-premise" version of App Engine, seeing limited value in platforms such as App Scale. "They may make deployment internally easier. You may be able to the same tool integration to get an app up and running," he says. "But there's still someone who's doing resource budgeting and making sure you have a capacity plan for the next year, worrying about versions and operating systems. If there's someone still worrying about that, you're not getting the benefit of a cloud-based platform-as-a-service."

When they started the App Scale project, after applying for a Google Research Award, Wolski and Krintz received seed money from the company. And on from the App Engine mailing list, when App Scale's engineers have questions about the public side of App Engine, Googlers are happy to help. After Google introduced the Go SDK for App Engine, an App Scale researcher asked for the SDK source code, and Google was happy to provide it. But the company had no interest in helping the open sourcers rebuild the Go incarnation of App Engine. And none too surprisingly, Google provides precious little information about the inner workings of App Engine.

"We had to infer everything," Krintz says. "They helped us in no way to understand any of the APIs. We had to emulate and run our own programs on their machines so that we could then implement the same functionality."

If it wasn't already obvious: Google believes everything should happen on the web.

Security for virtualized datacentres

More from The Register

next story
Just don't blame Bono! Apple iTunes music sales PLUMMET
Cupertino revenue hit by cheapo downloads, says report
The DRUGSTORES DON'T WORK, CVS makes IT WORSE ... for Apple Pay
Goog Wallet apparently also spurned in NFC lockdown
Cray-cray Met Office spaffs £97m on VERY AVERAGE HPC box
Only 250th most powerful in the world? Bring back Michael Fish
Microsoft brings the CLOUD that GOES ON FOREVER
Sky's the limit with unrestricted space in the cloud
'ANYTHING BUT STABLE' Netflix suffers BIG Europe-wide outage
Friday night LIVE? Nope. The only thing streaming are tears down my face
Google roolz! Nest buys Revolv, KILLS new sales of home hub
Take my temperature, I'm feeling a little bit dizzy
IBM, backing away from hardware? NEVER!
Don't be so sure, so-surers
prev story

Whitepapers

Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
The hidden costs of self-signed SSL certificates
Exploring the true TCO for self-signed SSL certificates, including a side-by-side comparison of a self-signed architecture versus working with a third-party SSL vendor.