Feeds

App Engine: Google's deepest secrets as a service

The software scales. But will the Google rulebook?

Protecting against web application threats using SSL

Er, where's MySQL?

For the average developer, one of the biggest challenges is writing for App Engine's Datastore API. On App Engine, you can't write to the file system. You have to use the Datastore API, the Memcache API, or other services provided by Google, and the Datastore API – based on BigTable – uses a data model vastly different from the relational databases most coders are used to.

"It takes some wrestling from a design standpoint, because it tends to want you to design your back end in a de-normalized sort of way," says Will Merydith, who's using App Engine to build a gaming platform known as SuperKablamo. "Unlike a MySQL environment, where you can do all these really interesting joins, you can't really do that with Datastore."

But you can scale. And with the rise of open source databases such as MongoDB and HBase, which grew out of Google's BigTable research paper, many developers are already adjusting to the non-relational mindset. "Two or three years ago, people said: 'Moving off relational? That's a pretty high hurdle. I don't think that's going to happen'. But now it's definitely happening," says Dwight Merriman, the CEO and cofounder of 10gen, MongoDB's chief steward.

"There is a pretty big hurdle to clear," he says. "Everybody learned relational in school and there's a lot of tooling around it that already exists and there's a lot of legacy code that uses it. ... But what has helped people get over the hurdle is the scaling imperative. Computer architectures are changing, and in the cloud we need horizontal scaling. Really, there's no choice."

Sean Lynch believes that if you've used MongoDB or HBase, you'll take to the Datastore API rather easily. "While it is close to a couple of competitors out there in terms of functionality, it's not identical, so there are things that would be specific to App Engine when you're using something like the Datastore," he says. "But it wouldn't be a big conceptual leap if you were moving from MongoDB or HBase to App Engine. The model should feel very much the same."

But even if you're steeped in this sort of non-relational data model, App Engine takes some getting used to. You can't throw any old code onto App Engine. You can use Python and many languages that compile to Java byte code, including Java, JRuby, Jython, Scala, and Groovy. And now there's an SDK for Go. But that's it. And you can only use certain Python libraries and only certain Java byte code languages, libraries, and frameworks.

What's more, App Engine restricts you to a tight sandbox. Your application can only access other net machines through App Engine's URL fetch, email, and XMPP services. Other machines can only connect to your application via HTTP requests on the standard ports. And an application code only runs in response to a web request, a queued task, or a scheduled task. An app must return a response to a web request within 30 seconds, while tasks (queued or scheduled) have ten minutes to complete.

These restrictions are in place, Lynch says, so that applications can scale, but also for security reasons. "There are some limitations around how long the code will run, and you can't use sockets. If you're used to just dropping in some code to call some socket somewhere, that may not work," he says. "But this is a balancing act. Do you give as much flexibility as possible or do you bake in security from the get-go? It's much easier to build in the security from the base layer and build from there, than it is to try and add the security layer later.

"The other nice thing is that we've been able to scale up over the last couple of years with a very high confidence that things aren't affecting each other when they're scaling. When they're moving around, there are not ways you could touch someone else's memory."

He adds that, as this confidence builds, Google is "peeling back" some restrictions. With App Engine version 1.4.3, it added the ability to do concurrent requests with Java applications (concurrency for Python is on the way). And in version 1.5.0, it introduced "Backends" – long-running, high-memory instances. These have a 24-hour request deadline, and they can use between 128MB and 1GB of memory and a proportional amount of CPU power.

Dwight Merriman

Dwight Merriman

Previously, says developer Jeff Schnitzer, he couldn't run an in-memory index on the service. "You have these little instances that are limited to about 100MB of RAM, and you don't really have a lot of control over how many of them are, and you can't individually address them. For a whole variety of reasons, you just don't have access to large quantities of RAM," he explains. But with Backends, such an index is possible.

The added catch is that – at least for now – Backends are too expensive for Schnitzer's purposes. He has already moved his in-memory index to Amazon. For Schnitzer, at least, App Engine's limitations ensure that applications often spill over into other clouds – or local servers. "I think this is something that anyone who spends a lot of time with app engine realizes," he says. "App Engine isn't really a complete set of tools. You have to have parts of an app in App Engine and parts in other places."

Choosing a cloud hosting partner with confidence

More from The Register

next story
Wanna keep your data for 1,000 YEARS? No? Hard luck, HDS wants you to anyway
Combine Blu-ray and M-DISC and you get this monster
US boffins demo 'twisted radio' mux
OAM takes wireless signals to 32 Gbps
Google+ GOING, GOING ... ? Newbie Gmailers no longer forced into mandatory ID slurp
Mountain View distances itself from lame 'network thingy'
Apple flops out 2FA for iCloud in bid to stop future nude selfie leaks
Millions of 4chan users howl with laughter as Cupertino slams stable door
Students playing with impressive racks? Yes, it's cluster comp time
The most comprehensive coverage the world has ever seen. Ever
Run little spreadsheet, run! IBM's Watson is coming to gobble you up
Big Blue's big super's big appetite for big data in big clouds for big analytics
Seagate's triple-headed Cerberus could SAVE the DISK WORLD
... and possibly bring us even more HAMR time. Yay!
prev story

Whitepapers

Secure remote control for conventional and virtual desktops
Balancing user privacy and privileged access, in accordance with compliance frameworks and legislation. Evaluating any potential remote control choice.
WIN a very cool portable ZX Spectrum
Win a one-off portable Spectrum built by legendary hardware hacker Ben Heck
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
The next step in data security
With recent increased privacy concerns and computers becoming more powerful, the chance of hackers being able to crack smaller-sized RSA keys increases.