Original URL: http://www.theregister.co.uk/2009/11/18/google_app_engine_and_the_developer/

Google and the myth of the open cloud

The world's most closed open company

By Cade Metz

Posted in Developer, 18th November 2009 22:52 GMT

It is a truth universally acknowledged that if Google offers the world a web service, large numbers of people will convince themselves that it's superior to anything else they can get their hands on - and less likely to condemn them to some sort of Redmondian future in which a single corporation has them in a metaphorical vice grip.

So it is with Google App Engine, the 18-month-old service that lets outside developers build and run web apps on the company's very own distributed infrastructure. According to the market research types at Evans Data, developers everywhere are convinced that App Engine will overtake Amazon in the race into the so-called public cloud. They've even decided that the Google cloud is their best bet for avoiding the dreaded "vendor lock-in."

Google Cloud Evans Data

You know the mindset. Google says it's open, so it must be open. Google says that it has opened up more than one million lines of code, that it's hosting more than 150,000 open source projects, that its web browser is open, and that its mobile OS is open - so its cloud must be open too. Its cloud must be much more accommodating than Amazon's.

Except that it's not.

Google's famously distributed infrastructure is in no way open. Whereas the likes of Facebook and Yahoo! run much of their back-end setup on open source code, Google builds its own proprietary platforms - from the Google File System (GFS) to the number-crunching MapReduce platform to the BigTable distributed database - and these are jealously guarded. Google is reluctant to even talk about them.

What's more, these Googly platforms place extensive restrictions on what you can build atop them. This is true whether you're an internal Google developer or an outsider tapping the App Engine. There's good reason for this. The idea is to fit all applications to predefined templates that can run across the company's worldwide network of data centers, so that performance can scale up (near-)instantly as needed.

"The question is: how do you actually get the applications to use the infrastructure? How do you distribute it? How do you optimize it? That's the hard part. To do that you require an insane amount of force of will," Google senior manager of engineering and architecture Vijay Gill told a cloudy conference this summer.

"People are lazy. They say 'I don't want to design my applications into this confined space.' And it is a confined space. So you need a large force of will to get people to do that."

Google, skis, and snowmobiles

For Google App Engine, the result is that Google severely limits the dev tools you can use to build stuff. "With App Engine, you can't use all your favorite open source tools," Sebastian Stadil, founder of the Silicon Valley Cloud Computing Group, tells The Reg. For example, he says, you can't use Drupal, the open source content-management system. "There are libraries you can't use. If the CMS software you want requires such a library, you're stuck."

What's more, it doesn't accommodate many of the most common email libraries. "So if you use an open source component that sends emails, it'll break, and you have to rewrite yourself." Google has a white list of libraries that its platform likes, and if your libraries aren't on the list, either you can't use App Engine or you have to rewrite.

For this reason, Stadil argues, App Engine isn't something a large number of devs will flock to - at least not anytime soon. "The restrictions it imposes to ensure easy scaling require too much learning, and will take forever to be adopted in the enterprise," he says.

"App Engine is a ski resort that only lets you use skis. No snowboards, no snowmobile. Just skis. Skis are fine, and they can get you downhill, but people might want alternative."

He offers up Ruby on Rails - that web-happy programming language - as an another analogy. "Take Ruby on Rails, which is great for writing applications. It is still lagging in the enterprise, because [it] requires [so much] learning...by IT staffs," he says. But, he adds, "the young Stanford undergrads will love App Engine like they loved Ruby on Rails."

The irony is that you can't use Ruby on Rails with App Engine. It only does Python and Java.

ClosedTable

But the bigger issue is that if you build an App Engine app that makes heavy use of the GFS or BigTable, you're also in for some heavy rewrites if you ever want to move the thing off Google's cloud. Since those platforms are not open sourced, you can't install them on your own cloud - or on anyone else's.

When we brought up this, um, inconvenience at Google's developer conference earlier this year, vp of engineering Vic Gundotra brusquely waved the issue aside. Developers don't code to BigTable, they code to Google's API, he said, arguing that switching to another cloud is a piece of cake.

That may or may not be the case for Google's own developers - people intimately familiar with its proprietary platform - but it's certainly a stretch when it comes to everyone else. "You'll need to change your database to a relational model [if you move off the Google cloud]," says Stadil. "The data model and data processing are different."

Despite what those Evans Data developers are telling themselves, this is what you call "vendor lock-in." Yes, there's a similar problem over on Amazon's cloud, with its proprietary SimpleDB database. But Amazon now offers MySQL as well. And unlike the Google's App Engine, Amazon's cloud is a place where you can use whatever development tools you like.

"So you invest in good skis, and then travel to the Amazon jungle," says Stadil. "There, you can still move around in skis, but it is suboptimal. It would be better to have a snowmobile."

Now, you might argue that because it has opted for a so-called "platform cloud" - as opposed to Amazon's infrastructure cloud - you can scale your apps much more quickly. "Concerns about lock-in and lack of flexibility loom big and large," says Thorsten von Eicken, a distributed-systems guru who now serves as CTO of a cloud-happy outfit called RightScale. "But they're obviously going for large scale. They've had to constrain the environment pretty drastically in order to be able to do this kind of scaling."

But do we really know that App Engine scales any better than Amazon? "That's a good question," says von Eicken. Google says it does, so it must. Right?

Let's say it does scale unlike anything else on the planet. Google is asking you to stomach the dreaded lock-in to ensure that your app can handle becoming ridiculously popular. But if you become ridiculously popular, do you really want Google lock-in?

"In a way, what Google says is bite the bullet for scale on day one, and I find that a bitter pill to swallow," von Eicken says. "Why would I spend the effort to now if I don't even know if my app is going to succeed?"

Certainly, Google is more open than, say, Microsoft. But its core platforms are preternaturally closed. And even an ostensibly open project like Android isn't as open as it might seem. Google open sources (most) of the Android platform after the fact, but the code that actually ships on big-name handsets is built behind closed doors.

It's something worth remembering as Google readies another ostensibly open platform: the Chrome OS. Which brings us to another irony. As Google prepares to unveil the openness of the Chrome OS at its Mountain View headquarters tomorrow, it won't even speak to a certain news organization about the possibility of attending the event. A small thing, really - but indicative of something much larger. ®

Update

Google has phoned to say that it is not able to provide The Reg with an invite to the Chrome OS announcement. The company tells us there's not enough space.