Original URL: http://www.theregister.co.uk/2009/02/24/the_meta_cloud/

The Meta Cloud - Flying data centers enter fourth dimension

Or do they?

By Cade Metz

Posted in Developer, 24th February 2009 01:51 GMT

Just when you thought the cloud metaphor couldn't stretch any further, it has. The tech world is still coming to terms with the rather abstract idea of a data center that floats on air, and now more than a few free-thinking web startups are hoping to abstract the abstraction. Beware of The Meta Cloud.

Today, several high-flying net outfits operate so-called infrastructure clouds - online services that offer processing power, storage, and other distributed resources. Amazon is the obvious example, with its Amazon Web Services (AWS), but others serve up much the same thing, including Flexiscale, GoGrid, and Joyent.

In theory, these clouds give you instant access to an infinite array of compute power. You can grab more cycles, more storage, and more bandwidth whenever you like. But in real life, no resource is infinite. And clouds have been known to vanish from time to time.

What's more, these are far from public resources. Conservative thinkers may be wary of entrusting their entire web infrastructure to a single operation - e.g., Amazon.

So, it comes as no surprise that would-be web visionaries have floated the idea of a meta-cloud - a cloud-of-clouds, if you will. With this extended metaphor, you could access multiple clouds from a single web interface, seamlessly moving tasks from one to the other. If one disappears, another is there. If one disappoints - prices rise or standards fall - you have an out. Resources are still less than infinite, but there are more of them.

At least, that's the pitch.

Meta-Startup

Alex Castro and the cloud go way back. At New York's Cornell University in the mid-90s, he shared a distributed-systems lab with latter-day cloud saint Werner Vogels. And once Vogels made his pilgrimage to the West Coast, conjuring a new cloud for web demigod Jeff Bezos, Castro joined him again, lending a hand with the birth of Amazon Web Services.

Naturally, when Castro started his own web outfit - a video delivery operation dubbed Delve Networks - he decided that Amazon should run at least a portion of his online infrastructure. And just as naturally, he insists that Amazon's Elastic Compute Cloud is a godsend.

"We do a lot of heavy computational work to process the video," Castro says. "We do speech recognition and analytics and transcoding - computationally intensive things. So signing up for this theoretically infinite amount of computing capacity just made sense. As our customers need more transcoding, we just spin up more machines, and when they need less transcoding, we just tear those machines down."

But Delve was among those who saw Amazon's cloud disappear for an eight-hour stretch last summer. And Castro can't help but wonder what would happen if his company made a freakish Facebook-like rise into the web's top-ten properties. Is that the point where Amazon's resources would prove less than infinite?

In an ideal world, Delve and other web apps would run on multiple clouds. And that, Castro insists, is a business opportunity. "If I didn't have my hands full with my own company," he says, "I'd launch a new one that offered a cloud of clouds."

He's not alone. According to Intel cloud guru Jason Waxman, countless startups have approached Intel Capital - the company's investment arm - with the idea of a meta-cloud. One of these, Waxman says, was an existing cloud-management operation known as Elastra - though when we phoned the company, it played coy on the matter, saying that some sort of mystery announcement will arrive in the second quarter.

Meanwhile, one philosophical outfit has already launched a meta-cloud. At least, it thinks it has.

Meta-Cloud 1.0

In the beginning, Amazon Web Services lacked a slick web GUI, and in 2007, RightScale sprung up to provide one. Co-founded by Thorsten von Eicken - a professor in Cornell's computer science department in the days of Werner Vogels and Alex Castro - the Santa Barbara, California outfit calls its online service "a fully-automated cloud management platform."

But Amazon has since extended its own interface beyond the command line, and in September, RightScale expanded its service to juggle clouds from GoGrid and Flexiscale. It's still in its early days, says CEO Michael Crandell, but the goal is to create a service that allows businesses to pool infrastructure resources from multiple clouds.

"What we're trying to do is provide a portability layer, become a neutral cloud management platform where people can then consume resources from different clouds," Crandell tells The Reg.

"All of this is in flux - and there are lots of potential gotchas - but we want to alleviate things for people, to make it easier, if not totally seamless, to move a system from one cloud to another, to create an abstraction layer across some of the differences there."

At the moment, RightScale offers what it calls "server templates" that run on Amazon, GoGrid, Flexiscale, or so-called "internal clouds" built in private data centers using the open-source Eucalyptus platform. After sitting your web app atop these templates, you can - in theory - move it from one cloud to another.

This makes sense, Crandell says, because many businesses chafe at the idea of putting their entire infrastructure in the hands of one company. "Lock-in is a huge concern among customers. They may choose a single cloud for their main deployment, but they want a disaster recovery setup on another cloud. That way, they're covered if an entire cloud should go down.

"And if a cloud's business model should change, if the cloud should start charging more money, they now have an alternative. A multi-cloud setup offers a certain freedom."

Plus, Crandell explains, you can run a web app across multiple geographies as a way of boosting performance from region to region. Flexiscale's cloud hovers over the UK, and just this month RightScale expanded this meta-cloud to the new European incarnation of Amazon EC2.

The Real Thing

The question is how well this cloud migration actually works - whether the idea of a meta-cloud is ultimately more attractive than the real thing. Even Michael Crandell hints that RightScale's current setup is still finding its feet. And if you ask Flexiscale CEO Tony Lucas how well it works, he can't tell you - though he's one of RightScale's key partners.

That said, Lucas likes the idea of a meta-cloud, and he believes that this is where the world is floating. "It's something we've been talking up for a long time, and it's something we're a big fan of," he says. "We think it's the future, where we're going to have real success."

But a true cloud of clouds - where single interface allows for the instant transfer of tasks between services - is still a dream. Today, Lucas explains, you could move an ordinary website from, say, Amazon to Flexiscale with relative ease. But anything more complex would prove painfully difficult.

Operations like Amazon and Flexiscale - not to mention Joyent - bill their clouds as open operations. And they're certainly more open than so-called cloud platforms such as Google App Engine and the upcoming Microsoft Azure, where you have no choice but to use proprietary tools, and where there's no hope of moving your app elsewhere. But we've yet to see cloud-specific open standards. Amazon and Flexiscale still use proprietary interfaces.

"We use open standards in a lot of ways, but we all operate different services, and there's still a lot of work involved moving from one to another," Lucas says. "The easiest job is to move is something that is just going to process data and kick something out at the end of it. There's a lot less issues with IP addressing and DNS and latency and everything else. You just shove it wherever it needs to run and it runs.

"After that, it gets much more complicated."

Joyent CTO and co-founder Jason Hoffman says much the same thing. "Companies like RightScale raised millions of dollars based on being the web GUI for Amazon. Now Amazon has the web GUI, so of course they're going to have to respond by supporting multiple clouds," Hoffman says. "It's easy to grab extra resources from another cloud, but that doesn't mean your app is going to scale."

If you're running a processing job on Amazon but it requires a terabyte of data on Joyent, he insists, it "just won't work." You could always move your data as well - but you'll pay a hefty fee for that. These operations charge for moving data off the cloud as well as on. Amazon Web Services is a business, and to a certain extent, Amazon would prefer that you stayed on Amazon.

The Meta-Meta-Cloud

Tony Lucas has spent months calling for cloud open standards, and he now says that the several high-profile cloud operators are working closely to make them happen, hinting that some sort of announcement is imminent. But even with standards in place, you have to wonder if a meta-cloud makes sense.

Transfer costs will always be a problem. And then there's a larger conundrum.

When trumpeting the idea, Delve's Alex Castro and RightScale's Michael Crandell argue that the meta-cloud avoids "vendor lock-in" - market-speak for "putting all your eggs in one basket." But if you opt for RightScale's meta-cloud, aren't you then locked in to RightScale?

"I just don't see why you would want an extra level of abstraction," says Intel's Jason Waxman. "I don't see where it gets you."

To avoid meta-cloud lock-in, you could build a meta-meta-cloud. But then you're locked-in again. We could go on. But we won't.

The Meta Cloud is wonderful thought. But perhaps that's it. ®