Original URL: http://www.theregister.co.uk/2013/07/24/cloudscaling_openstack_must_support_aws_api/

OpenStack faces 'irrelevance and death' without AWS compatibility

Cloudscaling's Bias has no bias about cloud API support

By Timothy Prickett Morgan

Posted in Cloud, 24th July 2013 22:55 GMT

Does the OpenStack cloud controller need to support the full set of APIs from Amazon Web Services and other public clouds? Randy Bias, the founder and CTO of OpenStack distie Cloudscaling thinks so. 

This week, as OpenStack celebrates its third birthday, Bias issued a call to arms to the OpenStack community to do what rival open source cloud controllers Eucalyptus and CloudStack are doing and embrace the full set of AWS APIs, as well as the ones created largely by Rackspace Hosting at the beginning of the OpenStack project that are not compatible with AWS.

If Amazon Web Services had decided to sell its homegrown server virtualization and cloud controller software as a retail product, like a camera or a bottle of soap, as well as its bespoke servers and storage arrays that make up its compute and storage clouds, then it is very likely that the Eucalyptus, CloudStack, and OpenStack alternatives would have never gotten off the ground, or at the very least might not be flying quite so high.

Despite many of the limitations in the AWS infrastructure cloud, and despite the desire to offer different features in a cloud than those offered by AWS, Bias has passionately argued in a blog post that the OpenStack community has to return to the roots of NASA's original Nova compute controller – created before the space agency aligned with Rackspace to create OpenStack – and get AWS compatibility back into OpenStack.

And, if that were not enough of a tall order, Bias goes on to argue that OpenStack then has to support the full API stacks of Microsoft's Windows Azure and Google's Compute Engine public clouds, too – eventually.

For the moment, getting full support for the AWS APIs – in effect, making an OpenStack public or private cloud look operationally and functionally like the AWS EC2 cloud and its related S3 and EBS storage service – is paramount, and a decision that the community must make regardless of Rackspace's desire to make OpenStack look essentially like its own public cloud's Cloud Servers and Cloud Files services.

"The community controls the direction of the project, and it's time we advocate a public cloud compatibility strategy that is in all our best interests, not just those of a single, albeit substantial, contributor," writes Bias. "Failing to make this change in strategy could ultimately lead to the project's irrelevance and death."

With AWS refusing to let go of its server and storage designs and all of the tweaks it has made to the Xen hypervisor, the Linux operating system, and the other components that make up its infrastructure cloud, the way is open for some alternative to dominate the private cloud.

Windows by far prevails in the corporate data center, and the very good support that VMware had for virtualizing Windows made it a powerhouse in the server virtualization era. But it is not a foregone conclusion that VMware's vCloud can become the cloud controller of choice in the corporate data center – especially not with Microsoft's Hyper-V 3.0, Windows Server 2012, and System Center 2012 offering a less expensive and more native alternative to those who might prefer to stick with a Redmond stack.

OpenStack, like CloudStack and Eucalyptus, is more aligned to Linux workloads and increasingly the KVM hypervisor for slicing up servers, but the Xen hypervisor is still getting some love and remains popular on the biggest public clouds. No one knows what Google is using for its Google Compute Engine (GCE) infrastructure cloud, but AWS and Rackspace use variants of Xen.

The open source cloud controllers are also able to support and manage virtualized Windows workloads, however, so they have a chance of taking some share away from either VMware or Microsoft as corporations transform their virtualized server farms into clouds by adding orchestration, billing, metering, disaster recovery, and other functions to their pooled server capacity.

In the opinion of Bias, the need to fully embrace the AWS APIs – to make OpenStack look and operate like AWS – is pretty clear-cut. It's based on an assumption that AWS will continue to dominate the public cloud market, no matter how hard Rackspace and the likes of Hewlett-Packard and IBM try to fluff up OpenStack-based clouds.

"It is clear that AWS (and quite likely GCE) will utterly dominate the public cloud race," Bias states emphatically. "But more importantly, who cares? Dominance by AWS and GCE does not mean that OpenStack fails. In fact, OpenStack is clearly on a trajectory to 'win' the private cloud race, and a rapid embracing of Amazon will put OpenStack in the pole position to dominate hybrid cloud."

Two and a half years ago, AWS and Rackspace were basically doubling their cloud businesses, according to Bias, but Rackspace's growth in the cloud has slowed down as Google has jumped in (and Microsoft as well, which Bias did not mention) and AWS has continued more or less on its trajectory.

Rackspace wanted to "control the innovation curve" like Amazon does with its public cloud, Bias argues, and therefore not only started OpenStack with NASA, but also was able, through the acquisition of ANSO Labs, to pick up the NASA people familiar with the Nova compute controller and get them to essentially shift to Rackspace's own Cloud Files API set. And now, says Bias, it is time to shift back.

"OpenStack can be in control of the innovation curve in private and hybrid cloud, but doing so requires that we support the services that are leading the innovation curve in public cloud," Bias writes. "For OpenStack to dominate innovation in private and hybrid, it must embrace the public clouds to which enterprises want to federate."

This, from a guy who admits that he was not precisely enamored with all this talk of private and hybrid clouds a few years back.

Essentially, what Bias is arguing is that no OpenStack-based public cloud will ever get the kind of momentum that AWS and GCE will get, and therefore to support the hybrid clouds that corporations – you know, the ones with the money – want will require OpenStack to look and feel like the most popular public clouds and integrate seamlessly with them.

Bias dismisses out of hand any suggestion that Amazon might sue anyone that clones the AWS API stack.

"Fear, uncertainty, and doubt on public cloud API protectability is complete foolishness," writes Bias. "There is no legal basis for stopping the OpenStack community from copying the AWS and GCE APIs. Also keep in mind that Amazon's APIs have already been copied. Their ability to succeed in a new legal action would therefore be compromised by the fact that they’ve not fought copying in the past."

(Once again, Bias does not mention Azure. And yes, no one thinks that Microsoft is making money with the Azure cloud, but it sure is building it out like a fiend and it is one of the few organizations on Earth that has the cash to meet or beat AWS and GCE on a capital expenditure pissing contest. And Windows is not only the dominant operating system in the data center these days, but Windows workloads are very likely the ones that will need federated private-public cloud support.)

Like many of us, Bias is concerned that Amazon might weaken and start offering private clouds based on its internal infrastructure. In a sense, it is doing this already with its GovCloud for government agencies in the United States, and the $600m deal that Amazon is trying to win with the CIA in a battle with IBM does involve building an Amazon private cloud.

After the PRISM fiasco at the NSA, Amazon may have little choice but to offer internal AWS slices to corporations that, for business, security, and legal reasons simply do not want to put their applications and data out on a public cloud, even one that is isolated like GovCloud.

El Reg would argue that it is precisely the lack of AWS API compatibility in the popular public clouds and the OpenStack, VMware, and Microsoft private cloud stacks that has allowed AWS to pooh-pooh the idea of private clouds and steer customers to quasi-compatible clouds built using the eponymous cloud controller from Eucalyptus Systems – which, as you may remember, was the former darling of NASA's cloud as well as the one that was going to take over the world before OpenStack came along.

Bias is not arguing that OpenStack should ditch all of the API goodness that Rackspace and others have woven into the Nova compute controller in the past three years. He is, however, suggesting that everyone be honest about it and call it the Rackspace Cloud Servers API, and then shift to a low-level API and a bridged API model to make OpenStack's management framework compatible with all of the popular public clouds. (Here he mentions AWS, GCE, and Azure by name and even gives vCloud a possible nod.) Finally, he says, OpenStack should embrace the interoperability testing frameworks developed by Cloudscaling and Eucalyptus.

While this all sounds well and good in theory, in practice supporting the cloud management APIs of five different styles of public cloud is a tall order indeed for the OpenStack community, when they also have to add features and functions to OpenStack that are missing from public clouds but are required by private ones.

This is a huge amount of work, and the first question is: Who on Earth is going to do it? Not only will it be expensive, but it will also be a never-ending task as each public cloud provider adds features and, most likely, does things a bit differently from its rivals to make it harder for a single cloud controller to rule them all. (Bias did not commit Cloudscaling explicitly to doing the work in his call to arms, but presumably the company, which has just pocketed $10m in venture capital, is willing to bear some of the coding burden.) 

The organizations that most naturally would help to make their clouds compatible with the OpenStack controller are running their own public clouds and would seem to have little incentive to make OpenStack the safe, Swiss bet. They will do better by balkanizing the public cloud as they have virtual server infrastructure in the data center, replicating the server silos found in the glass house up there in the heavens.

If Bias and fellow OpenStackers can convince some deep-pocketed equity firms that there is money to be made adding these other API layers to OpenStack, then it could happen. It is a good dream, and it is important to dream. But there never was one Unix and there probably will not be one cloud controller that speaks all clouds, either. ®