NASA and Rackspace open source cloud fluffer
OpenStack targets one million machine Nebula
NASA and Rackspace have joined forces to open source a new platform for building so-called infrastructure clouds. Known as OpenStack, the platform is available under an Apache license, and when completed – possibly by the end of the year – NASA and Rackspace will ditch their current infrastructure cloud platforms, which don't scale as they would like.
NASA's Nebula infrastructure cloud is installed installed at NASA's Ames Research Center and is in the process of being and extended to the Goddard Spaceflight Center outside of Washington, DC. The Nebula cloud was spearheaded by Chris Kemp, who is now NASA's chief technology officer, and in an interview with El Reg, Kemp had nice things to say about the Eucalyptus cloud fabric controller that Nebula was currently using, which is an open source clone of Amazon's EC2 compute cloud and S3 storage cloud.
But Kemp also said that the scalability of the product and other issues with Eucalyptus (including the inability by NASA to get some of its enhancements into the Eucalyptus code base) compelled Kemp to take the entire Nebula team and dedicate it – for the past six months – to creating a new fabric controller, called Nova, from scratch.
It was serendipity that at just the time when NASA was working on the guts of a new compute engine that Rackspace had decided to contact NASA about its plan to open source the "Ozone" code behind its Cloud Servers compute cloud and the Cloud Files storage cloud (formerly known as CloudFS). NASA had released the Nova cloud fabric controller under an Apache 2 license at http://novacc.org/ (the code is currently at the 0.42 release level and admittedly buggy), and according to Jim Curry, vice president of corporate development at Rackspace, decided not only to open source its Cloud Servers code but to start the OpenStack project to mix-and-match the best bits of its Ozone code base with NASA's Nova code base to forge a better cloud fabric controller.
Nova is written in C, C++, and Python and Ozone is written in Python, and at this point, the exact nature of the merged project is being hammered out by NASA, Rackspace, and 25 other partners who are lining up to support the OpenStack Compute fabric controller.
Like the Eucalyptus tools NASA is currently using in its Nebula cloud, the Nova cloud fabric controller is built to mimic Amazon's EC2 and S3 APIs. Most of the code is written in Python, using the Tornado and Twisted frameworks. It uses the open source Advanced Message Queuing Protocol (AMQP) that Red Hat has championed for its Enterprise MRG cloudy Linux stack as well as a Redis distributed key value store. The code was released at the end of May.
NASA will be adopting the OpenStack Object Storage as its default object-based storage, and this code is literally Rackspace's Cloud Files opened up under an Apache 2 license to match that of NASA's Nova fabric controller.
Sometime in the fourth quarter, after the OpenStack project gets fully organized and programmers from inside NASA and Rackspace as well as from the OpenStack community decide what features and standards to support, both NASA and Rackspace intend to put the OpenStack Compute and Object Storage code into production. At that point, Eucalyptus and VMware's vSphere will be relegated to jobs where NASA needs to support a cloud that has third party application certification, something that Kemp never envisioned doing on the Nebula cloud in the first place and still doesn't see happening. "There is a place in the enterprise for all of these platforms," says Kemp.
The problem with the tools being created and sold by Eucalyptus Systems, VMware, and others trying to provide tools to fluff up clouds is that the scale that NASA and Rackspace are wrestling with is beyond the capability of these products.
"We are not the target customer," says Kemp. "We are constantly working around issues and creating code because of this." The difference is between a platform, which you can pull apart in a modular way – perhaps changing the networking engine or the queuing engine to boost performance – and a product, which is tends to be a single, integrated thing bound by the needs of an average of customers, not those pushing the scalability limits.
What kind of scalability are NASA and Rackspace talking about with the OpenStack software? The initial design goals agreed to by the two parties (who do not have any kind of formal arrangement except to contribute code to the OpenStack project and work in the community) is for OpenStack to span 1 million host machines and control up to 60 million virtual machines.
"VMware has not built something that can scale up to Amazon-level scalability of hundreds of thousands to millions of servers," says Curry at Rackspace. And as for Eucalyptus, Kemp over at NASA said "our team's analysis is that it could not get anywhere near it," meaning the design goals for OpenStack. Curry joked that if anyone wants to provide Rackspace with the iron, it would be happy to test it across a million boxes. (What do you say, Google?)
The exact feeds and speeds of the OpenStack Compute fabric controller have not been hammered out, but Rackspace hosted a design summit in Austin, Texas from July 13 through 16 where 100 technical advisors from 25 companies were on hard to help hammer out the initial details. These include AMD, Autonomic Resources, Citrix Systems, Cloud.com, Cloudkick, CloudSwitch, Dell, enStratus, FathomDB, Intel, iomart Group, Limelight, Nicira, NTT Data, Opscode, Peer 1, Puppet Labs, RightScale, Riptano, Scalr, Sonian, Spiceworks, ThoughtWorks, Zenoss, and Zuora. The initial cloud fabric will support KVM virtual machines as well as Xen virtual machines based on the open source Xen as well as Citrix Systems' XenServer implementation. (It is not clear yet how the Xen community, which created its own cloud stack last summer, and Citrix, which sells a commercial cloud fluffer called Citrix Cloud Center (or C3 for short), will participate in the CloudStack project.)