Feeds

NASA and Rackspace open source cloud fluffer

OpenStack targets one million machine Nebula

Boost IT visibility and business value

NASA and Rackspace have joined forces to open source a new platform for building so-called infrastructure clouds. Known as OpenStack, the platform is available under an Apache license, and when completed – possibly by the end of the year – NASA and Rackspace will ditch their current infrastructure cloud platforms, which don't scale as they would like.

NASA's Nebula infrastructure cloud is installed installed at NASA's Ames Research Center and is in the process of being and extended to the Goddard Spaceflight Center outside of Washington, DC. The Nebula cloud was spearheaded by Chris Kemp, who is now NASA's chief technology officer, and in an interview with El Reg, Kemp had nice things to say about the Eucalyptus cloud fabric controller that Nebula was currently using, which is an open source clone of Amazon's EC2 compute cloud and S3 storage cloud.

But Kemp also said that the scalability of the product and other issues with Eucalyptus (including the inability by NASA to get some of its enhancements into the Eucalyptus code base) compelled Kemp to take the entire Nebula team and dedicate it – for the past six months – to creating a new fabric controller, called Nova, from scratch.

It was serendipity that at just the time when NASA was working on the guts of a new compute engine that Rackspace had decided to contact NASA about its plan to open source the "Ozone" code behind its Cloud Servers compute cloud and the Cloud Files storage cloud (formerly known as CloudFS). NASA had released the Nova cloud fabric controller under an Apache 2 license at http://novacc.org/ (the code is currently at the 0.42 release level and admittedly buggy), and according to Jim Curry, vice president of corporate development at Rackspace, decided not only to open source its Cloud Servers code but to start the OpenStack project to mix-and-match the best bits of its Ozone code base with NASA's Nova code base to forge a better cloud fabric controller.

Nova is written in C, C++, and Python and Ozone is written in Python, and at this point, the exact nature of the merged project is being hammered out by NASA, Rackspace, and 25 other partners who are lining up to support the OpenStack Compute fabric controller.

Like the Eucalyptus tools NASA is currently using in its Nebula cloud, the Nova cloud fabric controller is built to mimic Amazon's EC2 and S3 APIs. Most of the code is written in Python, using the Tornado and Twisted frameworks. It uses the open source Advanced Message Queuing Protocol (AMQP) that Red Hat has championed for its Enterprise MRG cloudy Linux stack as well as a Redis distributed key value store. The code was released at the end of May.

NASA will be adopting the OpenStack Object Storage as its default object-based storage, and this code is literally Rackspace's Cloud Files opened up under an Apache 2 license to match that of NASA's Nova fabric controller.

Sometime in the fourth quarter, after the OpenStack project gets fully organized and programmers from inside NASA and Rackspace as well as from the OpenStack community decide what features and standards to support, both NASA and Rackspace intend to put the OpenStack Compute and Object Storage code into production. At that point, Eucalyptus and VMware's vSphere will be relegated to jobs where NASA needs to support a cloud that has third party application certification, something that Kemp never envisioned doing on the Nebula cloud in the first place and still doesn't see happening. "There is a place in the enterprise for all of these platforms," says Kemp.

The problem with the tools being created and sold by Eucalyptus Systems, VMware, and others trying to provide tools to fluff up clouds is that the scale that NASA and Rackspace are wrestling with is beyond the capability of these products.

"We are not the target customer," says Kemp. "We are constantly working around issues and creating code because of this." The difference is between a platform, which you can pull apart in a modular way – perhaps changing the networking engine or the queuing engine to boost performance – and a product, which is tends to be a single, integrated thing bound by the needs of an average of customers, not those pushing the scalability limits.

What kind of scalability are NASA and Rackspace talking about with the OpenStack software? The initial design goals agreed to by the two parties (who do not have any kind of formal arrangement except to contribute code to the OpenStack project and work in the community) is for OpenStack to span 1 million host machines and control up to 60 million virtual machines.

"VMware has not built something that can scale up to Amazon-level scalability of hundreds of thousands to millions of servers," says Curry at Rackspace. And as for Eucalyptus, Kemp over at NASA said "our team's analysis is that it could not get anywhere near it," meaning the design goals for OpenStack. Curry joked that if anyone wants to provide Rackspace with the iron, it would be happy to test it across a million boxes. (What do you say, Google?)

The exact feeds and speeds of the OpenStack Compute fabric controller have not been hammered out, but Rackspace hosted a design summit in Austin, Texas from July 13 through 16 where 100 technical advisors from 25 companies were on hard to help hammer out the initial details. These include AMD, Autonomic Resources, Citrix Systems, Cloud.com, Cloudkick, CloudSwitch, Dell, enStratus, FathomDB, Intel, iomart Group, Limelight, Nicira, NTT Data, Opscode, Peer 1, Puppet Labs, RightScale, Riptano, Scalr, Sonian, Spiceworks, ThoughtWorks, Zenoss, and Zuora. The initial cloud fabric will support KVM virtual machines as well as Xen virtual machines based on the open source Xen as well as Citrix Systems' XenServer implementation. (It is not clear yet how the Xen community, which created its own cloud stack last summer, and Citrix, which sells a commercial cloud fluffer called Citrix Cloud Center (or C3 for short), will participate in the CloudStack project.)

The essential guide to IT transformation

More from The Register

next story
The Return of BSOD: Does ANYONE trust Microsoft patches?
Sysadmins, you're either fighting fires or seen as incompetents now
Microsoft: Azure isn't ready for biz-critical apps … yet
Microsoft will move its own IT to the cloud to avoid $200m server bill
Oracle reveals 32-core, 10 BEEELLION-transistor SPARC M7
New chip scales to 1024 cores, 8192 threads 64 TB RAM, at speeds over 3.6GHz
Docker kicks KVM's butt in IBM tests
Big Blue finds containers are speedy, but may not have much room to improve
US regulators OK sale of IBM's x86 server biz to Lenovo
Now all that remains is for gov't offices to ban the boxes
Flash could be CHEAPER than SAS DISK? Come off it, NetApp
Stats analysis reckons we'll hit that point in just three years
Nimble's latest mutants GORGE themselves on unlucky forerunners
Crossing Sandy Bridges without stopping for breath
prev story

Whitepapers

5 things you didn’t know about cloud backup
IT departments are embracing cloud backup, but there’s a lot you need to know before choosing a service provider. Learn all the critical things you need to know.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.
Rethinking backup and recovery in the modern data center
Combining intelligence, operational analytics, and automation to enable efficient, data-driven IT organizations using the HP ABR approach.
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.