OpenStack: Control freak is a platform, just like Android and iOS
And increasingly for enterprise users over cloud operators
The OpenStack Summit is underway this week in Portland, Oregon, and the top brass at the companies who have been building the cloud controller and selling it to customers want everyone to know that this is not some open source project with promise, but a real product that can be put to use in data centers today.
Jonathan Bryce, one of the execs at Rackspace Hosting who brought OpenStack into being along with NASA nearly three years ago and who is executive director at the OpenStack Foundation that controls the governance for the project, did the opening keynote at the conference, and what he mostly wanted to talk about was users, not code, and the fact that OpenStack was not just a cloud controller, but a platform with an ecosystem.
"This is obviously something that is growing, that's big, that has a lot of interest and contribution around it," said Bryce. "But what is it that we are really doing here? It is not just software and it is not just a great development community, and it is not just about having a lot of vendors and companies involved. These things are important, but any one of them on their own is not enough. I think that what we are really doing is building a new platform ecosystem for the cloud."
Platform ecosystems are, explained Bryce, what happens when you have a base technology that is widely adopted and that is extendable and therefore allows a lot of innovation to happen around it.
OpenStack Foundation executive director Jonathan Bryce
To illustrate his point, Bryce had told everyone in the 3,000-strong audience that had a smartphone to raise their hands. Damn near all of the audience did. Then he said for everyone who had an Android or iOS smartphone to put there hands down, and as far as this El Reg hack could see, there was only one hand that remained up. (Probably a Windows 8 phone that was a gift.)
And so, argued Bryce, OpenStack is a data center platform, like Android and iOS are smartphone platforms, and its goal is to orchestrate all of the servers, storage, and switching in the glass house, with a common API on top of it to make it extensible and adaptable. And to succeed against its handful of competitors, OpenStack needs three things.
The first thing is powerful software, and while many in the IT racket will argue that OpenStack is still not ready for prime-time, no one could claim that the software hasn't come a long way since the summer of 2010 or that it cannot be deployed running real clouds today. That software is being developed by an larger and more diverse pool of software engineers, drawn from the vendor community and increasingly the end user community as well as individual developers.
A year ago, the OpenStack Summit was half the size it is this week, and there were 250 developers who helped bring the "Essex" release of OpenStack out, less than half the number who contributed to the current "Grizzly" release that came out two weeks ago. The odds are, by the time the Havana release is done in the fall of this year, the roster of contributors will grow again.
The other thing you need for a platform is an innovative ecosystem.
"It's not just about the companies, and this is one of the things that I have loved to see over the last year or so," said Bryce. "We have a lot of open source projects that are aligning themselves with OpenStack, with the way we do our releases, they are using our development tools, continuous integration and testing frameworks. And I think that shows the power of building a platform like this."
The final thing you need is users, and Bryce trotted out a few of them as part of his opening keynote at the OpenStack Summit.
What is it good for?
First up was financial data and media giant Bloomberg, which said that it had to solve a lot of the problems with setting up its MySQL databases and Ceph storage below the OpenStack orchestration layer and log aggregation and metrics monitoring for hypervisors above OpenStack. The company has released its Chef recipes for its clustered private cloud nodes at https://github.com/Bloomberg/chef-bcpc.
Electronics retailer Best Buy is using open stack to provision test and development environments in a consistent way for the 40 different development teams it has, and says that before it moved to OpenStack it was paying an incredible $20,000 to provision a virtual machine on its production cloud and each web site release was costing it $500,000 to roll out.
Cable operator Comcast showed off its forthcoming X1 cloudy set-top box, which will have most of the function for the set-top running in an OpenStack cloud instead of being pushed out to the set-top, making the overall software in the cable system more malleable.
And HubSpot, a maker of online marketing software, is gradually weening itself off Amazon Web Services and to its own OpenStack private cloud. HubSpot started out dabbling with nine server nodes with 108 cores and 900MB of memory with the Diablo release of OpenStack in the fall of 2011, and boosted that to sixteen nodes with 224 cores and 1.8TB of memory in the summer of 2012 when it moved to Essex. With its move to the Folsom release in January this year, the cluster has 166 nodes with 2,004 cores and 2TB of memory. About a hundred of those nodes are configured with the bare-metal provisioning tools in the Nova controller to run the Hadoop big data muncher and the HBase database layer for the Hadoop Distributed File System. NameNodes, JobTrackers, and other key control components of the HubSpot Hadoop stack run in virtual mode on top of OpenStack, and the HubSpot marketing applications run inside of virtual machines as well.
"I think this is the year that the users take over," said Jim Curry, general manager of the Rackspace Private Cloud business and one of the co-founders of the OpenStack movement, who spoke in his own keynote.
Curry also gave some insight into how Rackspace is doing with the Folsom and Grizzly code that it has put into production on the Rackspace Cloud, which is its public cloud. Since OpenStack went live last summer, the Nova controller has done 549 million API requests from the users of the cloud, and it has been 99.94 per cent available. On the private cloud side, more than 8,500 downloads of the stack of software that Rackspace uses in its own Alamo private cloud offering have been downloaded in the past six weeks, with many shops that are putting it into production doing so on hundreds of nodes, according to Curry.
Speaking to El Reg ahead of the OpenStack Summit, Joshua McKenty, co-founder and CEO of Piston Cloud, one of many commercial OpenStack distributors, said that as far as he could tell from talking to other distributors and analysts, there were maybe 300 to 400 production OpenStack clouds running out there in the world, with a lot of them doing test and dev work.
This is reasonable, given that server virtualization on x86 iron originally got its start in test and dev a decade ago since this is where some of the most intense server configuration pain was. OpenStack orchestration makes firing up servers and their software stacks even easier, and as Best Buy above shows, can be a key element in agile programming and continuous development. McKenty reckoned that there were maybe another 2,000 to 3,000 OpenStack proofs of concept running in the world.
After his keynote, El Reg ran McKenty's estimates by Curry, and he thought they were a bit on the low side.
"Based on the number of customers we have seen, I think those numbers are wrong," Curry said. "If we were playing The Price Is Right, I would take 401 for the number of production customers, at the very least."
Peter ffolkes, research director for servers, virtualization and cloud at analyst firm 451 Research, tells El Reg that it is still early days for enterprise clouds, particularly when companies are only now coming to grips with server virtualization. Only 26 per cent of the shops polled by 451 Research are doing automation on top of their server virtualization, and only 10 per cent are doing the full-on orchestration that comes with a tool like OpenStack or VMware's vCloud. "I haven't talked to a lot of people who have true clouds yet, but people are starting to look at it."
The issue is not, oddly enough, a technical one, but rather that companies don't have the resources to do a full-on cloud, or they have political camps who are not in agreement of the timing of a move to a truly cloudy infrastructure. They may have limited people and money even if they are ready to move, and in some cases, their applications need to be tweaked to move to a cloud.
"A lot of companies are just not ready to do this yet," explains ffolkes. "This will give OpenStack more time to become a more mature cloud controller, and for it to better compete against VMware, which is firmly entrenched as the server virtualization platform under enterprise licenses that are going to run another two or three years."
OpenStackers don't want to hear talk like this, of course, particularly after three hard years of work. But the fact remains that server virtualization, after 25 years on mainframes, 15 years on Unix and proprietary midrange servers, and a decade on x86 iron, is only now becoming normal.
It is going to take some time, perhaps many years, for the vast majority of companies to get as comfortable with true cloud computing. They have a lot to learn and a lot to consider, and companies make decisions a lot more slowly than IT vendors wish. It has always been this way.
But ultimately, the users are in fact in charge, just as they will increasingly drive the OpenStack project, either directly by participating or by being customers of the OpenStack disties that they get their code from - and to whom they can give orders about the features they need. ®