Original URL: http://www.theregister.co.uk/2008/11/12/cloud_package_management/

Why clouds should be more like operating systems

Best things come in small management packages

By Dave Rosenberg

Posted in Developer, 12th November 2008 21:09 GMT

For the cloud to become more like an operating system and less like an amorphous mass, and by definition more useful to business, some basic functions need to become part of the fabric.

By that, I mean seamless (and near invisible) integration with the underlying infrastructure components (operating system, virtual machine, hardware, networking). These kinds of things will drive developers to the cloud and make it a more manageable environment. Package management, akin to what's found in Linux distros, is one milestone that seems the easiest for existing and aspiring providers of cloud services to achieve.

As the name implies, a package management systems help you install or remove, upgrade or configure the packages in a system - Linux and Unix mostly. Typical functions include: upgrading from a repository, verifying signatures and checksums and managing dependencies.

Examples of Linux package managers include Red Hat's RPM, the Fedora YUM or APT from Debian/Ubuntu. Each manager has some different features or functions and they also cross borders to other distributions. Sun Microsystems' vice president of developer and community marketing Ian Murdock has posited that package management is "the single biggest advancement Linux has brought to the industry" here.

On Ubuntu Linux, the apt-get utility provides a command-line interface to system packages. It's easily invoked with a simple command plus an argument.

Let's say I want to install Gimp on my Ubuntu box, then I type the following:

[Dave-ElReg:~] dave% sudo apt-get install gimp

If there is a new release of Gimp or other software I have installed, I run the update command to update my local package index of what's available:

[Dave-ElReg:~] dave% sudo apt-get update

And to run an upgrade of all the software on my machine I use the following command - note the -u option shows the complete list of packages, which will be upgraded:

[Dave-ElReg:~] dave% sudo apt-get -u upgrade

The upgrade command will detail out the packages that will be updated and then work it's magic.

OK, let's get back to the cloud.

Let's say you are running multiple different Amazon Machine Images (AMIs), which contain your applications, libraries, data and configuration settings on Amazon's Elastic Compute Cloud (EC2), and you are using Amazon's S3 for storage. Don't think running everything in the cloud will abstract away potential management problems. You'll still have a system administration headache until you script something or update your AMIs with your new software and application code.

A better - and obvious - answer would be if you could have all of your images, code and applications available in a dashboard where you could simply update everything on the fly.

An even better answer would be to not have to perform any system administration functions at all. Currently, the only way to make that happen with Amazon is through third-party tools like 3Tera and RightScale.

Amazon may be the best-known cloud provider, but it's certainly not the only one. There are a variety of arguments in favor of other approaches, such as Microsoft's yet-to-launch Azure Services Platform and the existing Salesforce.com service.

I guess we'll have to wait and see what transpires with Azure, as it's still available only in early access to a selected number of testers. That leaves Salesforce.com's hosted platform, Force.com, which Salesforce.com was pushing hard this month as a cloud.

The relevance of Force.com has been questioned outside of the Salesforce.com user base. Force.com is Salesforce.com platform consisting of the multi-tenancy hosting, utility provisioning and subscription pricing minus the trademark customer relationship management (CRM) interface that made the Salesforce.com name.

That said, though, the company does have a very large user base - more than 41,000 companies on its service. Force.com applications are written in the proprietary Apex language, and there are APIs for Java, .NET and other languages. The platform has some great pre-built wizards, analytics and tooling, and of course, your custom applications automatically scale.

Force.com is actually a very elegant solution to cloud development, but lacks any non-Force.com deployment options - you can't deploy your application to a third-party's data center or cloud. To be fair, Salesforce.com acknowledges that portability is not part of their design and considering the underlying technical approaches in the Amazon and Salesforce.com cloud are fundamentally different, chances are you're likely going to remain stuck for awhile once you've picked your platform.

When it comes to package management, though, Force.com is very appealing since you don't have to maintain any infrastructure - you forget about VMs, operating systems and other software. You simply deploy your application and the system knows what to do with it.

So far there are no exact replacements for EC2 or S3 for Force.com. And there aren't many clever facsimiles either, with the exception of Eucalyptus, an open source project that implements the EC2 interface.

So what does this mean in real-life?

Let's say I have an AMI running on EC2 and I love the data processing function. However, for security reasons my company decides the same application function needs to live inside the corporate firewall. There is no way I can replicate the EC2 environment. Nor is there a way to ensure that I can get my persisted data - assuming I've stored it on S3 - to automatically relate to my new internal system.

Give us the tools

Cloud providers have used APIs to obfuscate the underlying technology largely to the consumers' benefit though there are some issues. Force.com's APIs eliminate the need to manage infrastructure but hide the level of detail you might need. Amazon's web services require the user to manage the AMIs that make up their applications. This of course introduces complexity and potential downtime, and introduces another form of lock-in through Amazon's proprietary AMIs.

Let's assume the companies have acted in the interests of users, by hiding complexity to save us from ourselves through acts such as randomly introducing changes that accidentally cause lost data or corrupted applications. That's OK for the consumer. But the business needs to get into the nooks and crannies of the cloud in the same way most business IT users tailor their servers and desktop images. The answer is package managers, the kind used in operating systems.

Package mangers can maintain a vast array of items that ensure you don't accidentally screw everything up. As with a local server, in the cloud you can easily make a mess of things, and you don't necessarily have backups or replication options.

Cloud systems need to be more autonomic - good lord, was IBM right? - while not forgetting that developers stick with their personal choice of tools for everything they do. And operations people need to have visibility into cloud systems while making sure they can't accidentally blow everything away. And of course, these management tools should all be pretty, interoperable and easy to use.

Someone get on it while I grab a coffee.®

Dave Rosenberg is the co-founder and former chief executive of open source enterprise service bus and integration platform MuleSource. Dave is currently working on a new stealth start-up based in San Francisco.