Container orchestration top trumps: Let's just pretend you don't use Kubernetes already

Open source or Hotel California, there's something for everyone

conductor's hand on a black background

Container orchestration comes in different flavours, but actual effort must be put into identifying the system most palatable.

Yes, features matter, but so too does the long-term viability of the platform. There's been plenty of great technologies in the history of the industry, but what's mattered has been their viability, as defined by factors such as who owns them, whether they are open source (and therefore sustained by a community), or outright M&A.

CoreOS, recently bought by Red Hat, offered Fleet. Fleet, alas for Fleet users, was discontinued because Kubernetes "won".

First, however, the basics: what is container orchestration? Orchestration platforms are to containers as VMware's vSphere and vRealize Automation are to Virtual Machines: they are the management, automation and orchestration layers upon which a decade or more of an organization's IT will ultimately be built.

Just as few organizations with any meaningful automation oscillate between Microsoft's Hyper-V and VMware's ESXi, the container orchestration solutions will have staying power. Over the years an entire ecosystem of products, services, scripts and more will attach to our container orchestration solutions, and walking away from them would mean burning down a significant portion of our IT and starting over.

So let's look at who's who, and what's what in the world of containers, and see if you can find the right flavour for you.


Skipping right to the end, Kubernetes's flavour is that of victory. Kubernetes is now the open-system container orchestration system. Mainframe people – who like to refer to anything that's not a mainframe as an open system – will cringe at my using the term open system here. I make no apologies.

The major public clouds pretend to be open systems, but everywhere you turn there's lock-in. They're mainframes reborn, and when talking about containers, most of which probably run on the major public clouds.

Developed by Google, Kubernetes was designed specifically to be an open, accessible container management platform. Google handed the technology to the Cloud Native Computing Foundation (CNCF) – another in a long line of open-source foundations run by a group of technology vendors.

Kubernetes is part of an emerging stack of technologies that form the backbone of open source IT automation. The container part of the story starts with VMs or bare metal machines which are provisioned into container hosts. These have a Linux Operating System Environment (OSE), and a containerization platform (Docker or rkt) installed.

The containerization platform handles the creation of containers, the injection of workloads into containers and destroying containers. Kubernetes is the bit that lashes multiple containerization hosts together into a cluster and bosses the containerization platform around. Kubernetes handles resource distribution, scheduling, availability, stateful storage and more. Think of it like "vSphere for containers" and you're close enough for jazz.

Kubernetes was delivered as a 1.0 product that was already fairly mature. Thus far, the various CNCF members haven't had a chance to ruin it through development by committee, but it's early days yet.

For the moment, Kubernetes has won the container orchestration ways. Market dominance, however, isn't the same as a monopoly, and there is still room for others.


Kubernetes is fantastic for the sorts of workloads that most people place in containers: stateless, composable workloads. They're the cattle in the cattle versus pets discussion. Some organizations, however, have reason to keep a few pets around. That's where Mesosphere Marathon comes in.

Marathon is a container orchestration framework for Apache Mesos that is designed to launch long-running applications. It offers key features for running applications in a clustered environment.

Marathon is Kubernetes, but for the Mesophere DC/OS or Apache Mesos. It can boss around Mesos containers or Docker containers. If you don't know what any of that is, that's perfectly okay. What you need to know is that Marathon is where you go when you have stateful workloads that you want to run for long periods of time, and for some reason you want to do this in containers instead of VMs.

Persistent containers is Marathon's niche. Others may do it, but none do it quite as well.

Hotel California-class

For many IT vendors, lock-in is a feature, not a bug. As we head into the 2020s, the public cloud providers are the standard-bearers of this philosophy. While the major public cloud providers all embrace open source and standardization to varying degrees, the ultimate goal is to convince you to put your workloads into their clouds, and then pay them rent on those workloads forever.

Amazon's EC2 Container Service (ECS) stands up a series of EC2 instances and installs Docker on them. It then lashes them together into a cluster and lets you manage them. Basically, it's Kubernetes, but with a distinctly Hotel California aftertaste.

Azure Container Instances (ACI): ditto what was said about ECS. But with the Amazon replaced with Microsoft in the recipe.

Google Container Engine (GCE) is Google's version of the above. Of course, being Google, it's kind of terrible at locking you in. GCE is basically Kubernetes with some Google Cloud Platform (GCP)-specific features thrown in. If you build your business on top of GCE automation it will still be a pain to disentangle that automation and go elsewhere, but far less of a pain than it would be with EC2 or ACI.

Cloud Foundry

Cloud Foundry should be thought of as Openstack for containers. It is corporate Open Source at its finest. Written by VMware and then transferred to Pivotal when Pivotal was spun out, Cloud Foundry's original purpose was to run containers on VMware's vSphere, with an eye towards being the Platform as a Service (PaaS) choice for the world's enterprises.

Today, Cloud Foundry's intellectual property is held by the Cloud Foundry Foundation (CFF). The CFF is to Cloud Foundry as the CNCF is to Kubernetes. Both the CFF and CNCF are Linux Foundation projects. The Linux Foundation is another industry group mostly composed of various vendors. You'll find a lot of the same vendors hold power in all three groups.

If you want to roll your own PaaS, and/or don't like the PaaS options the major public cloud providers have to offer, Cloud Foundry is for you. If PaaS isn't your thing, give up, and use Kubernetes.

CoreOS versus Docker versus the world

Docker Swarm is Docker's container orchestration offering. Back before CoreOS was borged by Red Hat, there was great fun to be had on Twitter by provoking both CoreOS and Docker nerds into intricately detailed technical poo-flinging contests about which was better. Then Kubernetes won. Now Docker Enterprise supports Kubernetes and Swarm's days are numbered.

In a container

So where does this trot through the landscape leave us? No surprises: in the container orchestration world, Kubernetes is the container-farming king – but it isn't ruler of all we survey. Mesosphere occupies a decent niche as the kennel for your pets. Just beware Amazon, Azure and Google – these are Hotel California: you can check in your code, but it most likely won't ever leave. If portability is your particular king then I advise you to steer clear. If not, code on.

If you are using anything new, unheard of or just plain better than these three orchestration flavours let me know. ®

We'll be covering DevOps at our Continuous Lifecycle London 2018 event. Full details right here.

Sponsored: Minds Mastering Machines - Call for papers now open

Biting the hand that feeds IT © 1998–2018