Linux team tells VMware and Xen to get their acts together
Play nice and have a chat
A battle as to how Linux will handle future virtualization software from the likes of VMware and Xen has moved from a war of words to a war of indecision. The major parties involved - including Linux kernel maintainers - agree that a compromise over the virtualization interface must be reached, but no one seems to know exactly how to achieve this goal.
This particular virtualization war started last July when VMware introduced the Virtual Machine Interface (VMI) at the Linux Symposium - an event aimed at kernel developers and others touching the Linux operating system. VMware's engineering team hoped to give the Linux crowd, including competing Xen developers, an idea of how it planned to handle the latest and greatest form of server virtualization known as paravirtualization, which requires kernel level changes. VMware's pitch centered on a type of open interface that any server virtualization vendor - be it VMware, XenSource (the major backer of the open source Xen) or Microsoft - could plug into with their future products.
The idea of an open paravirtualization interface doesn't rankle the Xen crowd, but VMware's particular approach does. Some members of the Xen camp argue that VMI does not perform well enough to become the standard Linux interface. In addition, the most ardent open source backers question the long-term motives of a proprietary software vendor such as VMware and wonder if an "open" interface won't end up benefitting VMware the most down the road.
The last bit of thinking - some would say over-aggressive politicking - reflects how seriously folks take this issue. Xen, you see, has put forth a "Hypercall" paravirtualization interface of its own. The Xen backers once fully expected the Hypercall APIs to become part of the Linux kernel, and were caught off guard recently when kernel maintainer Andrew Morton was characterized in a media report  as backing VMware's VMI interface over the Xen code.
So, which approach is better? Which one will make its way into the Linux kernel? And why should anyone care?
As it turns out, no camp - including Morton's Linux folks - can claim much certainty around the interface issue or how it will progress.
Morton, for example, said his preference for VMI has been "overstated".
"VMware have proposed an implementation that would allow, in theory, different kinds of hypervisors to run beneath the kernel," Morton said, in an interview with The Register. "It is, if you like, a hypervisor-neutral interface. The question remains if we want to have a hypervisor neutral interface. There might be the case that this generic interface isn't going to buy us anything, so maybe we will just merge with Xen.
"That decision has not yet been made. We need to step back and have a think about how we are going to do this."
Morton admits that he knows "very little" about the inner-workings of hypervisors - the core software layer used by some to manage virtual servers. With that in mind, he wants to see VMware and Xen developers hammer out the interface issue without involvement from him or Linus Torvalds.
"Any time Linus or I have to make a decision, the system has failed," he said, noting that the kernel crew is eager to see this particular issue resolved.
The major problem, however, is that VMware and Xen developers don't seem very close to having a meeting of the minds.
The wait debate
The two camps have debated VMI for months on Linux kernel mailing lists, with other developers from IBM, Red Hat, Oracle and elsewhere chiming in.
"Most of our interactions have been on that level," VMware senior director of research and development Jack Lo said.
VMware has reached out to the OSDL (Open Source Development Labs) and other unnamed parties to help set up a public forum for discussing the interface issue with Xen backers and anyone else who wants to show up.
But, after interviewing a number of people for this story, we get the sense that such a meeting is a concept at best right now. The parties involved proved reticent to nail down a time when they actually plan to sit down and don't seem all that determined to approach a debate until all the behind-the-scenes political posturing has taken place.
VMware has received praise for being so open with VMI and encouraging debate on the kernel mailing lists  that include Xen backers. In addition, VMware has helped itself by setting up a slick VMI website  that displays votes of support from the likes of IBM, HP and Novell.
The Xen camp, by contrast, has been criticized for taking the interface issue for granted and assuming its technology would go into Linux.
"This will take more work than the Xen people are anticipating," Morton said.
One Xen backer said a disconnect occurred where the Xen camp thought SuSE and Red Hat would use their close ties with the Linux kernel maintainers to sell the Xen interface. "As it turns out, that was a mistake. Morton wants more contact from us," he said.
On the mailings lists and in discussions, Xen advocates say VMware's VMI simply will not allow for the performance levels they would like to see out of virtualizaton. Xen, which already relies on the paravirtulization model, has learned a myriad of lessons about how a paravirtualized API should operate - skills that those like VMware and Microsoft that are just making their way toward paravirtualization have yet to acquire, Xen backers say.
In particular, they argue that VMI will not support I/O virtualization or SMPs.
VMware responded in a statement saying, "the VMI patches that we recently sent to the Linux kernel mailing list do include SMP support. I/O virtualization isn't in the spec today - it's an area of future work, and we're looking to collaborate with people on it."
More broadly, some Xen backers argue that VMI is simply a "crippled software version" of the virtualization hardware tools AMD and Intel have started building into their chips to make running virtual operating systems easier. That would make VMI rather redundant. Other VMI critiques include a proliferation of kernel hook points, a design that can't support optimizations Xen backers have found to be critical for top performance and weak semantics.
VMware counters by saying its approach requires less kernel changes than Xen, while delivering solid performance.
There are, in fact, laundry lists of other disputes - down to simple issues such as the interface name - that will have to be worked out.
Our sense is that VMware is honest about wanting an open interface but part of its openness stems from a desire to thwart Xen's apparent Linux edge. Xen 3.0 will be bundled with upcoming versions of Red Hat and SuSE, and Xen has a virtualization protocol license from Microsoft. VMware can't claim victories on either of those fronts and, as the clear market leader, is being pigeonholed to a degree by rivals. An open VMI interface could keep it closer to the Linux camp.
Xen, by contrast, wants to make the most of its open source ties and create the tightest possible bonds with Linux. Behind closed doors, some Xen backers say that Sun, Microsoft and Novell will refuse to support VMI. Such political manoeuvering shows how seriously Xen backers take this debate.
It's in Linux customers' best interests that all of this get worked out as soon as possible. The worst scenario would require ISVs and customers to certify applications for a host of paravirtualization interfaces.
As you can tell, however, it seems pretty unlikely that either VMware or Xen advocates will reach an agreement anytime soon, despite their promises to do so. And you can image how exciting the battle will become should the likes of Microsoft, IBM, SWsoft and others decide they want in on the issue as well. Yes, that's right. Microsoft may be a key player in a Linux kernel squabble.
You can expect more and more of these kinds of debates in the coming years, as server virtualization hots up even more. Microsoft, the open source player Xen and the proprietary middleman with all the market share VMware. Can you ask for anything more? ®