Original URL: http://www.theregister.co.uk/2008/08/13/vmware_apologises_12_august_code/

VMware admits 'time bomb' rolled past quality control

Glum CEO says 'we're sorry'

By Kelly Fiveash

Posted in Servers, 13th August 2008 10:50 GMT

VMware’s CEO has blamed a chunk of leftover pre-release code for a bug that yesterday prevented virtual servers around the world from powering up when the clock hit 12 August.

Despite the company carrying out quality assurance of its product, VMware failed to spot that it had released the leftover bit of time-bombed code within the ESX 3.5 and ESXi 3.5 Update 2 final versions of its hypervisors that were released about two weeks ago.

VMware boss Paul Maritz apologised for the major balls-up. He said the code caused the product license to expire. And, worse still, he admitted that the problem had been overlooked in recent patches to the product.

The code time bomb prevented virtual machines from powering on or leaving suspend mode and unable to migrate using the firm's VMotion software.

Maritz said VMware has released an “express patch” for customers unlucky enough to have already installed or upgraded to ESX or ESXi 3.5u2.

Its temporary work-around requires sys admins to turn the clock back to kick-start virtual machines and imagine that 12 August was just a bad dream.

However, as we noted yesterday, a whole swathe of customers have grumbled that this is inadequate since it’s a legal requirement for most businesses to timestamp all server transactions.

The company, which claims to have "the industry’s most popular and reliable hypervisor", also plans to spit out a full replacement for Update 2 that should be used by customers who want to perform fresh installs of ESX or ESXi.

Here are a few choice words from a very sad-looking Maritz:

We are doing everything in our power to make sure this doesn’t happen again. VMware prides itself on the quality and reliability of our products, and this incident has prompted a thorough self-examination of how we create and deliver products to our customers. We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes.

I want to apologise for the disruption and difficulty this issue may have caused to our customers and our partners. Your confidence in VMware is extremely important to us, and we are committed to restoring that confidence fully and quickly.

Shares in VMware also took a hit yesterday, dipping more than five per cent on the New York Stock Exchange after news broke of the Palo Alto, California-based company's major code cock-up. ®