This article is more than 1 year old

ESXi's been borking VM backup and there's no fix yet

Patch lands, but prepare to reboot hosts and guests

Update VMware has reported a nasty bug that means virtual machine backups may not work.

The problem hits virtual machine backups that use Changed Block Tracking (CBT)in ESXi. “The CBT API call QueryDiskChangedAreas() … can sometimes return incorrect changed sectors, which results in inconsistent incremental virtual machine backups,” VMware writes.

VMware knows what the problem is, describing it as follows:

This issue occurs due to a problem with CBT in the disklib area, which results in the change tracking information of I/Os that occur during snapshot consolidation to be lost. The main backup payload data is never lost and it is always written to the backend device. However, the corresponding change tracking information entries which occur during the consolidation task are missed. Subsequent QueryDiskChangedAreas() calls do not include these missed blocks and, therefore a backup based on this CBT data is inconsistent.

VMware's working on a fix, but for now the only workarounds are downgrades, to either ESXi 5.5 and version 10 of VMware's virtual machine hardware. Or you can do a full VM backup. Another workaround requires a shutdown and restart of a VM.

Good luck getting lots of users to do the downgrade, Virtzilla. And good luck to users dealing with the bug: the notification of the problem was last updated on November 17th (so here's hoping you all have RPOs of more than a week).

Commvault has a fix for its software while Veeam users can just turn off CBT, keep calm and carry on backing up.

Hat-tip to up2v for spotting the problem. ®

Update, November 26th: VMware's posted a patch for the problem. Which is good news. Less good is that you'll need to reboot guests and hosts, so prepare for a little downtime if you want predictable backups again. And who doesn't want that?

More about

More about

More about

TIP US OFF

Send us news


Other stories you might like