New Xen vuln triggers Amazon, Rackspace reboot panic redux
Second hypervisor-related cloud meltdown in six months
Newly discovered vulnerabilities in the open source Xen virtualization hypervisor have once again sent major public cloud companies scurrying to patch and reboot their systems before attackers can pull off a massive exploit.
Amazon and Rackspace have both announced that they will need to reboot some of their servers to address the issue before March 10, when the Xen Project plans to disclose the latest bugs. Details of the vulns are being withheld for now, to give the cloud vendors time to patch.
In a FAQ about the upcoming maintenance, Amazon Web Services said that only some of its earliest Elastic Compute Cloud (EC2) customers should be affected.
"We have built the capability to live-update the vast majority of our fleet; however, we have not yet enabled this capability on some of our older hardware," the online retail giant said. "This older hardware is what’s being rebooted."
Rackspace also said that only a portion of its machines will be affected, but it cautioned customers to be prepared for potential outages.
"We understand that any downtime impacts your business and we do not make this decision lightly," Rackspace said. "In preparation for a potential reboot, we recommend that you take proactive steps to ensure your environment is configured to return to proper operations."
Like Amazon, Rackspace says it plans to have all of its affected First Generation and Next Generation cloud servers patched and rebooted by Monday, March 9, and that the first reboots will on Monday, March 2.
This isn't the first time cloud vendors have been bitten by bugs in Xen. In fact, the last time was less than six months ago, when a vuln that allowed unauthorized memory access forced a similar mass reboot.
Other cloud vendors are likely to be affected, but not all of them. Microsoft, for example, uses a homegrown hypervisor for its Azure cloud. But IBM's SoftLayer cloud reportedly also had to reboot systems to address last year's Xen bugs, so it likely will this time, too (although we haven't heard anything yet).
Admins who run Xen on their own machines should likewise be on the lookout for patches from their OS vendors in the next couple of weeks. ®