IBM melts down fixing Meltdown as processes and patches stutter
RHEL servers croaking, reporting in Excel, customer docs in signoff limbo
IBM has scrambled to fix the Meltdown and Spectre bugs, but has struggled to develop processes, reporting tools or reliable patches to get the job done for itself or its clients.
Internal documents seen by The Register reveal that Big Blue has ordered staff not to attempt any Meltdown/Spectre patches, but that the advice to do nothing is incorrect and needs to be changed. The documents also reveal that IBM is urging its people to stick to a script and use a pre-approved presentation when discussing Meltdown/Spectre remediation with customers. However neither the script nor the presentation has been completed or approved. Staff have been told to expect the documents “in coming days”.
The documents also report that patches for the twin CPU bugs are failing on Windows due to interactions with antivirus tools. That’s a known issue others have encountered. The documents also say some Red Hat Enterprise Linux servers aren’t rebooting after patching, which is of more concern given that Red Hat developed its own Meltdown/Spectre patches.
Staff have also been advised that there’s no documentation of such incidents: everything’s being done by word of mouth for now.
IBMers are therefore being urged to ensure client systems are thoroughly backed up before attempting patches, and even then to do so only after rigorous testing and securing users’ signoff of patching programs.
Big Blue’s remaining employees must also wait for reporting tools to track progress of Meltdown/Spectre fixes – as of early on January 9th they’re still being written ... in Excel, suggesting that IBM’s services team find spreadsheets faster to implement than a more formal incident management tool. And that even the planet's oldest IT services organisations just aren't geared up for a sudden and massive patching effort of core computing infrastructure. ®