Sysadmin sank IBM mainframe by going one VM too deep
Tried to blame it on a bug, but logs don't lie
Who, me? Welcome to another glimpse inside the dark-curtained (in)box that is "Who, me?" – El Reg's confessional column in which readers seek penance for sins of the past.
This week, meet "George", who many years ago was one of two sysadmins in the office products division of a three-letter computer company. And was bored. Bored, bored, bored!
Which was when he got curious about the virtualization capabilities of the IBM mainframe he and a few dozen other developers toiled on every day.
"Each developer had their own virtual machine running IBM's mainframe OS, CMS," he said. "Normally, CMS saw everything. But you could issue commands to the VM hypervisor by prefixing your input with a special character, the default being a hashtag #.
"For instance, to shut down VM – and thus the whole machine – the command was '#cp shutdown'." George, as a sysadmin, could do this to all instances of VM. The rest of the crew could only ditch their own stuff.
So there's George in his office, twiddling his thumbs and mulling over the concept of VM, when "the idea occurred to me that VM created virtual machines".
"So it might be possible to boot VM 'in a virtual machine'."
Moving his thought experiment into the real world, George gave it a whirl...
"It actually worked," he said. "I set the second level command character to ';' and proceeded to create a third level virtual machine inside the second level."
But that raised another question for our curious George... "Can I boot VM in the third level machine?"
The answer? Yes. "That worked and it had a command character of '%'," he said, adding: "Which begs the question..."
Tech team trapped in data centre as hypoxic gas flooded in. AgainREAD MORE
At this point cunning readers can probably see where this is going, because George eventually tired of nested virtualization and shut down his VMs.
"But, of course, my fingers automatically typed '#cp shutdown'; rather than use the commands I'd assigned to the intermediate machines," said George, the panic still palpable all these years later and over the medium of email.
"So, VM did what it was commanded to do and shut down the REAL machine, killing off all 72 developers at once."
Faced with a rather sticky situation, George did what all good sinners do, and looked to shift the blame. In this case, he said, it was an "unknown bug" – but as is often the case, there's always someone watching.
"The other developer could see my shutdown command in the log files," confessed George. "He was not happy with me that day."
Have you ever virtually killed your team's productivity? Has curiosity nearly killed your cat? Absolve yourself by writing to the priest of Who, me? ®
Sponsored: Becoming a Pragmatic Security Leader