Rookie almost wipes customer's entire inventory – unbeknownst to sysadmin
Spends three hours recreating the device tree by hand, leaves with heart in throat
Who, Me? Welcome once again to El Reg’s weekly instalment of Who, Me?, where readers get monumental cock-ups and heart-stopping near-misses off their chests.
This week, Reginald tells us a personal horror story from the ‘80s, when he worked for what was then a top five minicomputer biz.
At the time, he was on site at a customer – a chain of convenience stores – and was working on the computer that managed their inventory and ordered more goods for the stores.
“In those early UNIX systems we had to compile the kernel every time we patched it or added functionality,” he said.
How an over-zealous yank took down the trading floor of a US bankREAD MORE
“The system took the old kernel and renamed it
/dgux.old while the new kernel was named
/dgux. If memory recalls it correctly it had a bunch of older kernels named
/dgux.old.1 /dgux,old.2 and so on.”
Back at the store, Reginald finished up late one Friday afternoon and decided to clean up the old kernels.
“I was going to type
rm -rf /*.old* – which would have forcibly removed all
/dgux.old stuff, including any sub-directories I may have created with that name,” he said.
But – as regular readers will no doubt have guessed – he didn’t.
“I fat fingered and typed
rm -rf /* – and then I accidentally hit enter instead of the "." key.”
At that moment, everything from
/ and onward began deleting forcefully and Reginald described his subsequent actions as being akin to “flying flat like a dart in the air, arms stretched out, pointy finger fully extended” towards the power switch on the mini computer.
“Everything got quiet.”
Reginald tried to boot up the system, but it wouldn’t. So instead he booted up off a tape drive to run the mini Unix installer and mounted the boot
"/" file system as if he were upgrading – and then checked out the damage.
“Everything down to
/dev was deleted, but I was so relieved I hadn't deleted the customer's database and only system files.”
Reginald did what all the best accident-prone people do – kept the cock-up to himself, hoped no one would notice and started covering his tracks, by recreating all the system files.
Over the next three hours, he “painstakingly recreated the entire device tree by hand”, at which point he could boot the machine properly – “and even the application worked out”.
Jubilant at having managed the task, Reginald tried to keep a lid on the heart that was no doubt in his throat by this point and closed off his work, said goodbye to the sysadmin and went home to calm down. Luckily no one was any the wiser.
“If the admins read this message, this would be the first time they hear about it,” he said.
“At the time they didn't come in to check what I was doing, and the system was inaccessible to the users due to planned maintenance anyway.”
Did you feel the urge to confess to errors no one else at your work knew about? Do you know someone who kept something under their hat for years? Spill the beans to Who, Me? by emailing us here. ®
Sponsored: Becoming a Pragmatic Security Leader