Wham, bam, thank you scram button: Now we have to go all MacGyver on the server room
Gone fishing for power – with a coat hanger
Who, Me? Monday has arrived, and with it another tale to send administrators scurrying for their event logs, and engineers reaching for the coat hanger. Yes, it's Who, Me?
"Jon", for that is not his name, was working for a now long-defunct IT giant back at the end of the 1990s.
Fortunately for those depending on the big iron for number crunching, Jon's duties involved more down-to-earth hardware. While Novell Netware was still a big thing back then, several servers were running the upstart Windows NT 3.51 operating system. Windows NT 4 had also begun its slow slither into the data centre, and one of Jon's responsibilities was lugging kit around.
Recalling the day in question, Jon told us: "I was wheeling a trolley of servers out of a server room."
Perhaps not being quite as careful as he might, Jon backed towards the door of the server room when the inevitable happened.
"I reversed into the scram button."
The origin of the name of the big bad button is tricky to pin down. Some insist it comes from the act of shutting down an emergency reactor through the insertion of control rods. Legend has it that one of the first reactors required a rope to be cut in order to drop a control rod into the pile if things went wrong. The individual responsible for the cutting was known as the "Safety Control Rod Axe Man" or SCRAM.
Hence the "scram" button.
The United States Nuclear Regulatory Commission (NRC) has a far more satisfying explanation – after hitting that big red button, what do you do next? "You scram... out of here."
Remember the Dutch kid who stuck his finger in a dam to save the village? Here's the IT equivalentREAD MORE
In Jon's case, hitting the button had cut the power to the entire room. There was no Mark-like escape for him by keeping a buttock pressed on the switch. The server lights had gone out.
"Dozens of servers," he told us, "powered down and the whole room went ominously quiet."
Though the scram button was easily accessible (for safety reasons), the servers themselves weren't. The rack doors were securely locked and, of course, Jon didn't have all the keys to hand. There was no access to the power switches in order to turn things back on.
What to do? It was 6am, and as a humble contractor Jon could expect to receive the mother of all bollockings for his misdeed.
He could not poke his fingers through the mesh of the cabinet doors, but maybe something else would do the trick.
One straightened coat hanger later, "borrowed" from security, and Jon was gingerly firing up the servers by poking the wire through the doors.
"It took me about two hours to get the whole lot back again and working," he told us, as some boxes came up in the wrong order "and had to be rebooted with long-press on the power buttons with my fishing pole."
Astonishingly, no one appeared to really notice the outage. "Some logs were checked," said Jon, "but the systems were owned by several different groups (test, dev, prod, corp, etc) so no one actually registered that the whole room had gone down."
Gotta love that corporate communication.
"As the racks were locked, it was probably assumed that someone with keys had done the deed."
Jon did not have the keys, but added: "What almost certainly saved me was the fact that back then there was no central timesync, so the logs would have all showed different times of the restarts.
"A while later I noticed that some of the power buttons had tell-tale scratches on them..."
Twenty years later, the memory of power-button fishing still gives Jon "cold sweats whenever I see a locked rack door".
Ever gone fishing in the server room, or found yourself dealing with a self-inflicted problem with a MacGyver-esque solution that would gladden the hearts of the A-Team?
Of course you have, and you should share your ingenuity with the kindly vultures of Who, Me? ®