‘I crashed a rack full of servers with my butt’
There’s a reason racks are parallel, not L-shaped
Who, me? Welcome for the third time to Who, me? The Register’s new column in which readers ‘fess up to messes of their own making.
This week, meet “Digby” and his powerful rear.
Digby once worked for a business that started life as a division of another company. He arrived just as the new outfit was getting to his feet and found “the departing ‘engineers’ shoved everything into a fairly small server room, with three racks in an L-shape, cabled it up to get it working, and then buggered off.”
“One day,” Digby continued, “I was in the server room, re-patching some phones with the main server rack and UPS next to me”
“Suddenly, I heard the fans of our Oracle server behind me screaming up to full-crescendo - which typically means that only one of the two power supplies is working.”
That’s not good news, so Digby turned around to figure out what was going on. His instinct was right, one power supply was indeed off. A few seconds later Digby’s boss informed him that e-mail and the file server had gone down too.
Sysadmin crashed computer recording data from active space probeREAD MORE
Digby quickly figured out the cause of the problem: his own posterior.
“When I crouched down, my arse caught one of the trip switches at the top of the UPS,” he told Who, me? His unusual On/Off technique had only been made possible by the L-shaped rack arrangement, which meant cables and switches had let it all hang out.
“This knocked off one of the power feeds to the Oracle server. Then in my haste to quickly check this, I had unknowingly dislodged the feed to one of the distribution units in the rack.”
“The geniuses that had cabled up the servers in haste the previous year had fed several servers with twin feeds from the same distribution unit (and therefore the same UPS) - also a very dumb move indeed. When that distribution unit lost power, it knocked offline about six servers.”
Finding the dislodged cables didn’t take long and Digby quickly explained that the L-shaped server room was the reason everything had gone-pear-shaped. Better yet, his boss believed him – and even signed off on “a few hours of overtime one weekend to re-cable the servers so that they were being fed via two distribution units and two UPS's going forwards.”
What have you taken down with your extremities? Share your story with Who, me? and it could be your tale of woe – or, like this poor reader, tail of woe – here on some future Monday morning. ®
Sponsored: Becoming a Pragmatic Security Leader