Sysadmin sticks finger in pipe, saves data centre from flood
Then the suits boarded up an exhaust vent that dirtied their BMWs
On-Call Thank Crom it's Friday! At the end of today you can get on with what is best in life, which we hope includes reading this instalment of On-Call, our weekly look at legendary tales of IT gone wrong.
This week, meet “George” who tells us he “once worked for a fairly large web hosting company which had a small data centre in the Midlands and a larger one in London.”
George's story starts with the time his phone rang at 06:00 and the folks on the night shift squawked “THERE'S WATER COMING THROUGH THE CEILING!”
Indeed there was: a pipe joint on the first floor burst when the heating came on and a million-gallon header tank was obeying the laws of gravity.
“Cue panic as water rained from every available light fitting onto hundreds of racked and shelved machines, mostly colocations, on the ground floor.”
Amazingly, only one machine failed: a top-of-rack switch's power supply couldn't handle the deluge.
And the rest?
George says “We ended up with one of the night shift support guys shoving his thumb in the end of the pipe until our plumber arrived to isolate everything, a matter of about 45 minutes.”
On another occasion George's unplanned wake-up call came even earlier – 05:00 – and this time the capitalised panic was because “THE DATACENTRE IS ALL DARK EVERYTHING'S OFF IT'S QUIET AND IT SMELLS FUNNY AND I DON'T KNOW WHAT TO DO.”
This call set off a run of electrical incidents that took George years to shake.
On this occasion an uninterruptible power supply was the culprit. George says it “had literally exploded but unfortunately had exploded in a way that took out the bypass functionality.”
And also stank. He doesn't know why and we're afraid to ask. Big batteries are chemically scary before opining on their odours.
George's next electrical adventure came on a Friday afternoon when the mains failed, the UPS kicked in, the generator started – and the automatic transfer switch blew up as the UPS handed the load over to the generator.
The UPS depleted – what else could it do? – and the data centre went down. Completely.
Thankfully suppliers understood the urgency of the situation and new transfer switches appeared in time for a restart that evening.
As George's career progressed, his employer was acquired by a larger concern that had data centres all over Britain.
“Like a bad penny it seemed that every time I visited one, something power related would happen within a few weeks. We had an electrician dislodge a lug from a cable tray which fell into a distribution panel (the UPS on that system really didn't like that). In another data centre the building management decided they didn't like the dirty vents next to the car park so boarded them over, meaning the generator behind the vents had no air to breathe or be cooled with (and got predictably upset).”
George left that company because he tired of the travel and got a new job at home. “First big problem at new job? Electrical.”
He thinks he's shaken the curse now. If you've still got one that's dogging your job, write to me and we may decide to put your adventures into a future edition of On-Call. ®
Sponsored: Becoming a Pragmatic Security Leader