Sysadmin bloodied by icicle that overheated airport data centre
It's snowing at 3AM. Servers are dying by the dozen. So our reader went outside and climbed a ladder to fix things up
On-Call Why look at that: the calendar says “Friday” which means it's time for another edition of On-Call, The Register's weekly column in which we relate readers' recollections of being called out to fix nasty things under nastier circumstances.
This week, meet “Alan” who “used to work for a rather large airport company in England.”
One chilly mid-winter night the airport where he worked was struggling with a substantial amount of snow that had grounded planes and filled the place with passengers and staff who couldn't go anywhere. It wasn't so bad in the data centre, as Alan and his mates were watching television on what looked like being a hassle-free night shift.
Of course it didn't last. Alerts started appearing about the temperature in the data centre and Alan says “It went from 24°C to 50°C over the course of around 15 - 20 minutes.”
Alan quickly determined that “every single one of the blowers had tripped”. A quick call to facilities, which told him how to reset things … didn't work.
And now the quiet night shift was a full-blown emergency because “At this point, the UNIX cluster is down, the SAN is down, around 50 servers had suffered thermal failure and we're looking at another 1000 servers ready to go down.”
The facilities team quickly arrived, started lifting the data centre floor and quickly struck water “Literally everywhere underneath the floor.”
Alan and team found the source. “The blowers were pissing water out everywhere,” he told us, but “we couldn't figure out why because the drainage system should sort this as it had two pipes, one going outside and one on the internal waste water system.”
But it turned out that there was just one drain going to the great outdoors and seeing as it was below zero outside that drain had frozen shut.
At which point Alan got the job of going outside, scaling a ladder and wielding a blow torch to “heat up what seems to be a pretty big block of ice in the 10mm copper tube.”
Not long after ascending said ladder and applying some heat, Alan was “hit in the face with a rather large, rather cold, cylinder of ice that had clearly been under a lot of pressure from the water behind it.”
“It was a good thing it had been snowing because the six foot fall backwards onto concrete with claret all over my face was the least of my problems,” he told us. Higher on his list of immediate priorities was “the gush of water resembling Niagara Falls” that exited the tube and soaked poor Alan.
We've mentioned it was well below zero, so readers can imagine Alan was by now feeling very comfortable and very happy indeed. To make matters worse the airport was still snowed in and as it was 4:00 AM he was hours from being able to get home to find some dry, warm clothes.
Three hours of cold, wet, shivering later Alan made it home.
The situation sorted itself out. Alan says that “over the next few weeks we ended up replacing around 50 failed disks in the SAN. Two servers completely fell over and were unserviceable, the UNIX cluster blew a PSU and it was emotional, to say the least...
“It was quite entertaining, though I'd rather not do it again.”
Have you been injured when fixing a tech problem? Or done a colder, nastier, job than Alan's winter adventure? If so, write to let us know and your story could appear in a future edition of On-Call. ®
Sponsored: Becoming a Pragmatic Security Leader