Would YOU start a fire? TRAPPED in a new-build server farm
When the going gets tough, the tough get messy
This Damn War “What’s the smallest fire I could start to be noticed, but not so big that I risk burning down the building?” is one of the stranger thoughts to have entered my head, in many years of working in IT. No, I'm not a closet pyromaniac, so why was I entertaining such thoughts?
I had found myself stuck in a data centre on a Sunday afternoon and by that point, I’d been in there for over two hours with no sign of rescue. That shouldn’t have been the case, of course, but a series of unfortunate events had led me down that road.
The data centre in question was undergoing an enormous refurbishment, and the rack I was working on was one of the last stragglers in an aged hall that was waiting to be decommissioned. When I arrived, the on-site team were very conscientious in pointing out open floor tiles and various other construction-related trip hazards; unfortunately, the contractors had been rather less meticulous.
I completed the work I needed to, then packed up my bag to leave. When I got to the exit however, my access card wouldn’t unlock the door. After a few frustrated swipes I tried the emergency door release button, also to no avail. Following some rather pitiful shoulder bunts against the door, to see if the release was working but the door itself was physically stuck – it wasn’t, it hurt – I conceded that the contractors must have fouled up the access control system somehow.
“Not a problem,” I thought to myself, and picked up the telephone on the wall adjacent to the door, normally a direct line through to the operations centre. The line was dead, another casualty of the building work.
“Still nothing to worry about,” I mused, and started waving my arms in front of the CCTV camera above the door to get someone’s attention.
I jumped around in front of that camera for far longer than I’d care to admit, before I resigned myself to the fact that nobody was watching (or rather, able to watch) and sat down on the floor to take stock of my options.
I first reached for my mobile phone which was almost fully charged, but had no signal in the metal sarcophagus of the data centre’s innards. Next up was my laptop: I could grab a network cable, plug directly into the network switch in my rack, and contact the outside world to achieve my freedom. Unfortunately, that was my third stop of the day and I’d already worn my laptop’s meagre battery down… and the power supply was on my desk at home. There wasn’t even a KVM trolley in the hall to use one of our servers to call for help, another casualty of the decommissioning process.
Deflated, I sat on the floor, and wondered how long it would take for someone to notice I was gone. I hadn’t given my wife an ETA for arriving home and as far as my colleagues were concerned this was a “pop in on my way past” flying visit, so they wouldn’t be expecting to hear from me again. By this point, I’d been in there for well over an hour and nobody had been to check on me; I was starting to get worried.
What got me really worried was when I couldn’t open the fire door at the back of the hall. I could depress the release bar, but there was some construction detritus behind the door that meant it simply wouldn’t budge. Cue more pathetic shoulder charging, to no avail.
I hammered on the fire door for a while and shouted myself hoarse, but nobody heard, so I tried the main door. Shouting for help turned into profanities, which turned into pleading and plaintive wails, but no help was forthcoming.
Again, I sat on the floor and stewed for a while, and by the two-hour mark I was starting to feel desperate, which probably explains the (admittedly, very poorly-thought out) ‘start a small fire’ plan; quickly shelved, when I realised the fire alarm didn’t go off when I tried the door.
As I began to contemplate a night in my personal air-conditioned hell, a thought occurred to me: the fire alarm might have been yet another casualty of building works, but it wasn’t the only kind of monitoring.
I ran over to my rack and pulled a couple of redundant power cables; not serious enough for an outage, but enough to generate some alerts from our own monitoring. I replaced them and did the same with more servers – lather, rinse and repeat – until the constantly flapping monitors caused one of my team to ring the data centre, to ask them to eyeball our rack and check for ‘power problems’.
All told, I’d been trapped in that data centre for less than three hours, but even so when that door did finally open I felt like Shawshank Redemption’s Andy Dufresne. ®
Sponsored: Becoming a Pragmatic Security Leader