Tech team trapped in data centre as hypoxic gas flooded in. Again

Trouble started with a boiling battery and got worse when a ladder fell over

Who, me? Welcome once more to “Who, me?”, in which we help Reg readers to unburden themselves by telling anonymised stories of big, bad, mistakes.

This week meet “Luke” who told us that about 20 years ago he was in charge of disaster recovery “for an insurance company in San Francisco” and had “a series of ‘oh shit’ days.”

Lue’s troubles started because “Someone had put a small uninterruptible power supply (UPS) on a shelf on a rack in our main data center some years earlier.”

“This unit was NEVER serviced and eventually croaked by over-charging its battery.”

And croaked spectacularly! Luke told us the UPS boiled and “caused fumes which were seen by a passer-by looking into the floor-to-ceiling glass house windows.”

“That person ran into the data centre and pulled the halon” – the hypoxic gas used to snuff out fires in a data centre.

“Of course, everything shut down hard,” Luke told us. Which was bad – the incident took place mid-week so disruptions were significant.

Luke and his team sorted things out, but “Just as we had finished accepting the cleaned up data centre and we'd returned production back from the disaster recovery site, an electrician's ladder that was propped on the wall next to the door awaiting removal toppled and slid along the wall.”

Ethernet cable rises up like a snake (artist's impression). Image via shutterstock

Tired sysadmin plugged cable into wrong port, unleashed a 'virus'

READ MORE

And guess what? “The emergency power off box was smashed off, and the halon switch was caught and pulled.”

“The halon blew out the tiles again, but this time with CIO, me, facilities guys, and techies in the fog.”

Luke and his colleagues scampered out without incident and the insurer failed over to its DR site. Again.

“If we hadn't been preparing for Y2K we'd probably would have been sacked,” Luke said.

Have you ever breathed halon? Or been trapped in your bit barn? If so, click here to write to “Who, me?” and we might just give you a gig on a future Monday. ®




Biting the hand that feeds IT © 1998–2018