BOFH: The Silence of the Servers
The servers are quiet for now, Clarice
"Ah the long, dark reaches of intrigue!" I say faux-casually in the dark, scaring the Boss half to death.
"What the hell are you doing here?" he cries when he's collected himself, no doubt readjusting his underpants in response to the recent extra loading.
"It's a computer room, I belong here," I say, getting up from the chair I've been sitting in - strategically placed behind the swing of the server room door. "Which is more than I can say for you."
"I... was just stopping in on the way home to make sure everything is ship-shape," he snaps, reaching for the light switch.
"No, don't touch that!" I say as the server room door quietly >click<s closed. "Because if you're really here to check on things then it's best done in the dark..."
"In the dark?" he blurts nervously.
"Yes, in the dark," I reply calmly. "Look at the racks around you, what do you see?"
"Look closer agent Starling, what do you see?"
"I uh... servers?"
"Indeed. And what do these lights mean?"
"What do they mean? Each lamp has a purpose... Of each particular lamp ask: what is it in itself? What is its nature?"
"I... That the server's on?"
"No. That is incidental. What is the first and principal thing each lamp does? What needs does it serve by lighting?"
"I... That everything's ok?"
"NO! A lamp indicates STATUS. And STATUS is what we're interested in. And what do the lamps tell you?"
"That everything's ok?"
"Do they? Is that what they're telling you?"
"Because they tell me a completely different story."
"What... story... do they tell you?" the Boss asks uneasily.
"That orange lamp over there tells me the story of a server with a faulty power supply. That flashing red lamp over there tells me of a degraded RAID array. And those two flickering orange lamps over there tell me a server has two dodgy power supplies that it's switching between. And all these stories are part of one larger story."
"What's the larger story?"
"THE larger story."
"What's the larger story."
"You mean the name of the larger story?"
"I... yes, if you like."
"The name of the larger story is... The Silence of the Servers."
"The Silence of the Servers?!"
"Yes. It's an interesting story that I'm sure you'll like, involving a mid level manager, and electrical engineer and the deferment of the refurbishment of an UPS for financial reasons."
"I... don't know what you mean."
"Ok, once upon a time there was a technical person - me, for instance - who put the replacement of UPS batteries and refurbishment of the electronics onto his business plan - as a cheaper option that replacing the UPS outright. And in this story lets say that this technical person's manager would rather spend the money on something else - new computers with touch screen monitors for a call-centre - for instance."
"Those machines were in last year's budget! We'd committed to buying them when we replaced the call centre software!"
"As I say, this is just a story. So say, for instance the UPS wasn't refurbished and the batteries not replaced. Say the manager concerned didn't want to tell his technical person that this wasn't going to happen for financial reasons - for his own personal safety reasons. Say he got an electrical engineer to pretend to do some technical work and battery replacements but instead jumper the UPS input and output while taking off a small supply to power the lamps on the panel to look like it was still working normally..."
"I... don't know what you're talking about."
"Of course you don't. Because this is just a story. But say when this electrical engineer crept back in the wee small hours to do the internal bypass and panel rewire he didn't realise that the moment he disconnected the UPS internals he'd also be disconnecting the power to the SNMP agent inside the UPS that talks to our paging system..."
"Oh..." the Boss.
"And say, instead of jumpering input and output through a smoothing circuit he just connected all our expensive servers to dirty raw mains - dirty raw mains fed from the same board as the lift motor as it happens, a huge inductive load, I might add, introducing power factor changes, surges, back EMF, etc..."
"And say in the story that the technical person mentions to his manager that there could be a power problem. If the manager were innocent of wrongdoing he'd call the electrical engineer to check the UPS and if not he'd sneak back in the middle of the night to make sure that the lights on the UPS were still working properly so that noone would find out his dirty little financial secret."
"But anyway," I say, flicking the lights on. "The backstory is irrelevant, it's how the story ends that's important."
"How it ends?"
"Yes, say the story ends with a complete UPS replacement, then it's a happy ending. If it ends in lies and denial, well..."
"You know I think we might have budget for a UPS replacement," the Boss burbles.
"Excellent, I'll let the PFY know," I say, reaching for the cellphone.
"Really? Where is he?"
"He's in the basement carpark doing to your car what the engineer did to our UPS."
"I... but I thought you said it was a happy ending."
"Perhaps you should think of this more as a Roald Dahl happy ending then. Maybe that would help..."
Sponsored: Hyper-scale data management