Feeds

BOFH: The Silence of the Servers

The servers are quiet for now, Clarice

  • alert
  • submit to reddit

Choosing a cloud hosting partner with confidence

Episode 5

"Ah the long, dark reaches of intrigue!" I say faux-casually in the dark, scaring the Boss half to death.

"What the hell are you doing here?" he cries when he's collected himself, no doubt readjusting his underpants in response to the recent extra loading.

"It's a computer room, I belong here," I say, getting up from the chair I've been sitting in - strategically placed behind the swing of the server room door. "Which is more than I can say for you."

"I... was just stopping in on the way home to make sure everything is ship-shape," he snaps, reaching for the light switch.

"No, don't touch that!" I say as the server room door quietly >click<s closed. "Because if you're really here to check on things then it's best done in the dark..."

"In the dark?" he blurts nervously.

"Yes, in the dark," I reply calmly. "Look at the racks around you, what do you see?"

"Machines."

"Look closer agent Starling, what do you see?"

"I uh... servers?"

"Clo-ser..."

"Lights."

"Indeed. And what do these lights mean?"

"Pardon?"

"What do they mean? Each lamp has a purpose... Of each particular lamp ask: what is it in itself? What is its nature?"

"I... That the server's on?"

"No. That is incidental. What is the first and principal thing each lamp does? What needs does it serve by lighting?"

"I... That everything's ok?"

"NO! A lamp indicates STATUS. And STATUS is what we're interested in. And what do the lamps tell you?"

"That everything's ok?"

"Do they? Is that what they're telling you?"

"I..."

"Because they tell me a completely different story."

"What?"

"Hmm?"

"What... story... do they tell you?" the Boss asks uneasily.

"That orange lamp over there tells me the story of a server with a faulty power supply. That flashing red lamp over there tells me of a degraded RAID array. And those two flickering orange lamps over there tell me a server has two dodgy power supplies that it's switching between. And all these stories are part of one larger story."

"What?"

"Hmm?"

"What's the larger story?"

"THE larger story."

"What's the larger story."

"You mean the name of the larger story?"

"I... yes, if you like."

"The name of the larger story is... The Silence of the Servers."

"The Silence of the Servers?!"

"Yes. It's an interesting story that I'm sure you'll like, involving a mid level manager, and electrical engineer and the deferment of the refurbishment of an UPS for financial reasons."

"I... don't know what you mean."

"You're sure?"

"Positive..."

"Ok, once upon a time there was a technical person - me, for instance - who put the replacement of UPS batteries and refurbishment of the electronics onto his business plan - as a cheaper option that replacing the UPS outright. And in this story lets say that this technical person's manager would rather spend the money on something else - new computers with touch screen monitors for a call-centre - for instance."

"Those machines were in last year's budget! We'd committed to buying them when we replaced the call centre software!"

"As I say, this is just a story. So say, for instance the UPS wasn't refurbished and the batteries not replaced. Say the manager concerned didn't want to tell his technical person that this wasn't going to happen for financial reasons - for his own personal safety reasons. Say he got an electrical engineer to pretend to do some technical work and battery replacements but instead jumper the UPS input and output while taking off a small supply to power the lamps on the panel to look like it was still working normally..."

"I... don't know what you're talking about."

"Of course you don't. Because this is just a story. But say when this electrical engineer crept back in the wee small hours to do the internal bypass and panel rewire he didn't realise that the moment he disconnected the UPS internals he'd also be disconnecting the power to the SNMP agent inside the UPS that talks to our paging system..."

"Oh..." the Boss.

"And say, instead of jumpering input and output through a smoothing circuit he just connected all our expensive servers to dirty raw mains - dirty raw mains fed from the same board as the lift motor as it happens, a huge inductive load, I might add, introducing power factor changes, surges, back EMF, etc..."

"I..."

"And say in the story that the technical person mentions to his manager that there could be a power problem. If the manager were innocent of wrongdoing he'd call the electrical engineer to check the UPS and if not he'd sneak back in the middle of the night to make sure that the lights on the UPS were still working properly so that noone would find out his dirty little financial secret."

"Uh..."

"But anyway," I say, flicking the lights on. "The backstory is irrelevant, it's how the story ends that's important."

"How it ends?"

"Yes, say the story ends with a complete UPS replacement, then it's a happy ending. If it ends in lies and denial, well..."

"You know I think we might have budget for a UPS replacement," the Boss burbles.

"Excellent, I'll let the PFY know," I say, reaching for the cellphone.

"Really? Where is he?"

"He's in the basement carpark doing to your car what the engineer did to our UPS."

"I... but I thought you said it was a happy ending."

"Perhaps you should think of this more as a Roald Dahl happy ending then. Maybe that would help..."

Remote control for virtualized desktops

More from The Register

next story
Azure TITSUP caused by INFINITE LOOP
Fat fingered geo-block kept Aussies in the dark
NASA launches new climate model at SC14
75 days of supercomputing later ...
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
Cloud unicorns are extinct so DiData cloud mess was YOUR fault
Applications need to be built to handle TITSUP incidents
NSA SOURCE CODE LEAK: Information slurp tools to appear online
Now you can run your own intelligence agency
BOFH: WHERE did this 'fax-enabled' printer UPGRADE come from?
Don't worry about that cable, it's part of the config
Stop the IoT revolution! We need to figure out packet sizes first
Researchers test 802.15.4 and find we know nuh-think! about large scale sensor network ops
DEATH by COMMENTS: WordPress XSS vuln is BIGGEST for YEARS
Trio of XSS turns attackers into admins
SanDisk vows: We'll have a 16TB SSD WHOPPER by 2016
Flash WORM has a serious use for archived photos and videos
prev story

Whitepapers

Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
Getting started with customer-focused identity management
Learn why identity is a fundamental requirement to digital growth, and how without it there is no way to identify and engage customers in a meaningful way.
Go beyond APM with real-time IT operations analytics
How IT operations teams can harness the wealth of wire data already flowing through their environment for real-time operational intelligence.
Why CIOs should rethink endpoint data protection in the age of mobility
Assessing trends in data protection, specifically with respect to mobile devices, BYOD, and remote employees.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?