Feeds

BOFH: Uninterruptible patsy supply

'We are having a special this week on proton charging and storage of the beast'

  • alert
  • submit to reddit

Designing a Defense for Mobile Applications

Episode 9 "What the fuck just happened?" the Boss garbles, crashing around Mission Control like a madman after dashing down two flights of stairs from the 4th floor boardroom.

"Uh.... UPS failure," the PFY says calmly, glancing up from his monitor briefly.

"Well aren't you going to do anything about it?"

"I am," he responds. "I have to shut down everything on the working UPS units so that I can synchronise and start everything up again from scratch. If I just bypass the broken UPS there'll be all sorts of problems."

Which, on the Pinocchio scale of lies, is a sixteen-foot piece of dowel with some nostrils cut in the end. The only true thing the PFY said is that there is a UPS failure...

"What sort of problems?" the Boss snaps back.

"Well the server-client systems will be all out of whack for a start but the replication of data is bound to fall over - then there's the electrical problems - phase imbalances, hysteresis loops, the possibility of brown-outs and spikes, not to mention the power factor correction that we'd need to put in place."

Utter bollocks. None of the crap is going to happen and the PFY knows it. In actual fact, all that's happened is a cheap UPS in a comms cupboard two floors above us has failed, and instead of failing into bypass – like any reputable UPS that you paid more than three shiny beads for – it just turned off.

All because the Boss, in his wisdom, decided that instead of "wasting" all that money upgrading the UPS feed from the server room to the comms risers for the increased POE switch demands, he'd simply order some standalone UPS units for every comms room in the building.

And so it was that a box of 20 3KVA UPS units arrived late last week.

That's right, a box, not a pallet. That's the first concern. Your average 3KVA UPS weighs enough to have the courier driver dousing himself in Brut 33 before attempting to haul it out of the back of his van, but the whole boxlot of these babies was easily wheeled into Mission Control on a sack barrow. They've probably got the life expectancy of a box of chocolates at a Weightwatchers weigh-in.

One of my personal warning signs on gear is when I get a brand name the same as the device – ie, with this UPS brand UPS from UPS Ltd, I'm fairly sure I'm in trouble. Perusing the copious documentation (a sheet of badly photocopied paper in poorly worded English warning me about supplying the connector of the utility in the reversal format), I note the complete lack of company name, model number or service information.

This should be good.

...

Where your average UPS would have bar graphs to indicate battery capacity and load, this one appears to have a bulb behind a piece of green plastic with boxes crudely stencilled on it.

The supply transiting operator (power switch) had just the sort of tactile response you'd expect from a thin piece of plastic breaking under finger pressure as the device fired up (without the fire, surprisingly).

And they were installed.

It was only later that the PFY that discovered that if you liven up the network monitoring port from a PoE switch, the thing would overheat and crap out in about five minutes...

It's probably a design feature.

"Well what are you going to do meantime? We're trying to give a presentation to the board! It's very important!"

"I know it is," The PFY says. "You told me that three days ago and so I've made all the infrastructure you need super-reliable by putting UPS units inline for all the gear. There's one in the ceiling powering the datashow, one at the distribution switch, one on the fibre transceiver and even one on the fridge that's keeping the drinks cool - although according to nagios that one stopped responding late last night."

"Which one is broken now?"

"When you say now do you mean now or 10 minutes ago when you were in the boardroom? >clickety<" the PFY asks.

"I... 10 minutes ago."

"That would be the UPS on the distribution switch."

"And... what.. about now?" the Boss asks.

"Ah, well, the Datashow one stopped responding about a minute ago."

"Damn it!" the Boss snaps.

"Everything on the 6th floor is working OK if you wanted to move everyone up there?" the PFY suggests. "You could put get them to meet you up there while I put your presentation on a USB stick."

"Yes! Yes, let's do that."

The Boss makes a quick call to the 4th floor boardroom while the PFY drags some mundane PowerPoint files onto a USB drive.

"Right, 6th floor!" the Boss says, dashing for the door.

"Oh there's no hurry," the PFY says.

"No hurry? Why?"

"It looks like the UPS unit on the lift controls has stopped responding so it will have stopped. Looks like it's in between 4 and 5."

"WHY THE F*** DID YOU PUT A UPS UNIT ON THE BLOODY LIFT CONTROLS!!!?!?"

"Because the lift is critical infrastructure. You wouldn't want the lift stopping between floors because of a panel failure and trapping someone in there. Especially between 4&5, because the 5th floor door has the broken emergency access lever so they'd have to put a ladder down from 6 to get the board members out."

"Can't you put it in bypass?"

"Yes, well, I tried that on the one that failed yesterday but the unit just started smoking and an orange Ghostbusters light came on. Besides, it's on the roof of the lift car and that's stuck between 4&5 and I'd need a ladder to get to it..."

"Ffffff...!" the Boss blurts again, dashing to the door Mission Control

"So the lift really is stuck?" I ask.

"Oh yeah, It's as stuck and the automatic door release on our d... >crash< ..oor," the PFY says, and the Boss slumps to the floor.

"Yeeeeeessss," the PFY continues, firing up the console port of the basement distribution switch. "We're about 12 power inline static max commands away from a complete building shutdown."

"Time is awasting!" I yell, noticing the impending approach of lager o'clock.

>clickety!< ®

The Power of One eBook: Top reasons to choose HP BladeSystem

More from The Register

next story
Apple fanbois SCREAM as update BRICKS their Macbook Airs
Ragegasm spills over as firmware upgrade kills machines
Attack of the clones: Oracle's latest Red Hat Linux lookalike arrives
Oracle's Linux boss says Larry's Linux isn't just for Oracle apps anymore
THUD! WD plonks down SIX TERABYTE 'consumer NAS' fatboy
Now that's a LOT of porn or pirated movies. Or, you know, other consumer stuff
EU's top data cops to meet Google, Microsoft et al over 'right to be forgotten'
Plan to hammer out 'coherent' guidelines. Good luck chaps!
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
FLAPE – the next BIG THING in storage
Find cold data with flash, transmit it from tape
prev story

Whitepapers

Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.