It's BOFH Disaster Recovery Time
Run for the Hills
It's Thursday, Payday, and The Boss has his bee in a bonnet about something. You can tell, because he's wandering around outside the office rehearsing his lines like a C-grade actor.
"Morning, How are we all?" he Lou-Diamond-Phillips' us.
"Hello!" The PFY responds graciously, dragging a chair over for The Boss to join our morning coffee circle.
"Ah - I've just been reading a report from the Company Auditors," he starts, ignoring an unprecedented show of respect by The PFY.
"Really, and what did they have to say?" The PFY asks conversationally, leaning forward to feign interest like a professional.
"Well APPARENTLY, we don't have a Disaster plan - and without one, they won't give us an A double plus rating!"
"The BASTARDS!" The PFY gasps. "But wait a minute, I thought we had a disaster plan?"
"We have several, in fact," I respond, "although I don't believe the auditors have ever discussed the matter with us - so perhaps that's where the problem has arisen."
"Really?" The Boss blurts. "Well, I'd have to put my hand up and admit I talked to them about it, but didn't realise that you'd put work into it!"
"I'd like to think I've devoted part of my LIFE to it!" I respond, with just a hint of emotion bubbling under the surface.
"I see. Well would you mind running over a few of your options then?"
"By all means! Check out this white board. Plan One, for instance, is to start a fire in the basement furniture storage room, which, once it gets a hold, would move on through the non-fire-rated wall into the backup generator room, which has a heeeeeeuge tank of diesel in it. Once that baby gets going, it's all ov.."
"Ah, no. I think I mean.."
"Ok, not what you're looking for - bear with me. What about we drop something really heavy on the gas main where it comes into the building? It's just a matter of time 'til a spark wi...
"NO! I don't want to CAUSE a disaster, I want to FIX one!"
"Microsoft Out, Linux in then?"
"I think he's talking about Disaster RECOVERY planning," The PFY adds helpfully.
"OH I SEE!" I gush. "You want to know about the plans to fix up the place when something terrible happens!"
"YES!" The Boss blurts. "And we're on a tight timeframe for this as we need to get rated before the end of the month!"
"Oh, OK," I murmur sagely. "So you'd like to know what we'd do in a disaster?"
"OK, well obviously it depends on the disaster."
"Yes, yes, but what's the plan in case of a large fire?"
"Well first and foremost, if we're in the building at the time, whichever of us is closest goes into the computer room and disables the Halon lockout."
"And the other person?"
"They run to the Beancounters area and instruct them all to stay away from windows, and place themselves in the safe areas under tables, in doorways or in cupboards."
"Uh, isn't that the procedure for an Earthquake?"
"Not for Beancounters, no. The Beancounter earthquake procedure is..." I reply, handing over to The PFY.
"..is to stand in the safe areas in front of heavy bookcases, underneath large, heavy objects, or beside plate glass windows."
"EXACTLY!" I cry.
"I don't think you've really thought about that," The Boss comments.
"Oh no, we've thought about it alright. Just check out the EMERGENCY PROCEDURE pages in their internal phone book. I'm especially proud of the Bomb Scare section."
"Is that the one where they run straight at the armed police screaming 'You'll never take me alive you bastards!'?" The PFY asks.
"The very same!"
"I LOVE that bit!"
"Moving right along," The Boss continues, preferring to ignore that quagmire for a moment. "What I'm after is a plan for how we RECOVER our services once the immediate danger is over."
"I believe the plan is that we wander on into work and take whatever action seems appropriate after a survey of the site, the damage, and the services to the building," I respond.
"That's not a plan!"
"Yes it is! It's a great plan!" The PFY replies defensively "You paid for me to go to a three day disaster recovery course to learn about that!"
"But didn't they cover... uh.. >scrabble< 'Hot Sites'?" he asks, referring to his notes.
"DID THEY WHAT! There was this strip club down the road from my hotel..."
"WHERE YOU CAN RUN YOUR OPERATIONS FROM!!"
"Oh them! Yes, they mentioned them, but it's a disaster - we wouldn't have the data on hand to recover from! Let alone the database version installed on their kit to run it. IF we don't get bumped down the chain a bit because we're not the hot site's number one customer - and let's face it, if we're up against some Investment Banking group, we won't have the money to buy our way in."
"We don't have the data?" The Boss asks, avoiding the real issues like a pro. "We pay three thousand quid a year for an offsite tape service! They come every day! Sometimes TWICE a day! I've seen them!!!"
Now wouldn't be the time to tell The Boss that the tape bloke's delivering tapes alright, but the tapes in question come from his local video shop in Bromley.
"And a good service it is too!" I respond. "But in a real disaster, the roads and public transport will be up the pole, IF the Data service is allowed back in THEIR building to get the media for us. IF we're THEIR number one customer...."
"So we're screwed whatever we do?" The Boss sighs.
"Yep - that's why it's called a disaster. We only have personal recovery plans here."
"Send each other's contracting companies bust by suing each other for negligence before this company can get to us. Then hide in the Third World (Liverpool) till the noise dies down, and get a new contract with another company."
"I see..... Could I get a copy of that please?"
"Sure, Not a problem!" ®
BOFH is copyright © 1995-2001, Simon Travaglia. Don't mess with his rights
Sponsored: Hyper-scale data management