BOFH: Everyone deserves a little DOWNTIME

Episode 14

Isn't it always the way that when you're hours from the office sporting only a smartphone with a dodgy data connection that something critical claps out back at work.

And then something's wrong with the VPN so you have to gain access through a convoluted chain of remote desktop, ssh and telnet links just to end up with a PowerShell interface which should NOT be accessible to you - or anyone. All because you need to bypass the security you not-so-carefully crafted – to perform a reboot.

And just when you're about to hit the final <enter>, some *GIT* rings you on the phone, which sends the command session you just had going into limbo.

"Yes?" I seethe.

"Just wondering if you know that the financials server is down?" the Boss says.

"It's not down, it just looks like it's down because the DNS component of the domain controllers has some syncing issue which is causing it to return deprecated DNS data." "I... what now?"

"Something's wrong in the main brain of the servers.” I explain carefully. "I need to look at it to see why it's wrong, but I can't do it while I'm being phoned."

I hang up and continue.

>tapitty< >prod< >RRRIING!<

"I told you - I can't do it while I'm being rung!!!"

"Yes I know," the Boss blurts quickly. "I was just wondering if you could tell me when things would be back as there's people wanting to work."

"I was going to say I don't know," I respond, "but now it's I don't know PLUS the time of this phone call, PLUS the time to navigate back to the remote desktop app."

"So you don't really have any idea?"

"I don't know exactly what's wrong yet, so no."

"Should I leave you my number?"

"The one that's the last two entries on my recent calls list?"

"Yes. I see. Well I'll just wait for your call. Any idea when tha.. >click<"

>tappity< >prod< >prod< >swipe< >RIIIINNNNG<

"Yes?"

"Hi, I know you've talked to Alan but I was just wondering if you could give us a ROUGH idea of when things will be working again. ROUGH - doesn't have to be too accurate," the IT Director says.

"Oh well, in that case, this week some time."

"Ha ha, yes. Well played. Maybe a little more accurate than that?"

"I could narrow it down to the days in this week which have not already passed..."

"Well, will it be up today?" he asks.

"Before people started calling me, I would have said possibly - but now I'm not so sure."

"Well can you give me likelihoods? Like a percentage guess on how likely it would be to be up and running this afternoon."

"Well off the top of my head I'd say it was something like a 78 per cent chance."

"So 78 per cent?"

"No, LIKE 78 per cent. So it could be 7 or 8 or 78 per cent. It's hard to tell with all these calls."

"I see, well I'll just let you get on with it. If you could just give me a ca.."

>click<

>tappity< >tappity< >tweak< >tweak< >click< >prod< >prod<

Moments later I give the PFY a call.

"Where are you?" I ask.

"At the pub," the PFY says. "It's lunchtime."

"OK could you pop over the road and kick the domain controller in the guts, AFTER you pull the plugs on the backup DC? I think the backup is poisoning the primary."

"Not really," the PFY says.

"Why not?"

"I'm in Paris."

"YOU SAID YOU WERE IN THE PUB!"

"I am - I'm picking up some USB sticks."

"WHY FROM BLOODY PARIS?!!"

"Well you know how the Boss said he HAD TO HAVE a particular brand and a particular size of stick, AND he didn't care where I got them from so long as I got a good price? Well, I worked out that with the Eurostar special fare it was actually CHEAPER if I was buying more than 47 of them. So I'm sitting at a bar at Gare du Nord with a carton of 50. AND I've saved the company somewhere in the vicinity of 18 quid."

>click<

>tappity< >tappity< >click< >click<

Now I'm calling on the powers of memory that I don't have to remember to try and guess the address of the lights out management card. Worse still - I never really configured any of them outside of an address so they pretty much all look the same once you login.

>tappity< >clack!<

A quick ping tells me that the BDC is still there, so I've turned off something else in that server rack. I'm sure the monitoring system will page me in due course.

Unless I just turned off the monitoring system.

>tappity< >tappity< >click< >click<

>tappity< >clack<

Nope, another dud.

>tappity< >tappity< >click< >click<

>tappity< >click< SUCCESS! The BDC on port 9 has stopped responding to pings! I send a reboot to the PDC - which I know for a fact has an IP address one less than the BDC and pop into a pub to get myself a lunchtime lager.

>RING<

"Did you fix it?" the Director asks. "Only it looks like some people are getting back in."

"Yes. It should be right in half an hour or so. Now that the backup domain controller is out of the picture people will start getting information from the right one."

"So this... domain controller.. Should we replace it?"

"Undoubtedly."

"How soon can you do it so this doesn't happen again?"

"Well if I go through our normal suppliers, maybe a week or two."

"We have to do it sooner than that. We can't have this happen again! Are there no other options?"

"Well I could go and get one direct, which would take a day or two..." I say, directing Gina's gaze to the Amsterdam specials in a travel agents window...

I really should have written those IP addresses down.

• BOFH: The whole shebang
• The Compleat BOFH Archives 95-99