Original URL: http://www.theregister.co.uk/2007/08/03/bofh_episode_27/
BOFH: Dodging the auditors
The PFY has a lucky escape
"Oops," the PFY says, looking away from his screen guiltily.
"I think I've just dropped the indexes from the financials database."
"Ok," I say. "No need to panic, we can probably recreate the indexes on the fly from the table data."
"I dropped the tables as well."
"So what you meant to say when you first said 'oops' was 'oops I've deleted our financials database.'"
"Backups then," I say. "You didn't delete them as well did you?"
"No, no, they're automated - the finance people schedule them for 11:30pm from their web interface. The only interaction I have is to change the incremental tape each morning when it's ejected and the full tape every monday when it's ejected ."
"And when was the last time the full tape ejected?" I sigh, fearing that this conversation is dragging a little.
"Uhhh," the PFY says, rolling his eyes a little. "Three weeks ago."
"THREE WEEKS AGO!" I gasp. "You mean those idiot beancounters haven't scheduled a bloody full backup in THREE WEEKS! It'll take days to roll that forward!"
"Oh they've scheduled them alright," the PFY says. "They've just not run."
"WHY HAVEN'T THEY RUN?" I ask, gritting my teeth a little.
"It's because of the financial policy change," the PFY says.
"Policy change?" I prompt, wondering if the PFY has suddenly become unable to form a sentence complete with subject, object, verb, adverb and intelligence.
"Yeah, you know about three weeks and two days ago the Beancounters changed the way in which they would pay recurring invoices?"
"No, but go on."
"They decided that if an invoice was a recurring one that varied in amount month to month the fixed component of the bill could be paid in advance but the varying portion should be paid for in arrears."
"And whatever that means has... some effect on backup scheduling?"
"WHAT IT MEANS contractors like us who are paid in advance would get paid their normal time in advance but their overtime in arrears."
"Uh-huh - and this means?"
"It means that there would be a two month gap between successive overtime payments when the policy was enacted."
"...Three weeks and two days ago?" I reply.
"Yes. Which means that if you operate as I do - on a month to month basis - by then end of the month you have no money left."
"Yes, but you'll still get paid every month, just not your overtime - for one pay."
"But in the case where you commit your monthly income to substantial mortgage repayments on a studio apartment in the West End..."
"Oh!" I say. "So you're broke for a month!"
"Two months actually," the PFY sniffs.
"And because you're broke you can't even afford the time to put a tape in the backup drive?"
"No..." the PFY says.
"You can afford to put tapes in but you don't for... personal reasons?"
"THERE'S A TAPE IN THE BLOODY DRIVE!" the PFY snaps.
"So why haven't we taken a full backup properly?"
"Because the backups are timed to occur at 1:00am on Monday morning."
"Still not with you..."
"Well at 12am, I've been doing some database maintenance which takes about 90 minutes to complete."
"So why not run the maintenance after the backups?"
"I can't, because the accounts payable payments are processed and transferred to the bank at 3am."
"And you wouldn't want to impact my chances of getting paid," I say.
"No. So I run it at 12 and it takes till 1:30."
"Yet still the backups don't start at 1?"
"No, because I have to change the system clock to be 3:01am before I do my maintenance."
>CUE RAT AROMA!<
"What exactly does your database maintenance entail?" I ask, fearing the worst.
"I select all the accounts payable transactions in the database which ran three weeks and three days ago, order them by payee and drop anyone who isn't me."
"And run the payment with the timestamp which would have occurred had it been a legitimate payment going to the bank."
"Yes. And then when that's done I set the clock back to the proper time, which is generally 2:30 or so and the real run goes through."
"Wouldn't that mean you had lots of duplicate payments for the same day."
"It would - if I didn't delete the row when I get into work - from there and the audit table."
"A plan so complicated I need a memory upgrade and a maths coprocessor to understand it. So why did you delete the tables?"
"I was looking through the financials package manual..."
"A manual?! You HEATHEN!"
"...and happened to notice that the audit table has it's own audit table which is available to the auditors..."
"AND THE AUDITORS ARE COMING THIS WEEK!" I gasp, finally realising.
"Indeed," the PFY sighs, confession over with.
"You realise they take a hardcopy of the audit table's audit table every night?"
"Not at all. They made it mandatory last year when I managed to replay a desktop order enough times to build my own Beowulf cluster - though they couldn't prove anything."
"So what do I do?"
"You mean, what do you do now that you've push the company into a possible crisis of investor confidence by manipulating our financial systems and ruining any possibility of a quick backup?"
"Start a small fire with the paper audit logs - they're kept in the head beancounter's bottom drawer, then buzz every second incremental tape with the bulk eraser so they can't roll forward."
"But won't they still be peed off that we didn't do backups?"
"Not when we blame it on the SAN again - because of >flip< >flip< fibrechannel aspherical monochromousity."
"Ah, F.A.M!" the PFY says. "And I'm guessing you'll be faking web pages for Google to find?"
"Already done it. Although that's my second cluster down the toilet.”
"That's what I like about you," the PFY says. "You're a people person!"