BOFH: The Great Backup BACKDOWN
'We have to be sure the system works!'
"So I see that backup check went well?" the Boss asks, trying to drag me into a conversation that will have nothing to do with me, but sounds technical enough that I would be an idiot and say...
"What backup check?" the PFY pre-empts me.
"The backup check," says the Boss. "The Financial Director wanted to be sure that we would be able to recover our data in the event of a system failure, so he deleted some files and then asked for them to be recovered."
"Was that what I was doing yesterday," the PFY asks, "running around recovering files which were deleted on purpose just to test the system?"
"Well we have to be sure the system works," The Boss says.
"So what if it hadn't worked?" the PFY asks.
"In that case we would have recovered them from the USB stick where the Financial Director had saved them."
"So you had him running around recovering files which were deleted on purpose and were in fact just on a USB stick and could have been recovered in a couple of minutes?" I ask.
"WE HAVE TO BE SURE THE SYSTEM WORKS!!!" the Boss says. "And it does!"
"You realise that a far safer way of performing that test would have been to have CREATED SOME NEW FILES then deleted them a day or two later and seen if they were recoverable?" the PFY asks.
"I don't see how tha..."
"Because if you're REALLY testing the system you'd be making sure that files are continuously being backed up. I mean I could have probably recovered those Policy Document files and the Budget projection files from six weeks ago and you would never have known because they haven't changed for months..."
"Yes, but we needed to see if the system works. Which it did."
"It APPEARED to work," I say. "If you'd talked to us about it we could have told you about some files that would have actually PROVED that it worked. Files that had changed in the last 24 hours and that should have been recoverable."
"But we couldn't talk to you about it. If there'd been some problem with your backups then you might have realised and fixed them before the test."
"Ah, so the point of the exercise was to catch us out and not check the recoverability of files?"
"But surely if you wanted to prove that files could be recovered you could have just come here and asked us to get back a file from yesterday?"
"Uuuuh, no, we needed to test the system works."
"Let me make a wild guess. One of the beancounters was on some course or something in the past few weeks where they heard some horror story about a company that hadn't been taking proper backups and when something went wrong they couldn't recover ANYTHING!?"
"I... don't know the specifics."
"And part of what they recommended was that you run regular backup checks to make sure that your data is safe by deleting files and seeing if your IT people could recover them?"
"I guess so. But it's really important to know that these systems are working! Why else would we have them? The company RELIES on this data being recoverable. It's a huge business risk."
"Yes, I take your point," I say. "It is a good thing to know and I suppose in retrospect it's good that we know that the systems we have in place are working."
"I couldn't agree more," nods the Boss.
>>10 minutes later<<
"You went a little easy on him didn't you?" the PFY asks.
"Yeah well, you know they're going to delete a bunch of new files shortly just to make sure that we're not recovering stuff from 6 weeks ago..."
"Ah yes, of course.." the PFY says
>>Two days later<<
"I was wondering if you can recover some files for us?" the Boss asks, entering Mission Control with the Financial Director following closely behind.
"Which files?" the PFY asks.
"These ones," the Boss says, putting down a list of folders and files.
>>10 minutes later<<
"Yeah, I can't recover those. Probably the nearest I can get is an archive from about six months ago." the PFY says.
"Yeah, well after our chat the other day, I realised that if we have all this data from six weeks ago, there's no way to tell if the backup system is working, so I erased all the old backup data. Only problem was the erase took two days to complete and in the process the backup index files were locked so the backup system wouldn't write files."
"Well we had to be sure the system worked," the PFY says. "So we started with a blank slate."
"Did you take copies before you deleted all the backups?"
"Well no. I mean the indexes were locked."
"Why the hell didn't you tell anyone you were doing it - so we'd be careful?"
"But how would we know if the Beancounter system of being careful about deleting files was working?" the PFY asks.
"We'll talk about this some more," the Boss says, "mean time we'll be recovering the data from USB drive - which I hope you'll be backing up tonight?"
"By USB drive you mean the ones that were in the Financial Director's filing cabinet?" I ask
"Ah... yes. Why?"
"Oh, only I broke into his office last night and chucked all those usb sticks into the rubbish bin. I mean the office is alarmed so I just assumed that security would come in and discover them before the cleaner tipped them out?"
"We don't set the alarm at night," the Financial Director blurts. "It keeps tripping out because of the air-conditioning."
"Surely it's important to know that they alarm system is working? Why else would we have one? Surely the company RELIES on the intruder alarms? It's a huge business risk!"
For a while there things are looking grim, but then a ray of hope...
"Actually I took a backup as well," the Boss says smugly, "onto a portable hard drive."
"A portable hard drive in the top drawer of your office desk?" the PFY asks.
"Ah... yes. Why?"
"Only I slipped into your office this morning and replaced your top desk drawer with a metal one with a small incendiary device. It won't set the office on fire but it will burn everything in the drawer."
>JANGLE!< >JANGLE!< "WHY THE HELL WOULD YOU DO THAT!?!" the Boss shouts.
"To test the fire system!" the PFY gasps. "Surely the company relies on the fire systems working? Surely it's a huge business risk??!"
"Why the F*CK didn't you run this past someone?!"
"I did. I had a chat to security about it and they thought it'd be a good test of our evacuation procedures."
"BUT THERE WAS CRUCIAL INFORMATION IN THAT DRAWER!"
"Surely we've got a policy about storing crucial information in desk drawers?" I ask. "It could be a huge business risk..."
Sponsored: Becoming a Pragmatic Security Leader