Surviving Hurricane Katrina: A sysadmin's epic DR (as in Didn't Realise) odyssey

Tuesday, 30 August

I bought a new cellphone since mine was not accepting inbound calls. I had the 800 number shifted over to the CEO's cellphone at his request.

I started looking for a mail provider with webmail so our people could start working again.

Numerous phone calls with the CEO – still no action on shifting data centre operations, and I updated the company website homepage with non-504 contact numbers. We were struggling to locate all the employees.

Still no word from Henry.

Wednesday, 31 August

At 07:00 I get the word to come to Texas and set-up a new data centre. At 08:30 I leave Memphis. Then the fun begins.

09:30 – I get a phone call from our office manager. She's in Baton Rouge but can't get a flight to Texas for a couple of days. The CEO is telling her to sit tight in Baton Rouge since they 'might open an office there' and she'd need to run it. She thinks they're going to fire her. She's got friends near our new headquarters she can stay with. I decide to pick her up so the boss can fire her to her face.

14:00 – Dead battery while gassing up. Cheerfully they were full service. Sadly they didn't have a battery for my truck. Ninety minute wait for a new one to be delivered.

18:00 – Arrive at airport. Pick up office manager. Head to Texas.

Thursday, 1 September: Dell and its kind customers to the rescue!

10:00 – Arrive at the new office. CEO is pissed that I picked up the office manager.

11:00 – Get a reservation at the hotel across the street. Just for kicks I tell them I'll need the room for a month. Turns out I was an optimist.

12:30 – New mail host is up and running. Tons of pressure is off once people are able to send and receive email.

During BCP discussion the topic of banking comes up. CEO says we're fine since all cheques are sent to our bank. I point out that our bank has no branches outside of the New Orleans area. Suggest we get new bank.

I spent the remainder of the day talking with data centres, Dell, and CompUSA.

I was floored by the service we got that day: Dell offered to get us servers within three hours – it had customers that were willing to let us take servers they had ordered and wait for replacements.

CompUSA (RIP) extended us credit on the spot and I was able to pick up 10 laptops that afternoon.

We had a rack in a data centre that day. They let us in with a three month contract with the right to cancel whenever we wanted. It wasn't until a couple years later that I realised what a huge favour they had done us, when I was working for a different company it took us two months to nail down the details and they were total pricks about the terms. Even so, I still recommend those guys today.

I worked around the clock setting up and shipping out new laptops, racing and stacking servers and installing Oses, then moving data from the remote location to the new data centre.