Surviving Hurricane Katrina: A sysadmin's epic DR (as in Didn't Realise) odyssey
25 days of refugee sex and guns in odd places
Thursday, 15 September: Return to New Orleans
Henry's brother-in-law is in the National Guard. He claims he can get us to the office so we can recover our computers. Head out to our meet-up point in Baton Rouge.
Friday, 16 September
Offices on the 11th floor. No air conditioning. The temperature and humidity are in a race to 100. Five hours of up-and-down stairs. My truck is overloaded with computers (heavy 5U Compaq machines that should have been replaced years ago. I know I'm going to end up transferring their data to the new servers and then throw these out). Had to replace the suspension a few months later.
Saturday, 17 September
03:00 – Arrive at data centre with computers
17:00 – Full recovery complete. Only 20 days of downtime.
Sunday, 18 September
Final migration of DNS to our new data centre.
Here are the lessons I learned:
- DR stands for 'Didn't Realise'. 'Shit, I Didn't Realise we'd need that'. I've heard this phrase on every DR exercise I've been a part of since then.
- Make sure you have a plan to communicate with people. During a serious regional disaster you will not be able to call anyone with a phone in the affected area code.
- If you ever find yourself a refugee and have no idea if all of your worldly possessions have been destroyed, have lots of sex. Seriously, nothing beats refugee sex. I can't recommend it enough.
- Always evacuate with your booze. When I eventually returned to my apartment I discovered that three bottles of wine had gone bad in the intense heat. FEMA will not compensate you for this.
Besides the wine, I suffered no other damage.
* * *
Don't be intimidated by Jim's epic, dear readers: if you've more modest stories about what happened when the boss called at an odd time, share them by sending me a mail!
And did we say thanks, Jim, for the epic? Seriously, what kind of awesome community do we have when a reader volunteers something like this. Give the man a round of applause, nice comments and upvotes! ®