Forget your RTO*: Real world Disaster Recovery needs garbage bags and bubble wrap

On-Call If it's Friday morning it must once again be time for On-Call, El Reg's end-of-week meander down memory lane to explore readers' reminiscences of jobs that went bump in the night.

This week, reader “Olaf” shares his experience in Thailand's 2011 floods. Yup, those floods that left Western Digital's hard disk factory under water.

Olaf worked for a Thai university back then and that institution was located on a flood plain.

When the nearby dikes broke, the campus flooded. Thankfully, the computers in Olaf's faculty were on the second floor of a building where they were high and dry, literally and metaphorically because the power was out.

The academic community is tight-knit, so another university offered to host the machines. If Olaf and pals could get them out.

“Since the onset of the flood, the army had been asked to help and secure the place, they also provided dinghies to patrol the area. Through social networks, I managed to gather few students, we met at the main entrance and were transported to our department.”

The first floor was still under about 20cm of water and befouled with trashed furniture and collapsed bookshelves. The second floor was spotless, but basked in eerie silence.”

Olaf says he and his students un-racked kit, removed disks from servers, wrapped everything in garbage bags to offer rudimentary waterproofing and then added a layer of bubble wrap for extra protection from knocks.

Over two days they prepared sixty packages downstairs, wading through the dirty water to the dinghies. Back and forth they went, out to the university's main gate where a truck waited to take the kit away.

Life on a Thai beach

The university that offered to host Olaf's rig did so because it had an empty building. Which of course had no racks, so Olaf went shopping and found some metal shelves. Then he had to carry the kit up three flights of stairs by himself, because there was nobody to help in the empty building.

Nor was there any cabling on site. So Olaf again went shopping for kit with which to run a wire in from the university's data centre and then build a LAN.

Which is where his troubles really began, because the university that gave him a temporary home was at a beach resort.

The only bulk Ethernet cable on offer “was some unknown import brand and the cable was about twice as thin as normal Ethernet.”

“On a short distance, it would sort off work, but on the close to hundred meters that span the two buildings, it would never make it across; luckily, I had run three cables in parallel, one good and two piece of worthless copper, I had plan to install some redundancy and link aggregation, I ended up with only one working link.”

It took Olaf 20 days to ready this site for students. His final hurdle was a recalcitrant internet service provider that only flicked the switch weeks later and even then required stern phone calls to the C-suite.

Not long afterwards, he got to do it all again because by early 2012 the university's campus had dried out and Olaf scored the job of re-building the network, re-cabling the offices and making sure that everything was above the high water mark.

“But that,” he told us, “is another story.”

Speaking of other stories, if you have a tale to rival Olaf's write to me and share it. Who knows? You might end up in a future edition of On-Call. ®

Bootnote

Not sure what RTO is?

@RBaindourov Recovery Time Objective. Like if you’ve got pub flu and you hope you’ll be better by teatime pic.twitter.com/Q6oS3IKLlF

— The Register (@TheRegister) July 22, 2016

Recovery Time Objective