Sysadmin wiped two servers, left the country to escape the shame
Source/target mixup proved that mirroring software worked perfectly
Who, me? Grab a very small cake and a bunch of candles, dear readers, for today we mark the 10th edition of “Who, me?”, The Register’s confessional for IT pros who broke things badly.
This week, meet “Graham” who “ended up as an authority on a fledging new product called SFT III from Novell. SFT stood for “system fault tolerance”.
“This revolutionary product allowed you to basically ‘mirror’ two file servers,” Graham told The Register. “It was clever stuff back then with a high speed 100mb FDDI link doing the mirroring and the 10Mb LAN doing business as usual.”
Graham was called upon to install said software at a British insurance company, which involved a 300km trip on Britain’s famously brilliant motorways with a pair of servers in the back of a company car.
Maybe that drive was why Graham made a mistake after the first part of the job: getting the servers set up and talking.
“Sadly the software didn't make identifying the location of each disk easy,” Graham told us. “And – ummm - I mirrored it the wrong way.”
Sysadmin held a rack of servers off the ground for 15 mins, crashed ISP when he put them downREAD MORE
“The net result was two empty but beautifully-mirrored servers.”
Graham tried to find someone to blame, but as he was the only one on the job that wouldn’t work.
His next instinct was to run, but as the site had a stack of Quarter Inch Cartridge backup tapes, he quickly learned that “incremental back-ups are the work of the devil.”
Happily, all was well in the end.
“The client, and my employer never knew how close we were to losing their massive database.”
“I have subsequently moved to Australia to try and put this behind me,” Graham concluded.
How have you dealt with the shame of a cockup? Click here to share your story with Who, me? and we might pop it up here on a future Monday. But not next Monday, when the column will take a break for the Easter Monday holiday. ®
Sponsored: Becoming a Pragmatic Security Leader