If we can't find a working SCSI cable, the company will close tomorrow
It's 1AM, backup's failed and the boss says you're about to lose all insurance
On-Call Welcome again to On-Call, our Friday fumble through memories of jobs on which things didn't go as planned. Or sometimes went in ways it's not possible to plan.
This week, meet “Jean” who in the early noughties scored a gig as “a fairly new-to-the-game support engineer for a shifter of overpriced household furniture.”
Jean told us she “managed to blag my way in with almost zero actual IT service experience but a shed-load of related electronics background.”
When Jean scored the gig, the furniture shifter had “just completed a major project, replacing the standalone IBM server and green screen terminals in each of their stores with a set of consolidated rack servers running NT4 terminal services back at HQ.” The terminal packages ran on full fat PCs in each store. The company had also “migrated all their data to a huge Oracle 7 database running on SCO Unix.” Yup, SCO!
The move the new database server meant that backups were critical. Which meant that on Jean's first day on the job, things got a bit tense when an IBM tape drive went TITSUP (Total Inability To Support Usual Performance) in the middle of the night during a backup run.
Jean was on-call when the drive failed and by her own admission “extravagantly failed” to figure out what was going on with the backup.
The good news was that the furniture company had signed up for Gold level support from IBM, complete with a promise of visits within four hours of calls being logged. Said engineer staggered in on a frigid January night, verified the drive was dead and promised a new one would be provided ASAP.
But by now the company had been without backups for two days and Jean and the rest of the IT crew received a visit from the head bean-counter.
“He told us that if we don’t have a database backup by the end of the third night, the underwriters for the company insurance would pull our cover (including employer's liability) essentially putting the company out of business.”
No pressure for Jean and the team, then.
And certainly no pressure when the drive finally arrived in the afternoon and was verified as alive and well by IBM's operative.
Nor was there any more pressure when Jean, by now logged in remotely, watched the backup crash at 22:10.
By 22:15 she'd told the IT manager of the mess, resulting in much swearing. There was nothing for it but for Jean to head to the office, again on terrifyingly cold January night.
By 23:30 Jean had tried to run the backup several times, and they all failed.
On the stroke of midnight, the problem persisted. Jean admitted she “hadn't a scoobies whats’s wrong.” Nor was she confident she could figure it out, because it was cold, late, she'd worked an 18 hour day and that came after the abbreviated sleep the night before.
The final countdown
But Jean did have the presence of mind to get back to very basic testing and inspection. During that effort she realised “The IBM engineer bent a pin on the f*cking SCSI cable when he plugged it back in!”
Setting aside homicidal thoughts, Jean tried to find some pliers with which to put the pin back in shape. But that cupboard was locked and she did not have the key.
Next step: look for a replacement cable.
No luck there either.
“To make it even worse, its an Ultra-SCSI cable with more pins than a shark has teeth and my eyes are so tired that the tiny little f*cking pins are dancing gleefully in front of me like I’m tripping,” Jean recalls.
By 00:50 she decided the only thing for it was to go home and retrieve her own pliers.
At 02:15 Jean made it back, bent the pins back into shape and watched as – hurrah! - the backup performed flawlessly.
By 06:30 on the third day the backup finished, the IT manager thanked Jean for her stalwart effort and the company kept trading.
But nobody from management or accounting batted an eyelid or offered Jean a word of thanks. Things were worse for the IT manager, who was shown the door a week later.
As was Jean, about six weeks later and “quite acrimoniously.”
She was therefore glad when, years later, she heard that “the wholesale shafting of their IT staff had continued at such a pace to the point where they couldn’t hire permanent staff and could only get contractors at extortionate rates.”
Have you found yourself under pressure to match or exceed that faced by Jean? If so, click here to send me an email and you might find your own story in a future On-Call. ®
Sponsored: Becoming a Pragmatic Security Leader