Software engineer sobers up to deal with 2:00 AM trouble at mill
Boss who 'couldn't hear pager' coded problem that threatened steel mill shut-down
On-Call It's Christmas Eve Eve, which means we're assuming you don't particularly want to read industry news right now and so we're instead offering an extra instalment of On-Call, our regular reader-contributed tales of nasty jobs at nasty times in nasty places.
Today's yarn comes from reader “BT”, who told us that “More years ago than I care to remember I worked as a software engineer in a steel mill in Rotherham (when they still existed).”
BT's job included alternate weeks of on-call action, but his boss was the other person on the roster and claimed he couldn't hear his pager. So more often than not, BT was the on-call, even when he wasn't.
This “arrangement” culminated in a call to BT's home at 02:00AM, after the mill crew gave up after trying to call the boss for two hours.
BT was happy to help, but felt he was probably over the legal drink-drive blood alcohol limit.
“No problem”, said the folks at the mill. “We'll send transport.”
BT imagined a nice warm taxi but about 30 minutes later heard “a massive diesel engined thing stop outside the door.”
“The mill had sent a fully-loaded articulated lorry from the transport department to fetch me.”
BT didn't mind. But as the truck had to navigate to the end of the cul-de-sac on which he lived, and then reverse out beep-beep-beeping all the way, it made for some rather cranky neighbours.
But not as cranky as the folks at the mill, because the supervisory system had failed and the plant was inoperable.
“A stoppage of an hour was quite common given the combination of humans, machinery, computers and hot steel,” BT recalls. “The problem was that we were now into the third hour of stoppage and after four hours the mill would need to go into 'turn-down' mode which means the furnaces were turned-down, followed by a 10-14 hour restart procedure.”
That kind of outage costs lots of cash, to BT recalls “there was an element of hysteria in the call-out.”
BT says he arrived on site “still very much worse for wear” but tossed down a few swift coffees which “had an unfortunate effect on my bladder but little effect on my state of inebriation.”
Through the haze, our hero was able to determine that the problem lay in a big slab of FORTRAN written by – of course – the Boss. Who had left no comments other than copyright header and the single comment - “initialise X” next to a line that said X=0.
“The rest of the code was a mess of stuff like code that jumped out of for-loops into the middle of subroutine code before jumping back into the for-loop in a different place without actually calling the subroutine.”
BT kept at it, but soon started getting calls from people wondering if he'd hit the deadline to avoid the 'turn-down'.
That close attention made BT predictably irritable, culminating in one question too many receiving the answer “Constant interruptions aren't helping sort this so why don't you f*** **f and I'll call you when it's ready.”
That question, BT learned when he turned around, was asked by “a rather red-faced mill manager (i.e. God). He stood and stared for what felt like forever, turned around and left without saying a word. I rather sheepishly got back on with finding the problem.”
It was all the fault of BT's boss, who had written something “which meant that the system was trying to talk to the control system on the wrong LUN and was crashing out because there was no error handling. Testing, I'd heard of it but clearly the boss belonged to the 'it compiles, it works' school of software. With minutes to go I managed to patch in the correct LUN number, rebuild the system and it up and running.”
“ I then went in search of the mill manager with a grovelling apology - he just laughed and thanked me for coming in at such short-notice. A real gentleman.”
Sadly the mill wasn't willing to take BT home, so he ended up sleeping on the control room floor.
“I was about to head home the following morning for a proper recovery except that the boss turned up just as I was heading home. His excuse for not answering the call out? He thought it was my week on, a new excuse!”
Where have you slept in the call of duty? Let me know by sending seasonal salutations. ®
Sponsored: Becoming a Pragmatic Security Leader