If we can't fix this printer tonight, the bank's core app will stop working
Reader jolted from the matrimonial bed in the dark tells of happy ending
On-Call Thank Arioch it's Friday, because the working week is nearly over and we can ease you into it with another instalment of On-Call, our end-of-week trawl through readers' memories of awkward jobs.
This week, meet “Jonny G”, resident of Canada who “Once upon a very long time ago, worked for a large IT vendor in a role as an on-site support specialist at a large financial institution.”
Jonny G says the gig was all about making sure the institution had the support it needed for a “large and varied portfolio of hardware, software and services.” He had a pager, “but it didn't go off much; when it did, it was usually someone from the financial institution wanting me to expedite support from the vendor.”
“One bright day in the middle of the night, my pager went off right beside my head,” Jonny recalls. “Naturally, I slept on through. My then-wife, a lighter sleeper, didn't and gave me a shove. I checked my pager, and I made the phone call. Then she told me to get the hell out to another room for the resulting conversation.”
Jonny's not sure why she didn't also tell him not to pack his bags, but that's another story.
Strangely, the call was from his employer, not the client.
The reason for the call was a severity one incident regarding a printer, which was odd because there was no such incident code for printers.
Jonny managed to discern that it was something to do with a printer and an application log. And Jonny knew all about that application, which he told us was “a large, complex, homegrown affair that handled most of this institution's normal account transactions - deposits, withdrawals, transfers and the like.”
The application log was output to a large chain printer, which was broken. This was a problem for the application, because if it couldn't print its logs it stored them. So its buffer was filling and filling and filling, but had no way to empty itself.
“When it was full,” Jonny tells us, “the application would stop processing transactions.”
The good news: it was the middle of the night, so transaction volumes were low. The bad? Transactions were still trickling in and nobody could say when the buffer would be full.
Jonny quickly realised probably had just an hour or two to sort this out.
“Needless to say, I gently encouraged our support desk to get a printer repairman dispatched to the customer site as quickly as possible. They agreed, and I went back to sleep.”
And when the cock crowed the next morning and he check the situation, all was well.
“I found out that we got the printer fixed before the transaction flow got hung, everyone was happy and I kept my job at the vendor and my contract at the customer site,” Jonny said. “So this one gives you a rare happy ending.”
If you've a story of waking up in the middle of the night and then experiencing a happy ending [Keep it clean - Ed], write to me and you might feature in a future On-Call. ®
Sponsored: Becoming a Pragmatic Security Leader