'Twas the night before Y2K and a grinch stole the IT department's overtime payout
Plus: The Christmas kill switch that backfired
On-Call Not just the week but The Register's working year has drawn to an end, and that means it's time to wrap things up with the last of our slow-news-week festive editions of On-Call, the column we compile after kind readers send in their stories of support jobs that went south.
Today: tales of tech support over the holiday period.
Let's start with "Mike" and his story of working "the dreaded Christmas 1999/2000 period" for "a major part of what is now known as .gov.uk".
Mike and his team were told to have the UK government's Y2K situation sorted by Easter 1999 and Mike's bit of the problem was sorted by that time. However, lots of other limbs of Her Majesty's government were still hard at it much later in the year.
"So we were all told that we had to record all overtime and be available for all of December and January, our reward would be paid in February and would be huge," Mike told On-Call.
"So we put in the hours, including months away from home and many seven-day weeks." And then on December 20 the team went on-call 24/7, and were expected to stay that way until January 2, 2000, to avert the digital apocalypse.
The way Mike told it, his crew did their sums and knew what their overtime payments would amount to. He reckons all were due £6,000 to £8,000 (US$8,000-$10,500) – serious cash in 1999 or now!
But then came two ScroogeMails from a "Senior HR Muppet". "The first one told everyone that the whole reward was tax free and £50."
Not £5,000. £50.
"The second email told me to sack 50% of my helpdesk call takers but to emphasise that they would still get £50 tax free."
Makes you wonder why Y2K turned out to be such a nothing!
The Christmas Kill Switch that backfired
"Rick" wrote with the story of his time in the 1980s working for "a very small company doing industrial control systems for many different sectors including the water industry".
Things weren't going well on one job at a water treatment plant, where Rick's company wasn't being paid by the main contractor. Rick's boss was trying to sort things out, without much success.
While negotiations continued, Rick's company decided to build some protection into their system, in the form of a kill switch that would turn off the automation system and stop water flowing if the team wasn't around to continue its work.
A manual fix would sort things out quickly, so there was no danger to the public.
The job of writing the kill switch fell to Rick, who set it to "trigger at midnight on 24th December" and to "simply wipe the memory of all the machines including part of the self destruct code before the system crashed leaving everything safely shut down."
Rick tested it and "it worked every time leaving very little trace behind."
Rick's boss loved this program and headed off to another round of "hurry up and pay us" negotiations in a rather better mood, and perhaps a little leverage too.
But the negotiations failed and Rick said: "We were all made redundant on the last working day before Christmas 1987.
"I never installed the destruct code," he told us, "but the possible threat seems to have been enough for them to not trust the system. The main contractor employed someone else to finish the job and they rewrote all the software from scratch in case we had hidden something."
At least the main contractor copped some pain too: Rick told us that "having failed to plan ahead they had to rewire all the control panels to match the new software – it took them another two years to complete the project".
Good luck getting client sign-off on New Year's Even
"Dean" sent us a story about how "I always worked the late shift back in the 1990's and ran a network support team of six engineers for a large third party outsourcer."
Dean liked the time between Christmas and New Year, because "most customers have change freezes and hardly anyone is in their offices, so the call rate crashes and we would catch up with some outstanding back burner issue".
One year, this plan worked well: the team enjoyed a nice quiet holiday period and Dean let them go early.
He also let himself be the last person On-Call, which he felt was fair enough given he was team leader and lived closer to the office than anyone else on the team.
And then, at 5:55PM on New Year's Eve, a priority one call landed.
And Dean of course took it and ran with it hard, because he didn't want the customer's team to miss out on New Year's Eve.
By 7:30pm, Dean had figured out that the problem wasn't actually one his team could control or fix. So as soon as the customer acknowledged that, he'd be free to leave.
But despite lots of back and forth, the customer wouldn't sign off. Not at 8pm. Not at 9. Not at 10, either.
"Finally at 11:55pm they said 'OK, off you go'," so Dean did. Fast. But as it took more than five minutes to exit the data centre – damn that security – the clock ticked past midnight.
"The Ministry of Defence still owe me a New Year's Eve," Dean lamented to On-Call.
And that's it for On-Call in 2017. It's become a Reg regular that I hugely enjoy writing. It's humbling that readers keep sending in their tech support stories and that so many of you enjoy reading and commenting on them too. There's a few hours a week during which a new edition of On-Call's comments page generates more traffic than On-Call itself, and the column averages over 130 comments a week, plus hundreds more upvotes and downvotes.
Thanks so much for making that happen.
On-Call will return on Friday, January 12, and continue as long as you keep sending in stories. Even if I were hit by the proverbial bus tomorrow, On-Call is the probably the thing The Register's bosses would want to keep alive.
To you and yours, all the best for a wonderful holiday season, and a healthy and prosperous 2018. ®
Sponsored: Becoming a Pragmatic Security Leader