T'was the night before Christmas, and an industrial control system needed an upgrade
No documentation. Crusty code. Christmas Eve. Will our reader make it home?
On-Call Welcome again to On-Call, our Friday column in which readers tell their tales of being asked to get stuff done under awkward circumstances.
This week, reader “WG” shared a tale from 2010, when a client asked for some very major changes to be made to an industrial system that had been left largely untouched for 25 years.
WG and his colleagues tried to dissuade the client, because documentation of the system had somehow managed to disappear over the years. The folks that built the system were retired and scattered to the winds.
“Everyone in the industry advised us not to touch a thing and to walk away from the job,” WG told us.
But then, just a couple of weeks before Christmas came the request to do the job, before Christmas of course.
“A less than really sharp manager agreed,” which meant that in the days before December 23rd WG “was going flat out trying to get my head round a program written by a novice programmer for an ancient logic controller in a horribly arcane language.”
“Then on Christmas Eve myself and a colleague went on site to do the installation. By the way, did I mention that we couldn't do any preliminary tests because they wouldn't shut down the system until the night before?”
“Initially we couldn't get the laptop to talk to the old controller. It took us several hours to work out that not only was the connecting cable non-standard wiring, but so was the communications. Eventually we got the software on, and started making cautious tests - we hadn't wired the new valve in at this point just in case there were problems.”
And of course there were problems.
“First, the control sequence was not the same as what we'd been told, so that involved rewriting the program on-site to take this into account.”
“Next, the original programmer liked to do 'clever' tricks with his programs. These always come back to bite you years later - and indeed we were seriously puzzled as to why variables were apparently changing themselves and screwing up our corrections.”
WG and his colleague “realised the guy had discovered a way of bulk updating large blocks of data, which he'd proceeded to use enthusiastically, blissfully unaware that this also corrupted variables in the 'gaps' that he wasn't using but we were trying to!”
“After this was sorted we connected up the new valve, which had been pre-wired for us, only to discover that the 'telltale' that was supposed to send back a message to tell us which way the valve was positioned wasn't changing, so we went to examine the valve itself.”
Inspecting the valve mean climbing narrow metal stairs and gantries.
“My colleague got there first,” WG recalls, “and came out with a burst of the most interesting invective. You see, this valve was driven by compressed air, and our electronics just supplied a trigger. This normally works fine, but you do actually have to pipe in an air supply! We then had our only short break while the factory maintenance crew frantically rigged up a temporary air line.”
Before long the valve was switching over correctly. But WG and his mate wanted to double-check to make sure all was well and when they did, found that the main plumbing had been done in reverse.
“Our first thought was to just change the software so the 'off' state of the valve was reversed, only we were quickly informed that when unpowered it had to be in the open position.”
The maintenance crew climbed the gantries again and, amid “much banging and swearing” went over everything in as much detail as possible.”
“We weren't happy that the valve common was shared with the signal common, but weren't going to try to re-run over 70 ft of cable woven into the factory structure, so put filters on the PLC inputs and hoped.”
Job done! Or so WG thought until control system's monitor died. WG thinks all the switching things off and on did for it, and fair enough after 25 years' service. One trip down into the offices to find an unused monitor later, and WG thought it was finally time to head home for some egg nogg.
But first, a final test to make sure the night's work hadn't caused a problem elsewhere in the factory.
A main divert valve stuck but that had nothing to do with WG's work. But he had to hang around to make sure all was well.
“We started at 8:30AM and worked through until 10:00PM without stopping, except the one time for a cup of tea and a snack (supplied by considerate shop-floor staff).
WG made it home for Christmas, albeit without knowing if his work had done the trick because the client wouldn't run the whole system until the new year. His clients, however, declared themselves happy. Which was enough at the time!
Have you been called out to work over the festive season? We'll be running On-Call stories every day next week, so if you've a Christmas cracker to share, send me a Christmas card. ®
Sponsored: Becoming a Pragmatic Security Leader