Customer: We fancy changing a 25-year-old installation. C'mon, it's just one extra valve... Only wafer thin...

On Call Good Friday morning, and welcome to our weekly installment of On Call, where Reg readers share the tech support moments that made them want to rip their hair out in despair.

This time, we meet "Geoff" and travel with him back to December 2010, to what sounds like one of the most frustrating client visits of all time.

The customer in question had been toying with the idea of making some changes to an installation that had been done about 25 years before. Suffice to say, the team really didn't want to get involved.

"Apart from anything else, all drawings and notes had been put in their possession, as per the original contract, and guess what? They'd lost them," said Geoff.

"All that was left was a 5.25in floppy disk with the program. The original design team had also long since retired. Everyone in the industry advised us not to touch a thing and to walk away from the job."

But this customer was not about to listen to logic and came back to them saying they wanted to go ahead and were installing the necessary hardware – could Geoff's team come in, get it wired up, and make and test the software changes before Christmas?

"After all," the client said, "it was only one valve being added to the system..."

And so Geoff was soon running flat out trying to get his head around a "program written by a novice programmer, for an ancient logic controller in a horribly arcane language".

On Christmas Eve, he and a colleague went on site to do the installation – by which point there had been no preliminary tests because the client wouldn't shut the system down until the night before.

And then the fun began.

"Initially we couldn't get the laptop to talk to the old controller," said Geoff. "It took us several hours to work out that not only was the connecting cable non-standard wiring, but so were the communications."

Soon, they got the software on and started making "cautious" tests – but they had yet to wire in the new valve.

And the problems continued to roll in.

"First, the control sequence was not the same as what we'd been told, so we had to rewrite the program on-site to take this into account."

Then it turned out that the original program liked doing "clever tricks" – and these, unsurprisingly, came back to bite poor Geoff many years later.

"We were seriously puzzled as to why variables were apparently changing themselves and screwing up our corrections," he said.

"We eventually realised the guy had discovered a way of bulk updating large blocks of data. He'd used this enthusiastically, blissfully unaware that it also corrupted variables in the 'gaps' that he wasn't using – but we were trying to."

Once that was sorted, Geoff and his colleague connected up the new valve, which had been pre-wired.

However, they then found that the "telltale" that was supposed to send back a message to show which way the valve was positioned wasn't changing.

There was only one thing for it: examine the valve itself. But this was a long way up narrow metal stairs and gantries.

"My colleague got there first and came out with a burst of the most interesting invective," Geoff recalled. "You see, this valve was driven by compressed air, and our electronics just supplied a trigger."

Normally, Geoff said, this works fine – but only if you actually pipe in an air supply.

"We then had our only short break while the factory maintenance crew frantically rigged up a temporary air line."

When the valve was switching over correctly, Geoff and his buddy decided – based on the experience to date – that it was worth checking the pipework. And lo and behold, the main plumbing had been done in reverse.

"Our first thought was to just change the software so the 'off' state of the valve was reversed," Geoff said. "But we were quickly informed that when unpowered, it had to be in the open position."

So off the maintenance crew went again, with much banging and swearing, and the duo left on the floor went over everything else in as much detail as possible.

"We weren't happy that the valve common was shared with the signal common, but weren't going to try to re-run over 70ft of cable woven into the factory structure, so put filters on the PLC inputs and hoped," Geoff said.

At long last, the end was in sight – but, just as it was time to start wrapping things up, the display for the control system died, although as Geoff said, 25 years is a "damn good life for a monitor".

Because they had to deliver a working system, the duo "snatched a flat screen monitor from their offices and wedged/gaffer-taped it in place behind the original safety screen".

And so that was that. The pair packed up, and were ready to go but for the final test, when a main divert valve ("nothing to do with us," said Geoff) stuck.

"So we had to hang around and wait for them to clear it before we could complete the test," said Geoff. "And guess where it was."

All told, the pair started work at 8:30am and worked through until 10pm without stopping, bar the cuppa and snack provided by the sympathetic canteen staff.

Could you have kept your cool in the face of such a litany of failures? When was the last time a normal job turned into an epic battle? Tell On Call, and your fist-balling, scream-inducing tale could feature here next week. ®