Attempt to clean up tech area has shocking effect on kit

On Call Hello, Friday, El Reg’s old friend. We’ve come to talk with you again… because the vision that has softly crept in must be the latest instalment of On-Call.

This week, our reader’s tale of tech support conundrums solved is a real shocker – so without further ado, let's meet “Gerald”.

He takes us back a couple of decades to when he was working as a test engineer at a very large IT firm in the US.

He said that, at his location, there was a lot of sub-assembly for other plants, “so lots of backplane circuit boards, drive arrays and so on got shipped from this location”.

Any boards that failed testing weren’t shipped out, and the firm employed an army of technicians to work on them, often fixing a load by the afternoon.

“Then they would rack the boards up in these little toast rack arrangements on their top shelf,” Gerald said, pointing out that they couldn’t use bags because the whole area was ESD (electrostatic discharge) safe.

The boards then weathered the night in their temporary accommodation and were formally retested the next day, in the hope they'd then be on their way.

But Gerald and his team noticed that something slightly odd kept happening. “Quite a few of the boards that the techs fixed would fail retest,” he said.

“The techs would swear up and down that they'd fixed and bench-tested them, and in fact, the new failures were usually different than the previous ones.”

A closer inspection revealed that a lot of the failures were “random damage to MOSFET* parts and other ESD-sensitive components", Gerald said – which pointed to a static problem.

Knowing this didn’t immediately help the team figure out what was going on, though, because the entire area was ESD-controlled.

“The tech area on the factory floor was divided from the central aisle by cubicle walls. The whole area was ESD-controlled, so even to walk down the central aisle you needed heel straps and a smock,” Gerald said.

“It wasn't until I happened to work late one night that we discovered the problem.”

The problem, it turned out, was the cleaning crew. Or, more accurately, their kit.

“The evening cleaning crew came in about 7pm, and I watched as a properly smocked and heel-strapped crew member pushed their cleaning cart down the central aisle while feather-dusting the top of the cubicle wall... and all along the shiny gold connectors on the rear of all the recently fixed boards,” said Gerald of his Eureka moment.

“Some ESD re-education and a change in cleaning process soon cleared up most of the issues.”

And so, we have a happy end to this electrifying story. If you’ve had any shocking revelations while trying to figure out your tech support woes, let El Reg know by emailing us here. ®

* Metal-oxide-semiconductor field-effect transistor