Security guard cost bank millions by hitting emergency Off button

Who, me? Special Welcome to a special edition of “Who, me?” The Register’s opportunity for readers to get their worst mistakes off their chests. We’re usually here on Mondays, but with the United States Independence Day making for slow news days, we decided to keep The Register’s servers red-lining by running an extra column.

When we sneak in extras to Who, me? or On-Call, they’re usually omnibuses that let us empty the mailbag of similarly-themed submissions that are worth a run, but don’t quite stand alone.

And boy do a few of you have stories about pressing emergency power off switches under the worst possible circumstances.

Consider the tale from “Pete” who told us he once worked on an upgrade at a large bank. This job had an absolute drop dead 2:00 AM deadline so the bank could service clients in another time zone. But at about 1:00AM a server’s power supply blew, making a splendid noise and filling the data centre with the acrid smell of burned computer.

At which point a security guard emerged to figure out what was going on.

“Suddenly, with substantial purpose, he started to stride towards the wall opposite the racks, when we realised with equal measures of horror and terror he was heading for the emergency power cut off, which quick as a flash, he lifted the see-through Perspex cover and hit the large red button while we screamed ‘Noooooooo!’ at the top of our voices.”

Cue frantic phone calls to get anyone capable of sorting things out into the office ASAP.

“By 5:00 AM we'd managed to get maybe 30 per cent of operational services back and it took till the following weekend to get everything working again,” Pete told us. “The contractors all loved it as some of us did over 100 hours that week and a large part of it on 'out of hours' rates.”

“We estimated that the total cost above and below the line for the cockup was probably upwards of a couple of million sterling, not including any trading exposure that couldn't get closed out,” Pete said, adding that he and his mates wondered if they had just witnessed the most expensive power supply mess in history.

“The security guard and his boss both got fired of course,” he added. “The guard for not talking to us before taking action and his boss for not making sure he was properly trained.”

The pub problem

Another big button tale came from “Kim” who told us about the time he did acceptance testing for a new mainframe. The test was simple: if the rig ran for a certain amount of time, with a real workload, the kit would be considered commissioned and the vendor for which Ken worked would be paid.

The test went well. So well that the Ken and his client’s team planned a long liquid lunch in a nearby pub.

With ten minutes before the acceptance threshold, and the pub date, Kim decided to call it.

“I went to the small comms rack and picked up the handset alongside a good old BT modem, he told Who, me? Next, he “called the salesman to let him know he could unlock his wallet.”

And then the room “went very quiet. Very, very quiet. All computer equipment - CPUs, disk, tape units, line printer - were dead.”

“I looked at the client. He looked at me, then pointed over my shoulder to the emergency power off button on the wall, at shoulder height, right next to where I'd been standing when talking on the phone.”

And the client “looked at his watch, added ten minutes, signed the paperwork and came along to the pub.”

“I bought him a pint.” Kim said.

More than one, we hope!

Cutting the cable

Lastly, meet “Pat” who told us that he once worked for an early cable TV service, on a team that “was putting forward the idea that rather than have Ceefax etc piggyback on a few lines of the TV picture we could use a whole cable channel to fill the screen with data and give the user instant page loads and even software downloads from their TV.”

The team had a demo rig built and working and, come the day on which they were to present it to the Suits, “we were shown into a backroom to assemble it, filled to the ceiling with tape copying machines all beavering away making copies of TV programmes for reasons I never understood.”

“The only space left was a small workbench with a big red button clamped to the edge pointing outwards labelled ‘Emergency Shutoff’.”

Pat and his team worked very carefully to get their rig working. All that remained was the demo, so Pat relaxed.

And instantly “felt something round depress in the small of my back.”

“Sure enough the power in the room went off and all those tape machines ground to a halt mid-programme,” Pat said. He and his team offered to help but were met with so much chaos they wondered if they’d missed a fire alarm.

If only: not even the alarms were working because Pat’s back had turned off the whole building. The whole building pumping out Cable TV to millions.

“Eventually some rather bad tempered people restored the power,” Pat said adding that “my boss never blamed me and pointed out a big red button on the side on a desk with no guard over it was asking for trouble.”

And that fab-sounding interactive service? “Our project never did get the green light,” Pat lamented.

What have you turned off that really, really should have stayed on? Feel free to write to Who, me? and you could end up in a Monday column or a slow-day space-filler. ®