A boss pinching pennies may have cost his firm many, many pounds

Who, Me? Welcome once more to Who, Me?, where El Reg readers share their IT catastrophes. And it doesn't get much more catastrophic than this week's story from "Marty".

At the time of the incident, Marty was working in the trenches at a financial institution.

"When I was first employed, the rack-mount servers for our division were piled high on desks in the only office that had air-conditioning," he said. "This kit was composed of grotesquely rare early revision silicon from most of the network and CPU manufacturers."

The good news: Marty's job was to move them to a purpose-built server room in new premises. The bad news: Marty had a penny-pinching boss.

"Despite the servers having redundant power supplies and the racks having dual power buses, the powers that be wouldn't pay for either the rails to mount the servers, or the power cables to connect them to both buses."

The end result was that the servers were piled in the racks – "and sounded like hovercraft" as they were running off just one, not two, PSUs.

Without slide rails, Marty said that every time anyone needed to changed components in one of them, "it required several people to power-lift the topmost equipment while you slid out the server in question".

Worse for Marty and his fellow techies was that the work being done involved trying the same algorithm on different types and capacities of memory. "Changes quickly became frequent and exhausting," he recalled, wearily.

Then, in preparation for a visit from an engineer about an issue the team had been having, Marty was asked to set up the kit to a precise configuration.

"I'd roped in a colleague to help extract the problematic kit, but in the process of lifting the servers off it, there was a bang and the term 'deafening silence' became all too tangible," Marty told us.

"One of the machines we were trying to lift had a dodgy power supply, which didn't take kindly to being manhandled."

Normally, of course, the redundant PSU would have kept the machine running despite the fault – but Marty's boss hadn't funded that second power cable.

"Unfortunately the PSU blew in quite a spectacular fashion and not only did it kill the machine, it also killed the breaker for the power bus and took out every other machine on that circuit," Marty said.

Not to worry! There's a redundant power bus, surely? "Well, that bus was OK, but it wasn't powering anything because TPTB wouldn't pay for the cables."

Within minutes, Marty said, someone had come running into the deathly quiet server room to enquire whether he'd accidentally turned off power to the switch instead of the server.

"Err... yes and no," was Marty's answer. "The switch was off, but so was every server in this division of the stock exchange..."

Have you ever been hamstrung by your boss's tight purse strings? When did you last power down a major money-making business? Tell Who, Me? and you might see your tale on The Register next week. ®