How an over-zealous yank took down the trading floor of a US bank
A minute's malfunction? Mere millions
Who, Me? Good Monday morning, Reg readers, and welcome once more to Who, Me? – our regular trip down memory lane for those with something to get off their chest.
This week, we meet “Alan” who once took out an entire trading floor at an arm of a US bank.
Back in 1996, Alan had just graduated from a software engineering degree and had moved to a big city in search of his fortune.
He got a job in devops with a company whose software took price feeds and news feeds and integrated them into a single screen for traders and brokers.
“It used Sun IPX Workstations because PCs just couldn't handle the load in those days… Back then, PCs could just about handle a strenuous game of chess,” he said.
“SunOS handled the workload centrally, and each trader had a thin client on their desktop that just showed the X11 results. It was all relatively straightforward.”
One of his firm’s main customers was the bank in question, and Alan visited their local office on a regular basis to service the systems, upgrade software and generally keep an eye on anything else that cropped up.
It was on one of those trips that things went awry for our anti-hero.
“On this occasion, there appeared to be a problem with one of the price feeds,” Alan said. “I was dispatched to find out what was going on and fix it.”
Alan entered the frame room at the back of the trading floor, and took his seat at the cabinet holding the SunOS workstation.
“I pulled the keyboard tray towards me, and was rewarded with the happy news that the workstation had gone into kernel panic as it had lost
stdin,” said Alan – who at this point in his career didn’t even know what
“The screen then went black as the machine rebooted.”
Sysadmin misses out on paycheck after student test runs amokREAD MORE
The brief 30-second silence could only be described as the calm before the storm. As Alan puts it, it was soon shattered by “a roomful of screaming traders assaulted the frame room baying for my blood”.
Noting that the barrage of abuse “did nothing to help the fault-finding process”, Alan eventually he came up with an idea – although it was more by luck than judgment.
“I looked down and noticed that the machine was still in kernel panic and still rebooting,” he said.
“Then I noticed that the keyboard was unplugged. This must be related, surely.
"I had not lost my razor-sharp instincts," our resourceful man on the inside observed.
Because, after he had rectified the mess his rather hard yank on the keyboard had done, by plugging it back in – hey presto – no more kernel panic.
“We were all back on track in a couple of minutes,” he added.
It's perhaps a good job financial institutions didn't operate in an environment where milliseconds meant millions back in 1996...
Have you ever solved a problem of your own making? What about someone else's? Tell us your traumas and we might share them with your fellow Reg readers next week. ®
Sponsored: Becoming a Pragmatic Security Leader