Feeds

How one bad algorithm cost traders $440m

A look at the worst software testing day ever

Providing a secure and efficient Helpdesk

Knight Capital, a firm that specialises in executing trades for retail brokers, took $440m in cash losses Wednesday due to a faulty test of new trading software. This morning reports were calling it a trading “glitch", which isn’t nearly as accurate as the term I’d use: “f**king disaster".

The broad outline of the story is here and more colourful, bloody details are here.

Briefly, here’s what happened: Knight Capital’s worst day in IT started Wednesday morning with a test run of its new trading software. An old pal of mine who’s following the story closely (and is also deep in both IT and trading) told me that the company set up the software to work with only a few stocks. They also set the buy/sell points well outside where the markets were currently trading to ensure that nothing would actually execute.

But somehow – and this will probably the be the subject of several lawsuits, books, and maybe even a Broadway musical – the software didn’t behave as expected. It went out and did what it was designed to do: execute lots and lots of trades very, very quickly.

Unfortunately, the trading algorithm the program was using was a bit eccentric as well. On every stock exchange, there is a "bid" and an "ask" price. The bid price is what you’d like to pay the holder of the stock if you want to buy their shares. The ask price is what they’ll pay to buy those same shares from you. There’s always a spread between the two prices, with the "ask" being a few cents or more above the "bid". If the stock is thinly traded, then the spread between the ask and the bid is higher than what you’d see for, say, IBM.

Knight Capital’s software went out and bought at the "market", meaning it paid ask price and then sold at the bid price – instantly. Over and over and over again. One of the stocks the program was trading, electric utility Exelon, had a bid/ask spread of 15 cents. Knight Capital was trading blocks of Exelon common stock at a rate as high as 40 trades per second – and taking a 15 cent per share loss on each round-trip transaction. As one observer put it: "Do that 40 times a second, 2,400 times a minute, and you now have a system that’s very efficient at burning money".

As the program continued its ill-fated test run, Knight’s fast buys and sells moved prices up and attracted more action from other trading programs. This only increased the amount of losses resulting from their trades to the point where, at the end of the debacle 45 minutes later, Knight Capital had lost $440m and was teetering on the brink of insolvency.

They may get at least a partial reprieve. The NYSE will reverse trades in six stocks during the time period when the prices were at least 30 per cent outside the normal trading range for the stocks. This will significantly defray much of Knight Capital’s losses for the day, but we don’t know if it’s enough to allow the firm to survive the blow.

We also don’t yet know exactly what happened. I find it hard to believe that the software was so faulty that it could just go into berserker mode and start wildly trading. Seems to me that that’s the type of issue that ISV regression testing would find and correct right away. On the other hand, I don’t think that the Knight Capital IT guys would just fire the program up for testing and not put in solid parameters to ensure that it doesn’t play with real money.

I think we’ll find that the culprit was a combination of ISV software bugs, bad documentation, and human error from Knight Capital. In short, plenty of blame to go around. But apportioning blame will have to wait until the forensics are complete. Then will come the lawsuits, settlements, high-level blue-ribbon commission hearings, and, finally, the insider tell-all books. Put me down for a Kindle version of the books, please. ®

Internet Security Threat Report 2014

More from The Register

next story
Microsoft on the Threshold of a new name for Windows next week
Rebranded OS reportedly set to be flung open by Redmond
'In... 15 feet... you will be HIT BY A TRAIN' Google patents the SPLAT-NAV
Alert system tips oblivious phone junkies to oncoming traffic
Apple: SO sorry for the iOS 8.0.1 UPDATE BUNGLE HORROR
Apple kills 'upgrade'. Hey, Microsoft. You sure you want to be like these guys?
SMASH the Bash bug! Red Hat, Apple scramble for patch batches
'Applying multiple security updates is extremely difficult'
'Google is NOT the gatekeeper to the web, as some claim'
Plus: 'Pretty sure iOS 8.0.2 will just turn the iPhone into a fax machine'
ARM gives Internet of Things a piece of its mind – the Cortex-M7
32-bit core packs some DSP for VIP IoT CPU LOL
'People have forgotten just how late the first iPhone arrived ...'
Plus: 'Google's IDEALISM is an injudicious justification for inappropriate biz practices'
prev story

Whitepapers

Providing a secure and efficient Helpdesk
A single remote control platform for user support is be key to providing an efficient helpdesk. Retain full control over the way in which screen and keystroke data is transmitted.
Intelligent flash storage arrays
Tegile Intelligent Storage Arrays with IntelliFlash helps IT boost storage utilization and effciency while delivering unmatched storage savings and performance.
Beginner's guide to SSL certificates
De-mystify the technology involved and give you the information you need to make the best decision when considering your online security options.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
Secure remote control for conventional and virtual desktops
Balancing user privacy and privileged access, in accordance with compliance frameworks and legislation. Evaluating any potential remote control choice.