Rise of the Machines: How computers took over the stock market
Inside the super-secret world of algorithmic trading
Feature Trading used to be limited by how fast one human could shout at another and agree upon a price. Now it's limited by the speed of an electron through copper wire. This has caused, to put it mildly, some changes.
In April 2013 bombs went off at the White House and Barack Obama was injured, the Associated Press reported. The news sent markets lurching downward with the Global Dow index losing 150 points (thereby wiping out $136bn in market value) before the AP announced that its Twitter account had been hacked - no bombs had gone off, Obama was fine. But that didn't stop robotic trading algorithms from quickly selling off their holdings within seconds of the news.
This was not the first time automated electronic trading platforms – "high-frequency traders" – led downward lurches in the market. In 2010 the Dow Jones Industrial Average plummeted by over 1,000 points with the stocks of some major companies falling by over 90 per cent, a market disturbance so potent that it came to be known as "the flash crash". Various shares' values were wiped out. Billions were lost. And it all happened in less than an hour.
These are not isolated incidents: Nanex, a firm that monitors high-frequency trading within the stock markets, regularly tracks specific shares whose prices are being influenced by trading algorithms battling with each other. Days after the AP tweet fiasco, shares of anti-virus specialist Symantec fell 10 per cent in a matter of minutes after a trader put in a very large sell order with no sell price limit – a market signal that caused watching robo-traders to initiate their own sell orders, and drive the price down.
So just how much of the market is made up of these computer trading engines? Around 80 per cent of US stocks are traded this way, according to a 2012 analysis by Morgan Stanley beancounters, versus at least a third of UK stocks, according to a substantial 2012 report by the UK Department of Business Innovation and Skills' Foresight committee. This compares with an overall figure of 40 per cent for European markets, according to a 2010 study by Bank of England.
"These fractions have risen from single figures as recently as a few years ago. And they look set to continue to rise," the bank wrote (PDF).
The beginnings of HFT
High-frequency trading (HFT) has been around since the late '90s, when changes to trading regulations meant electronic communications networks (ECNs) could let traders reach into markets and start making bids. The first US ECN was INSTINET, that was closely followed by the ISLAND ECN in the late-'90s. Others proliferated.
"All these things started to talk to one another. Networks got faster, communications got better, data centres got better, and it went from there," says Scott Ignall, chief technology officer of financial trading firm Lightspeed.
Alongside the proliferation of ECNs, regulatory changes in the US altered how trading worked. What started HFT in a serious way was when the SEC introduced Regulation ATS (Alternative Trading System) in 1998, which allowed people other than stock exchanges to conduct electronic trading, and the Regulation NMS (National Market System) in 2005, which made it possible for multiple exchanges to distribute quotes for stocks. ATS made HFT possible, and NMS led to a technological arms race that reduced the number of firms in the market by dramatically increasing the investment needed to compete.
In response to the initial ATS regulation, ISLAND began offering a service in the late '90s whereby traders could stick their trading gear right next to the matching engine used by the NASDAQ market. "What that meant was they can read the data and place the trade in a sub-millisecond fashion," Ignall explains. "The ISLAND tech was fast enough to handle all the throughput, [HFT] snowballed from there."
These days all major exchanges offer such services, and have seen great demand for it: the London Stock Exchange recently expanded the slots available for robo-traders keen to get a direct feed to its market data, via its Exchange Hosting service – "the ultimate in low latency connectivity".
'Off to the races'
Other exchanges began offering similar services as well. The time it took to execute trades fell as firms sought to gain an advantage over each other. "We were off to the races – data flying everywhere, trade flying everywhere," Ignall says.
Things kept on growing until about 2006 or 2007, when NMS came in. "If you want to point into any specific point in the regulatory landscape that caused the huge explosion in HFT trading, it was Reg NMS," says HFT critic Joe Saluzzi, co-founder of the firm Themis Trading.
Reg NMS's introduction would make the business of making money harder for HFT firms by increasing the technical requirements of trading engines. But it also gave them an advantage in that they could start trying to exploit minute differences between the different prices at difference exchanges in the market.
"Some exchanges will process [data] faster than others," Eric Hunsader, the chief of market analysis firm Nanex, says. "When they do that, orders may appear on a network somewhere before it has even been processed somewhere in other exchanges. When that occurs HFT firms can adjust orders in other exchanges before trading takes place."
Since NMS's introduction, the industry has been dogged by problems: the flash crash, regulation, falls in the stock market, frequent blips in share and market prices due to HFT's capitalising on information faster than any trading engine ever before, and so on.
"We still have a lot of trading compared to what we had ten years ago, but it's about half, at least in the US equity markets, compared to what it was 3/4/5 years ago," Ignall says.
Though growth in trade volumes has tapered off, the amount of quotes – requests by HFTs to exchanges to get the price of a stock – has rocketed. This is because as more and more capable trading systems have come to dominate the market, they have been fighting tooth and nail to try and disrupt the flow of stock information to other rival algorithms, Hunsader says.
This means that the trading engines can exploit slight pricing discrepancies for a lucrative payoff. "It's gauging the reaction of the marketplace, it's the ability to gain information," Hunsader says.
Quoting more than ordering can sometimes give trading engines an advantage, and also provides them with the information they need to accurately make trades.
"We find that whenever there is higher message traffic, smart order routers will get behind," Hunsader says. "They'll route to places they won't get otherwise. It gives them [HFT traders] an advantage for discovering a smart order that's being split up on exchanges."
But to trade in this tangled web of markets requires a punishing level of technical sophistication and infrastructure that weeds out other firms. Just as in other businesses where technology is a profit centre (cloud computing infrastructure, semiconductor manufacturing), HFT companies are caught in a dilemma where they need to invest ever more to keep up with their competitors, and the cost for falling behind technically only magnifies over time.
Sponsored: DevOps and continuous delivery