Knight Capital Lost $460 Million in 45 Minutes. Code It Had Stopped Using in 2003 Woke Up.

At 9:30 a.m. on 1 August 2012, the U.S. stock market opened and one of its largest trading firms began buying and selling shares it never meant to touch. Over roughly 45 minutes, Knight Capital’s order router sent millions of orders into the market. When it was over, Knight had lost more than $460 million.

No hacker was involved. No trader went rogue. The cause, according to the U.S. Securities and Exchange Commission, was a piece of software that had been switched off for nine years, a configuration flag that was given a new meaning, and one server out of eight that never received an update.

A firm at the center of the market

Knight was not a minor player. By the SEC’s account, its trading in 2011 and 2012 generally represented about ten percent of all trading in listed U.S. equities. One of its key systems was SMARS, a high-speed algorithmic router. SMARS took large “parent” orders from other parts of Knight’s platform and broke them into smaller “child” orders that it sent to trading venues.

In the summer of 2012, Knight prepared SMARS for a new program at the New York Stock Exchange, the Retail Liquidity Program, which was scheduled to start on 1 August. That meant writing new code and deploying it to production on a fixed date.

Old code, new meaning

The new feature was meant to replace a piece of code that nobody used anymore. It belonged to an old function called Power Peg, which Knight had stopped using in 2003. The code had never been removed. It was still present in SMARS and could still be called.

The new Retail Liquidity Program code also reused a flag that had once switched Power Peg on. Knight intended to delete the old code, so that setting the flag to “yes” would now activate the new feature instead.

There was a second, older problem hidden underneath. When Power Peg was in use, a function tracked how many shares of each parent order had been filled, and told the router to stop once the order was complete. In 2005, Knight moved that counting function to a different place in the code. It never retested Power Peg afterwards. If the dormant code ever woke up, it would no longer know when to stop.

Seven servers out of eight

Starting on 27 July, Knight rolled out the new code in stages across the eight SMARS servers. On one of them, the technician did not copy the new code. Nobody else reviewed the deployment, and Knight had no written procedure requiring that someone should. The eighth server kept the old Power Peg code and lacked the new feature.

On the morning of 1 August, the warning signs were already arriving. From about 8:01 a.m., an internal system sent automated e-mails referencing SMARS and an error described as “Power Peg disabled.” By the 9:30 a.m. open, 97 of these messages had gone to a group of Knight staff. They had not been designed as alerts, and nobody acted on them.

Forty-five minutes

When the market opened, orders carrying the repurposed flag arrived. The seven updated servers handled them correctly. The eighth server ran Power Peg. Because the share-counting logic had been moved years earlier, it kept firing child orders for each parent order without regard to how many shares had already been bought or sold.

The SEC’s figures are stark. For just 212 incoming parent orders, SMARS sent millions of child orders, producing more than four million executions in 154 stocks, totalling more than 397 million shares. Knight ended up with a net long position of about $3.5 billion in 80 stocks and a net short position of about $3.15 billion in 74 others. In dozens of stocks, Knight’s trades made up a large share of volume and moved prices by more than five percent.

The response made things worse. Knight’s monitoring tool reported positions after the trades had already happened and depended on people watching screens rather than raising automated alarms. Knight had no incident-response procedures for a case like this. While engineers searched for the cause in a live market, they uninstalled the new code from the seven servers where it had been deployed correctly. Those servers then also began routing orders through the old Power Peg logic.

The bill

Knight’s loss on the unwanted positions came to about $460 million. Within days, the firm needed a rescue: a group of investors put in $400 million in exchange for a large majority stake. Later that year Knight agreed to be acquired by the trading firm Getco, and the two combined in 2013 as KCG Holdings. In October 2013, the SEC found that Knight had violated the Market Access Rule, which requires brokers to have controls that prevent erroneous orders from reaching the market. Knight paid a $12 million penalty.

What engineers still take from it

It would be easy to blame the technician who missed one server. The SEC’s findings point somewhere more useful. Every link in the chain was a missing safeguard, not a single bad decision:

Dead code that stayed callable for nine years.
A flag reused with a new meaning while the old behavior still existed.
A deployment with no second review and no automated check that all servers matched.
Warnings that went to an inbox instead of to someone responsible for stopping the system.
No pre-set plan for halting trading when the system misbehaved.

None of these is exotic. Most production systems contain at least one of them today. Knight’s failure shows how quickly they compound when they meet at the same moment, at market speed.

The lasting lesson is simple to state and hard to practise: code you no longer use is not harmless just because you no longer call it. Delete it, or assume that one day something will.

Explore

Stay connected