Flash Boys

The spread between Chicago and New York is the tiny price difference between the same thing on two different exchanges. Being first to exploit that spread was enormously valuable, so Dan Spivey started building a direct line between the Chicago Mercantile Exchange data center in Chicago and the Nasdaq data center in Carteret, New Jersey.

The routes offered by telecom carriers like Verizon and AT&T were slow and inconsistent: 15-17 milliseconds, when theoretically it could be 12 milliseconds with a straight line of fiber.

In late 2008, Spivey started the build and kept it secret. The company was called Spread Networks but hid behind shell companies with dull names. They spent $300 million to build it and didn't tell anyone about it until March 2010, three months before it was finished. Along the line were signal amplifiers surrounded by maximum-security bunkers.

When they finally went to Wall Street to find customers, every firm was in disbelief. Spivey didn't even know why speed mattered so much to all these firms, but if they wanted to keep running their current strategies they'd have to be on the line or their competitors on the line would smoke them. Everyone wanted in and wanted everyone else excluded.

In one meeting, Spivey told a big Wall Street firm's boss the price of the line: $10.6 million plus costs if paid up front, around $20 million if paid in installments. The boss went away to think and came back with one question: "Can you double the price?"

By 2007, what people saw when they looked at the US stock market, the numbers on the screens of professional traders and the ticker on CNBC, was an illusion. Traders would hit a button to buy or sell a stock and the market would move away from them. You'd try to buy 10,000 shares of Apple that seemed to be on offer, but only 2,000 would actually get bought. You weren't told from which exchanges the missing 8,000 shares had vanished.

The public market had thirteen stock exchanges scattered in four different sites run by the NYSE, Nasdaq, BATS, and Direct Edge.

Brad Katsuyama at Royal Bank of Canada had Rob Park and Allen Zhang write a program that allowed Brad to send orders to a single exchange. When he sent to a single exchange, he was able to buy everything. But the more exchanges he added, the less of his order was filled. BATS always filled one hundred percent of what was offered.

That made Rob connect the dots. They had a bar chart of the time it took orders to travel from Brad's desk in the World Financial Center to the various exchanges. BATS had the shortest travel time, about 2 milliseconds, and the slowest was Carteret at around 4 milliseconds. In practice, times varied more than that. Allen wrote a program that built delays into the orders Brad sent to the faster exchanges so that they arrived at exactly the same time as the slower exchanges. His order then filled one hundred percent.

That meant someone was using the fact that stock-market orders arrived at different times at different exchanges to front-run orders from one market to another. Some other trader was noticing his demand for stock on one exchange and buying it on the other exchanges in anticipation of selling it back to him at a higher price.

Front-running, in this context, meant they saw you wanted 10,000 shares at $10, and their technology was so fast that they could detect your order and buy the shares before you. Then they held the shares and sold them back to you at a higher price.

Light in a vacuum travels at 186 miles a millisecond. Inside fiber it bounces and travels at about two-thirds that speed.

Ronan Ryan, an Irish telecom guy, moved a Kansas City trading firm's computers from Kansas City to Radianz's data center in Nutley, New Jersey, and reduced the time it took them to find out what they had bought and sold from 43 milliseconds to 3.8 milliseconds. Suddenly everyone wanted that.

Proximity services, co-locating near the exchange, became very popular. Between 2005 and 2008, Radianz billed HFT firms $80 million just for setting up their computers near exchange matching engines. Faster data switches, shorter fiber cables, better glass. The HFT firms tried to keep every improvement secret, though anything that gained three microseconds seemed to spread to everyone else within weeks.

The HFT firms posted tiny orders on BATS, buy or sell 100 shares of basically every stock in the US market, so they could find out what investors wanted to buy and sell before the investors did it. Once they learned there was a buyer of company X's shares, they raced ahead to the other exchanges to buy or sell accordingly.

The HFT firms had proprietary maps from telecom providers because they paid them so much. They knew the path each order traveled to each exchange. BATS was the first one out of lower Manhattan, and it had been created by HFTs.

Each exchange charged a small fee per transaction. Some even gave you a kickback and paid you to transact on it, which created bizarre incentives.

Say you wanted to buy 100,000 shares of XYZ at no more than $25 a share. To buy them all at once would move the market. So the trading algorithm would split the order up, maybe twenty pieces of 5,000 shares every five minutes, so long as the price stayed under $25.

The router decided where the order was sent. It might first go to a Wall Street firm's dark pool, then to venues that paid the broker to trade there before venues that charged the broker. These were sequential cost-effective routers.

Imagine 100,000 shares available at $25 across 10 exchanges, 10,000 shares each, and each exchange charges the broker. Then imagine there's also 100 shares available on BATS, which pays the broker for the trade. The cost-effective router goes first to BATS and buys the 100 shares, and in doing so reveals demand. The remaining 100,000 shares vanish into the hands of the HFT firms, who then sell them back at a higher price.

Once Spread Networks got its fiber line running, an exchange in Chicago started giving kickbacks for transactions, causing trades to be routed there and giving traders plenty of time to beat the information back to New Jersey.

Brokers were expected to find the best price for their customer, but despite protests, a huge percentage of customer orders sent into a dark pool were executed inside that pool. Goldman Sachs' dark pool, for example, was less than 2 percent of the entire stock market, but nearly 50 percent of customer orders routed into it ended up being executed inside that pool. Rates of internalization varied from broker to broker.

Regulation NMS required brokers to find the best market prices for the investors they represented. To define best price, it relied on the concept of National Best Bid and Offer, NBBO. Whatever the cheapest offer on an exchange was, your broker had to purchase those shares first.

So if BATS offered 100 shares a penny cheaper than the next lowest venue, your broker was required to buy from BATS first. And you got front-run.

Reg NMS also required a mechanism for measuring the whole market to create the NBBO. That place was the Securities Information Processor, SIP. The thirteen stock markets piped their prices into the SIP, which calculated the NBBO, which is what most investors saw when they looked at the market.

The HFTs set up their own computers inside exchanges and built a much faster version of the SIP that was 25 milliseconds faster. It was like insider knowledge except technically public. They knew Apple was about to move to 400.01, so they bought it at the outdated 400.00 and then sold it. A 2013 study showed that the price for Apple differed between the public SIP and private feeds 55,000 times a day.

So HFTs wanted volatility and fragmentation. More exchanges, farther apart, meant more opportunities.

All this led Wall Street banks to create private exchanges, dark pools. By 2011, 30 percent of all stock market trades occurred off the public exchange because so many people were fed up with being exploited.

Rich Gates figured he was getting screwed in dark pools. He found a stock that hardly traded, Chipotle, and put in a mid-market order. If the stock was trading at 100-100.10, he tried to buy at 100.05 and wait for a seller to come down. Seconds later he went to the public exchange and sold at 100.01. His order to buy at 100.05 should have been immediately filled at 100.01. Instead, someone bought his shares at 100.01 on the public exchange and someone sold to him at 100.05 in the dark pool.

Of all the dark pools they tested, Goldman Sachs' Sigma X ripped him off more than half the time. After he had a story published in the Wall Street Journal without naming Goldman, every bank's dark pool was screwing him except Goldman. Credit Suisse was the worst.

Wall Street banks were selling access to their dark pools to high-frequency traders. A pension fund might want to buy 100,000 shares of Microsoft and route the order into a dark pool so the wider world wasn't informed. The HFTs would ping the pool with tiny buy and sell orders in every listed stock, searching for activity. Once they discovered the Microsoft buyer, they'd wait for the moment Microsoft ticked lower on the public exchanges and sell it to the pension fund in the dark pool at the stale, higher best price.

Sergey Aleynikov was a talented Russian programmer who first worked at IDT writing code that routed millions of calls a day to the cheapest phone lines. Goldman poached him to refactor its electronic trading department: new hardware, better code, better strategies. He didn't really see the whole picture, he just liked the technical challenge. When a friend offered him over a million a year to start an HFT firm from scratch, he said yes. But he was arrested by the FBI for sending himself code he had worked on while at Goldman and ended up with years of legal hell. Lewis presents him as mostly a coder who loved coding and open source.

Brad, Ronan, Rob, and Allen Zhang all left RBC to create a new exchange to protect investors from HFTs. Their idea was to use Thor, the software that made orders hit different exchanges at the same time, to prevent front-running. By 2013 they had enough funding. They named the exchange IEX, Investors Exchange.

For many sophisticated traders, the stock market was no longer a mechanism for channeling capital to productive enterprise. It was a puzzle to be solved.

IEX had to remove the puzzle. Rebates, the maker-taker system of fees and kickbacks used by all the exchanges, were basically a method for paying the big Wall Street banks to screw the investors they were supposed to protect. The rebates were the bait in HFT flash traps. The moving parts were the order types. There were around 150 order types, many effectively built for HFT.

One example was the Post-Only order: I want to buy 100 shares at 80.02, but only if I am on the passive side of the trade and getting a kickback from the exchange. That led to rebate arbitrage, collecting the kickback without actually providing useful liquidity.

Two basic order types matter. A market order means you will pay the offering price now. A limit order means you will pay no more than your limit. The market order risks the market moving while the order is in flight. The limit order risks the market moving away and never filling.

Broadly speaking, three activities drove a huge amount of grotesquely unfair trading: electronic front-running, rebate arbitrage, and slow-market arbitrage, seeing price discrepancies across exchanges and beating everyone to them before the exchanges reacted.

All three depended on speed. IEX's challenge was to design an exchange where every dollar stood the same chance, even if some traders would always be faster than everyone else.

First, they wouldn't let HFTs co-locate in the same way. But that alone didn't solve it, because HFTs could still process everything faster. IEX had to execute its own orders and route any order it couldn't fully fill to other exchanges. If a buy order for a million shares could only get 100,000 filled on IEX, they had to make sure they beat the HFTs to the other exchanges before the HFTs figured out the demand was still unsatisfied.

The solution was distance. They separated the point where traders connected to IEX from the matching engine by enough fiber to wipe out most speed advantages. The delay needed to be long enough for IEX, after filling part of a customer's order, to beat HFTs in the race to the rest of the market. It also needed to be long enough for IEX to reprice its resting orders when prices changed elsewhere so it didn't get picked off like Rich Gates had in dark pools.

The necessary delay turned out to be 320 microseconds, rounded to 350. About one-third of a millisecond. That translated into 38 miles of fiber. Instead of moving the point of presence 38 miles away, they coiled 38 miles of fiber in a box.

Ronan handpicked the fiber routes that went to each exchange so the signals arrived at each one at the same time, achieving with hardware what Thor had achieved with software. To avoid stale pricing, IEX didn't rely on the SIP or some enhanced version of it. They built their own private HFT-style picture of the stock market using the fastest possible direct feeds.

So IEX gave itself a 350-microsecond head start and paired that with HFT-grade market data technology.

A good image from the book: the NYSE building on the corner of Wall Street and Broadway was almost ten times smaller than the NYSE data center in Mahwah, which housed the real exchange but had no people inside it. The space around the black box was valuable because it could be sold to HFT firms.

The investing public had lost faith in the US stock market after the flash crash in May 2010. After the crash, the S&P index rose 65 percent, yet trading volume fell 50 percent. For the first time in history, investors' desire to trade did not rise with market prices. Before the crash, 67 percent of US households owned stocks. By the end of 2013, only 52 percent did. The post-crisis bull market was notable for how many Americans chose not to participate.

IEX opened on October 25, 2013. It traded 568,524 shares on the first day and 12 million the first week. It kept growing, reaching 50 million shares a week by December, though it needed 50 million a day to cover costs.

Goldman Sachs had two new heads of electronic trading, Ron Morgan and Brian Levine. They talked with Brad and were seeing what the market had become and worried another flash crash was bound to happen. Even though Morgan Stanley had outgrown them by making a killing from HFT, they decided to make both a business decision and a moral one.

On December 19, 2013, Goldman used IEX. The IEX office went berserk: twenty million, thirty million, fucking Goldman Sachs, champagne popping. Goldman called and said they weren't even big yet, they were coming big tomorrow. For just 51 minutes, Goldman entrusted IEX with most of its orders. Brad basically just needed one major participant to say, you're right, and that was Goldman Sachs.

Ninety-two percent of those Goldman orders on IEX traded at the midpoint, compared with 17 percent in dark pools and even less on the public market.

The likely reason Goldman changed its mind was that it realized HFT was a game it could never really win or control. The small HFT firms took around 85 percent of the profits anyway, and the big banks could never compete with the best HFT shops. If the markets collapsed, or if another flash crash occurred, the banks would take most of the blame and most of the cost.