« Floating Point Benchmark: Algol 68 Added | Main | Floating Point Benchmark: ALGOL 60 Added »

Monday, May 5, 2014

Reading List: Flash Boys

Lewis, Michael. Flash Boys. New York: W. W. Norton, 2014. ISBN 978-0-393-24466-3.
Back in the bad old days before regulation of financial markets, one of the most common scams perpetrated by stockbrokers against their customers was “front running”. When a customer placed an order to buy a large block of stock, which order would be sufficient to move the market price of the stock higher, the broker would first place a smaller order to buy the same stock for its own account which would be filled without moving the market very much. Then the customer order would be placed, resulting in the market moving higher. The broker would then immediately sell the stock it had bought at the higher market price and pocket the difference. The profit on each individual transaction would be small, but if you add this up over all the volume of a broker's trades it is substantial. (For a sell order, the broker simply inverts the sense of the transactions.) Front running amounts to picking the customer's pocket to line that of the broker: if the customer's order were placed directly, it would execute at a better price had it not been front run. Consequently, front running has long been illegal and market regulators look closely at transaction histories to detect evidence of such criminality.

In the first decade of the 21st century, traders in the U.S. stock market discovered the market was behaving in a distinctly odd fashion. They had been used to seeing the bids (offers to buy) and asks (offers to sell) on their terminals and were accustomed to placing an order and seeing it hit by the offers in the market. But now, when they placed an order, the offers on the other side of the trade would instantly evaporate, only to come back at a price adverse to them. Many people running hundreds of billions of dollars in hedge, mutual, and pension funds had no idea what was going on, but they were certain the markets were rigged against them. Brad Katsuyama, working at the Royal Bank of Canada's Wall Street office, decided to get to the bottom of the mystery, and eventually discovered the financial equivalent of what you see when you lift up a sheet of wet cardboard in your yard. Due to regulations intended to make financial markets more efficient and fair, the monolithic stock exchanges in the U.S. had fractured into dozens of computer-mediated exchanges which traded the same securities. A broker seeking to buy stock on behalf of a customer could route the order to any of these exchanges based upon its own proprietary algorithm, or might match the order with that of another customer within its own “dark pool”, whence the transaction was completely opaque to the outside market.

But there were other players involved. Often co-located in or near the buildings housing the exchanges (most of which are in New Jersey, which has such a sterling reputation for probity) were the servers of “high frequency traders” (HFTs), who placed and cancelled orders in times measured in microseconds. What the HFTs were doing was, in a nutshell, front running. Here's how it works: the HFT places orders of a minimum size (typically 100 shares) for a large number of frequently traded stocks on numerous exchanges. When one of these orders is hit, the HFT immediately blasts in orders to other exchanges, which have not yet reacted to the buy order, and acquires sufficient shares to fill the original order before the price moves higher. This will, in turn, move the market higher and once it does, the original buy order is filled at the higher price. The HFT pockets the difference. A millisecond in advance can, and does, turn into billions of dollars of profit looted from investors. And all of this is not only completely legal, many of the exchanges bend over backward to attract and support HFTs in return for the fees they pay, creating bizarre kinds of orders whose only purpose for existing is to facilitate HFT strategies.

As Brad investigated the secretive world of HFTs, he discovered the curious subculture of Russian programmers who, having spent part of their lives learning how to game the Soviet system, took naturally to discovering how to game the much more lucrative world of Wall Street. Finally, he decides there is a business opportunity in creating an exchange which distinguishes itself from the others by not being crooked. This exchange, IEX, (it was originally to be called “Investors Exchange”, but the founders realised that the obvious Internet domain name, investorsexchange.com, could be infelicitously parsed into three words as well as two), would include technological constraints (including 38 miles of fibre optic cable in a box to create latency between the point of presence where traders could attach and the servers which matched bids and asks) which rendered the strategies of the HFTs impotent and obsolete.

Was it conceivable one could be successful on Wall Street by being honest? Perhaps one had to be a Canadian to entertain such a notion, but in the event, it was. But it wasn't easy. IEX rapidly discovered that Wall Street firms, given orders by customers to be executed on IEX, sent them elsewhere to venues more profitable to the broker. Confidentiality rules prohibited IEX from identifying the miscreants, but nothing prevented them, with the brokers' permission, from identifying those who weren't crooked. This worked quite well.

I'm usually pretty difficult to shock when it comes to the underside of the financial system. For decades, my working assumption is that anything, until proven otherwise, is a scam aimed at picking the pockets of customers, and sadly I have found this presumption correct in a large majority of cases. Still, this book was startling. It's amazing the creepy crawlers you see when you lift up that piece of cardboard, and to anybody with an engineering background the rickety structure and fantastic instability of what are supposed to be the capital markets of the world's leading economy is nothing less than shocking. It is no wonder such a system is prone to “flash crashes” and other excursions. An operating system designer who built such a system would be considered guilty of malfeasance (unless, I suppose, he worked for Microsoft, in which case he'd be a candidate for employee of the year), and yet it is tolerated at the heart of a financial system which, if it collapses, can bring down the world's economy.

Now, one can argue that it isn't such a big thing if somebody shaves a penny or two off the price of a stock you buy or sell. If you're a medium- or long-term investor, that'll make little difference in the results. But what will make your blood boil is that the stock broker with whom you're doing business may be complicit in this, and pocketing part of the take. Many people in the real world look at Wall Street and conclude “The markets are rigged; the banks and brokers are crooked; and the system is stacked against the investor.” As this book demonstrates, they are, for the most part, absolutely right.

Posted at May 5, 2014 21:37