High Frequency Trading: A Guide from the Hero of the Flash Boys

Translator's note: Earlier in our blog on Habré, we looked at various stages of developing trading systems (there are online courses on the topic), and even described the development of an event-oriented backtest module in Python . Today we bring to your attention a short guide to high-frequency trading from Brad Katsuyama (Brad Katsuyama) - the famous quantum and hero of the bestseller Michael Lews " Flash Boys: A Wall Street Revolt " (we published an adaptation of this work in the blog).



In the release of the Wall Street Week program, Brad Katsuyama talked about high-frequency trading (HFT) and market structure, and we made a review on HFT trading and financial regulation from the point of view of the order execution process on the exchange. The video is available here .

')

What is high frequency trading (HFT)?

In fact, high frequency trading (HFT) is the conduct of electronic trading at a very high speed. Despite the fact that the activities of HFT traders are often criticized, only certain types of HFT trading havoc in the modern financial market. The line between algorithmic trading, electronic market-making and harmful HFT trading is quite blurred, and high-frequency trading often means electronic trading. In fact, the phenomenon of HFT-trading itself is neither good nor bad, but the devil is in the details.



To clearly represent the possibilities of HFT trading, it is worthwhile to consider some types of market activity in more detail.

What is algorithmic / system trading?

• Algorithmic / system trading is a common name for the process of using programmable systems that use a specific mathematical model for automatic execution of transactions. A person creates a program on a computer for a specific financial strategy based on this criterion and manages the developed system from this computer.
• HFT trading is a kind of algorithmic trading, but not all algorithmic trading can be considered high-frequency.

What is manual / discretionary trading?

Manual / discretionary trading is the common name for a set of subjective decisions made by a person, as a rule, based on a number of subjective criteria. A manual trader can be both a small investor making one deal per month, and an intraday trader making several hundred deals a day, and a large organization selling large blocks of shares at various intervals.



In 2011, the Commodity Futures Trading Commission (CFTC) acknowledged that it was not “trying to find a precise definition” for high-frequency trading. Instead, she proposed "seven major signs of HFT trading":

1. Using systems that implement extremely fast placement, cancellation and modification of an order in less than 5 milliseconds or with almost minimal delay.
2. The use of computer programs or algorithms to automate the decision-making process, during which the setting, execution, direction and execution of orders is determined by the system and does not require human intervention in the case of each individual order or transaction.
3. Use of colocation services [eng. co-location - placing foreign servers near the exchange], direct access to the market or a dedicated data channel offered by exchanges and other organizations in order to reduce network and other delays.
4. Very short timeframes for opening and closing positions.
5. High daily turnover of the securities portfolio and / or a high proportion of bids in relation to the number of transactions [eng. order-to-trade ratio].
6. Placing a large number of orders that are canceled immediately or within a few milliseconds.
7. Completion of the trading day in the position as close as possible to zero [eng. flat position] (without holding large unhedged positions at night).

How does the stock market work

The stock market was created to enable companies to access public investment and thus attract financing in order to increase their growth rates. Stock exchanges were supposed to be a place where ordinary sellers and buyers would get together and make deals among themselves in the prescribed manner. However, after the appearance of various intermediaries, the stock market was divided into a large number of segments, and its structure became much more complicated.



Currently, there are 11 public stock exchanges in the United States, about 50 alternative trading systems (ATS), also called “hidden pools”, and about 200 internalizers (usually broker-dealers who can sell / buy securities, acting on their own or someone else's name). The largest public exchanges are the New York Stock Exchange (NYSE) and the NASDAQ. Public exchanges are heavily influenced by regulators, while ATS systems, as a rule, under the control of large banks and financial institutions, are less influenced by the authorities.



When you place an order on your brokerage online account, this order can be transferred to a specific exchange or to the ATS system. After submitting your order, you almost immediately receive confirmation of the fact of the deal, but you cannot see whose hands your order managed to pass within a few milliseconds.

Extensively studied HFT strategies

Shortly after the sharp collapse of US stock indices in May 2010 (Flash Crash), the Securities and Exchange Commission attempted to gather more information about various types of HFT strategies that adversely affect the market.

1. Passive market-making

• Passive market-making in general consists in placing bids for the purchase (bid) and sale (ask) (limit orders) in order to ensure liquidity in the market. Players buying a bid and selling asuks make a profit from the commission for adding liquidity, which is paid in accordance with the maker-taker stock exchange model.
• Regulators are concerned about the quality of liquidity that is supposedly provided by passive HFT marketing. Improper stimulation in the maker-taker model leads to leerings [eng. Layering - market manipulation by artificially shifting quotations for the purchase and sale of securities] and a high level of early failures (at least 90%).

2. Arbitrage

• Arbitration strategies are aimed at making a profit, based on the difference in prices for the same assets that are traded on different trading platforms. Arbitration will always be present in financial markets, but it should not be based on the desire to win in speed and before others to gain access to exchanges in related markets.
• Statistical arbitrage consists in making a profit from the price difference between correlating securities or markets. This type of arbitration uses the methods of mathematical modeling and can be used on any time interval.
• Delay arbitration [eng. latency arbitrage] is to use modern technology in order to gain speed advantage. Under the delay refers to the time that passes from the moment the signal is sent until it is received. When discussing the differences between harmful and harmless types of HFT trading, statistical arbitration is often confused with delayed arbitration.

3. Structural strategies

• Structural strategies, as a rule, are aimed solely at exploiting vulnerabilities in the market structure and obtaining the necessary market data before others. To do this, you need to place your servers close enough to the exchange or get direct access to its information. Having access to the latest data quotes, companies can earn on the execution of orders in the event that they find players to whom information reaches slower.

4. Directional strategies

• The goal of directional strategies is to generate profits by predicting the directional movement of securities prices. In this way, they differ from the three previous types of strategies, taking unhedged risks.
• Using a directional strategy with waiting, the player tries to detect large orders that may affect the price, and place their order before them. Thus, he can use these large orders as “free options” (exit points) if the value of the security does not move in the expected direction.
• When using the strategy of "incitement of momentum" [eng. momentum ignition] the company sets a group of orders in an attempt to “ignite” a sharp price movement in a certain direction. The essence of the principle, called "spoofing", lies in the manipulation of algorithms and manual traders who are forced to conduct aggressive trading. In addition, when using this strategy, the player can cause further price movement due to the set of stop losses. Spoofing is considered an illegal strategy, but it is difficult to detect and prove.

What is the regulation of the national market system?

The National Market Regulations (Reg NMS) is a set of rules and regulations approved in 2005 by the US Securities and Exchange Commission and entered into force in 2007. They are aimed at upgrading US stock exchanges by establishing market equity in pricing, in ways to display quotes, and in providing access to market data.



The following are the basic rules for Reg NMS:

1. The rule of profitable execution of orders: It guarantees investors that their securities warrants will be executed at the best price on any exchange. The rule was created to protect investors by eliminating shortcomings in the traditional rules of trading, since they allow for the disadvantageous execution of orders, especially in the case of limit orders. Despite the fact that this rule was developed with a noble purpose, its unintended consequences had a negative impact on the market structure.
2. Access Rule: The consequence of this rule was the emergence of a system called the “Securities Data Handler” [Eng. Securities Information Processor, SIP], the principle of which is to collect information on quotes on all public stock exchanges and issue the best quotation for the purchase / sale (NBBO) for each security. Data is transferred from all data centers of exchange platforms to one central processor, which then generates a universal spread. Regulators introduced this rule in order to eliminate the negative effects of the growing fragmentation of public exchanges, but instead only increased their fragmentation and provoked the growth of trade in hidden pools.

Unforeseen consequences of the introduction of the rule of profitable execution of orders

An organization that intends to execute a large order is likely to do so in a hidden pool, so as not to disclose its intentions before the conclusion of the transaction. The stock spread consists of the highest bid (sell orders) and the lowest ask (buy orders). If you place a market order for the purchase of 200 shares, you will select an order to sell at the lowest price (considering that the size of this order is at least 200 shares), and your order will be executed at this price.



But what if you want to buy 1,000 or 10,000 shares? As the size of the application increases, more and more problems arise. Suppose you want to buy 1,000 shares, and the application with the lowest price in the "domestic market" is designed only for 100 shares. Let the price per share in this bid for sale be $ 10.10, another 1,000 shares put up for sale a little more expensive - $ 10.11 per share, and then it is proposed to purchase 900 shares at a price of $ 10.15 per share.



If you place a market order for 1,000 shares, first of all you must execute it at the “best price”: in our case it is 100 shares at $ 10.10. Immediately after the execution of your order, the exchange, on which it was executed, will issue a confirmation of a transaction for 100 shares at $ 10.10. After that, you can hope that 900 out of 1,000 shares sold for $ 10.11 will be added to the rest of your application. However, often things are different.



As soon as a confirmation appears on the market for the purchase of 100 shares at $ 10.10, the computer may reject the offer to sell 1,000 shares at $ 10.11, and your application will automatically accept the following "best" offer to sell 900 shares at $ 10.15. The average price of your transaction will be $ 10,145. In this case, the problem is that it is impossible to accept an offer for $ 10.11 after buying shares at $ 10.10 without giving the market a preliminary signal of its intention.



For a small individual investor, the price difference may not turn out to be critical, but for large organizations like pension and mutual funds [Eng. mutual funds], in which money is kept for the majority of the US population, a regular increase in the average price of a transaction over time can lead to significant financial losses.

How did hidden pools appear

It is believed that hidden pools have become even more relevant as a result of economic incentives that have arisen due to imperfect legislation. Hidden pools appeared in the 1980s, when large investment organizations tried to find a way to conclude transactions with each other, in which they do not need to pay commissions and remain hidden from the eyes of public stock exchanges. They wanted to be able to buy and sell large blocks of shares, without disclosing their plans to public stock exchanges and thus making better deals.



In 1998, the Securities and Exchange Commission laid the foundation for today's market structure after the regulation of alternative trading systems (Reg ATS) came into force, and then in 2007 introduced a number of additional regulations for the national market system (Reg NMS).

Growth of trade in hidden pools

More transparent public exchanges are usually referred to as “white markets”, while less transparent alternative trading systems and hidden pools are usually referred to as “shadow markets”. The situation described above is one of the main reasons for the development of trade in “shadow markets”, conducted, in particular, by large investment organizations.



In 2005, even before the norms of regulation of the national market system came into force, the share of hidden pools accounted for 3-5% of the market volume. Today, this figure is 15-18% and continues to grow. Approximately 40% of the total volume of transactions goes through the reporting system on the trading activities of enterprises [Eng. Trade Reporting Facility, TRF]. It covers the entire "over-the-counter trade", including the activities of alternative trading systems, trading in the "upper tier" market [Eng. upstairs trading] (when stock traders directly negotiate with each other and enter into major transactions) and the retail flow of orders. In addition, according to Tabb Group estimates, 54% of the market for derivative financial instruments falls to high-frequency trading.



Why? In simple terms, large organizations can trade in hidden pools anonymously and not report the closure of a large position. The development of shadow trade has had a negative impact on the natural pricing process. Broker-dealers and the largest investment banks, taking into account the consequences of the entry into force of new regulations, were quick to organize several hidden pools to lure the flow of orders from large organizations.



Now, instead of becoming a meeting place for ordinary buyers and sellers — and the stock exchanges were supposed to be that place — for some institutions, they were the last chance to get liquidity. Despite the fact that 60% of trades are still held in the “white markets”, the exchanges cause many problems due to the orders attracted to the maker-taker model (and the reverse taker-maker model). And although public exchanges today are a source of stable liquidity, the reality is that their activity and reliability gradually decrease.

Unforeseen consequences of processing data on securities

Among other things, the regulations of the national market system provide for the creation of a data processor for securities (SIP), which is a regulation that allows for the implementation of arbitration strategies. These strategies are one of the main topics that concern opponents of HFT trading.



SIP is a centralized processor to which all exchanges send their market data in order to create a universal “internal market” for each security listed on the exchange. There are two SIP processors in total: one works with securities registered on the NASDAQ and is placed on its technology platforms, and the second works with securities of the New York Stock Exchange and other organizations trading via technology platforms NYSE. In short, all 11 public exchanges are connected via cable to SIP-processors, which, in turn, collect the data, analyze them and give the best quotes (NBBO) of the securities.



The process of transferring market data from the exchange to SIP processors takes a split second. So, if we take into account the fact that the signal is transmitted at different speeds depending on how close the servers of its clients are to the exchange, there is a small lag compared to the “direct data flow”.



What does the direct data flow mean? At each exchange there is a direct data stream that provides quotes data to those who are “subscribed” to it, faster than the others. The closer you are to the exchange server (or use a microwave connection), the faster you will receive and transmit this data. As for the delay, you should understand that we live in a world where a decisive role can play a few milliseconds or even microseconds.



HFT-companies are ready to pay a decent amount of money to place their servers next to the servers of the exchanges. So they can receive data faster than from a slower SIP processor. The exchange engine, bringing together applications for the purchase and sale, was once on the trading floor of the New York Stock Exchange, and a separate specialist worked with it. Today, these engines are placed in large rooms along with the servers of the exchange.



Supporters of HFT trading claim that everyone can get direct access to the exchange and install their servers next to it, but in reality reputable companies that have the funds to develop and implement HFT strategies on a large scale are involved. Services of placing servers near the exchange bring her a good profit, if we recall that the exchange is first and foremost a commercial organization. HFT companies pay tens of millions of dollars to install their servers in the stock exchange building: the demand for space, connection, speed and bandwidth services is quite high.



As a result, HFT firms receive data faster due to direct access to the exchange, while the rest of the market follows the quotes that are transmitted by a slower SIP processor. Due to this, HFT companies can use delay arbitrage to outrun other orders and thus earn on a large number of transactions. Formally, this strategy differs from the “game ahead of the curve” [eng. front-running] and, rather, is a loophole caused by a conflict of interests and differences in incentive mechanisms.

Layering and order cancellation

If you look at the current state of the US stock market, it will seem to you that this is the most liquid financial market in history, but it has some serious flaws.



Defenders of HFT trading believe that the stock market is in the most favorable condition for the ordinary individual investor. The size of the spread has decreased, the volume of transactions has increased, and this leads to more favorable execution of orders and low commission fees. At first glance, such a situation should suit relatively small individual investors who do not conclude large transactions on an ongoing basis.



However, from the destructive effect of arbitration of HFT companies based on delays, large organizations suffered the most, deriving profit from large transactions. Revenue from high-frequency trading is distributed throughout the market, but most of it is concentrated in the hands of several HFT firms.



Earlier, we already talked about the method by which aggressive HFT companies apply delay arbitrage strategies: thus, they can manipulate a glass of orders without giving market participants the opportunity to execute their orders at the best average price. Although this is just one of the possible scenarios, the market is indeed packed with orders that no one was ever going to execute: they rather serve as bait to attract other market players and get their money.



When supporters of HFT trading talk about the merits of fast automated trading, they overlook the fact that most liquidity is just an illusion. It often happens that we see the minimum spread of any stocks, but the size of the order on the domestic market can be so small that no trader will execute it. When a trader places a small market order, all other orders of a larger size outside the domestic market immediately disappear, and the trader realizes that he will not be able to conclude such a deal. The chief economist of the Bank of England, Andy Haldane, once said: “High-frequency trading adds liquidity in the rainy season and takes it away in the dry season” [in English, the word “liquidity” means not only liquidity, but also a state of fluidity - approx. trans.].

Model maker-taker

With the development of shadow trade, commercial exchanges have to withhold order flows from which they derive their income. Colocation services and surcharges make up a significant part of their profits, but exchanges are forced to maintain a steady stream of orders in order to remain competitive.



To lure order streams flowing into hidden pools - which usually can make a better offer - exchanges resort to such contradictory practices as payment for order flow. Its essence lies in the fact that the exchanges and wholesale companies pay each individual broker-dealer for sending his order along a specific route. As compensation for the sent order stream, the exchange pays the broker-dealer a fee for limit orders for each share. This remuneration is usually a share of pennies per share (0.002 or 0.003 cents), but when it comes to several million shares, the total amount is quite impressive.



One of the variants of such a scheme is used when sending a stream of orders by large organizations: this model is called the maker-taker. Not so long ago, it caused the real wrath of opponents of HFT trading. According to this model, you are considered to be the “source” (maker) of liquidity when you place bids to buy and sell (limit orders) on the stock exchange and receive a reward immediately after the other party accepts the bid you have placed. At the same time, you become a “recipient” (taker) of liquidity when you accept applications to buy or sell (market orders or aggressive limit orders) and actually pay a fine for it. Over the past few years, the size of the remuneration has increased significantly, as each exchange fights for its piece of diminishing pie from the stream of orders still passing through the “white” markets.



The introduction of fees for order flows and the fact of using the maker-taker model are far from the real purpose of financial markets, which is to create a platform that reduces buyers and sellers in the prescribed manner and favors natural pricing, which is a result of the law of supply and demand. If the situation had not been distorted as a result of improper stimulation, which contributed to the development of shadow markets, the introduction of rewards would be superfluous.

Conclusions and future of high frequency trading

If Michael Lewis has somewhat embellished the current state of the stock market, calling it “technically advanced” in his book The Flash Boys, then at least he launched a very important discussion for the entire financial industry on whether the development of financial markets is going in the right direction. .



The high-speed arms race led to a glut and consolidation of the HFT industry. Large fish eat smaller ones; only the fastest HFT companies with the most resources survive today. Any HFT-strategy can be remade by itself, which means that if one HFT-company learns about the strategy of another, it can develop a high-speed algorithm on its basis and take possession of the advantage. If the pie, made up of potential profit from delay arbitration, is still big, many players in the HFT industry will fall into smaller pieces over time.



To restore equity in the financial market, we need to improve its regulation and seize the opportunities for free trade. New investor-oriented exchanges will help even out the situation on the market. Regulatory standards should always be as simple and effective as possible in order to meet the original goal of the stock market, namely, to become a place where pairs of buyers and sellers will be selected in an organized manner. Incorrect motivation has led to the emergence of whole groups of intermediaries, which is not consistent with our ultimate goal - the conclusion of transactions in a free market.