Geekly Articles each Day
⬆️ ⬇️

Choosing a location for the server and software, testing market inefficiencies: how to actually create trading robots





Financial Hacker, the author of the blog, told about how the process of developing high-frequency strategies for trading on the stock exchange is organized, from the importance of analyzing possible delays to data acquisition and testing (all with sample code). For example, using the strategy of arbitrage trading on US exchanges. We have prepared an adapted translation of this material.



Introduction



Compared to machine learning or signal processing algorithms used in traditional trading strategies, high-frequency trading systems (HFTs) can be surprisingly simple. They do not need to try to predict the future price of shares - they already know it. More precisely, they know the current price a little earlier than the other, slower market participants.

')

The advantage of HFT in obtaining market data and the execution of their applications before most of the participants. The total profitability of the system depends on its speed of delay, the time between the receipt of the quotation and the execution of the order in the trading core of the exchange. Delay (latency) - the most relevant factor in assessing the HFT-system. It can be optimized in two ways: minimizing the physical distance to the exchange, and increasing the speed of the system itself. And the first is much more important than the second.



Location



Ideally, the HFT server should be located directly on the exchange. And most of the marketplaces in the world are happy to sell server places in their data centers - the closer to the hub of the main network of the exchange, the better is the place. Electric signals in a shielded wire are transmitted at a speed of 0.7 - 0.9 of the speed of light (300 km / ms). Reducing the distance to a signal source by one meter results in as many as 8 nanoseconds of advantage per roundtrip (from the time of sending an application to receiving information about its performance). How many trading opportunities can you miss in 8 nanoseconds? No one knows, but people are willing to pay for every nanosecond saved.



Unfortunately (or, fortunately, from the point of view of economy, the placement in the data centers of the exchanges costs a lot of money), the HFT system analyzed in this article cannot be placed on the colocation in the data center of the marketplace for several reasons. At the same time, in order to trade, it needs to receive data from the NYSE exchanges in New York and CME (Chicago) at the same time.



High-speed cables are stretched between these two cities, and there is also a microwave network. In theory, the ideal location for a system with similar requirements is the town of Warren, Ohio. It is located exactly midway between New York and Chicago. It is not known if there is a hub for high-speed merchants, but the distance of 357 miles to both exchanges results in approximately 4 ms delay of the round trip.



image



Warren, Ohio - Mecca HFT Traders (Image: Jack Pearce / Wikipedia Commons)



Without a doubt, the server in this wonderful town will be much cheaper than the server in the rack on the stock exchange in New York. The idea for a startup: buy a couple of garages in Warren, connect to a high-speed cable between New York and Chicago and earn money by renting the server racks!



Soft



When you have already invested money in choosing the optimal location and communication channels for the HFT system, you will definitely want to receive software that will correspond to the required speed. Commercial trading platforms are usually not fast enough, moreover, their code is always closed, it is not known exactly what and how it works in them. Therefore, HFT-systems are almost never based on existing platforms, but are written from scratch. Not in R or Python, but in any of the "fast" languages. This list includes:





With the exception of VHDL, all of the above should be familiar to many specialists (especially developers of computer games in 3D). But the standard language for high-frequency strategy can be called C / C ++. In this material it is used.



Algorithm



Many HFT systems “hunt” competitor traders using “overtaking methods”. They notice your application, and then buy the same asset at the same price a couple of microseconds earlier than you and sell it to you a little more expensive, earning on it. On some exchanges, such trade is prohibited to create equal conditions for all participants, other sites may allow it, hoping to earn more on commissions. In the example from this article, such mechanisms will not be used; instead, an arbitration strategy will be described. Suppose our servers are located in Warren and we have a high-speed channel to Chicago and New York.



Arbitrage will occur between ES and SPY financial instruments. ES is a Chicago-traded S & P500 futures contract. SPY is a New York-based ETF that is also linked to the S & P500 index. One point of ES equals 10 cents of SPY, so the price of ES is about ten times higher than SPY. Since both assets are based on the same index, a high correlation of their prices can be expected. There are publications, the authors of which prove that this correlation will “break” in small time intervals. Any short-term price difference in an ES-SPY pair that exceeds the bid-ask spread creates opportunities for arbitration. The algorithm from the example will work according to the following strategy:





The algorithm is written in C. If you have never seen the code of HFT algorithms before, it may seem a bit strange: 



#define THRESHOLD 0.4 // Entry/Exit threshold //  HFT  //  0     //  1      ES,   SPY //  2      SPY,   ES //     -1 int tradeHFT(double AskSPY,double BidSPY,double AskES,double BidES) { double SpreadSPY = AskSPY-BidSPY, SpreadES = AskES-BidES; double Arbitrage = 0.5*(AskSPY+BidSPY-AskES-BidES); static double ArbMean = Arbitrage; ArbMean = 0.999*ArbMean + 0.001*Arbitrage; static double Deviation = 0; Deviation = 0.75*Deviation + 0.25*(Arbitrage - ArbMean); static int Position = 0; if(Position == 0) { if(Deviation > SpreadSPY+THRESHOLD) return Position = 1; if(-Deviation > SpreadES+THRESHOLD) return Position = 2; } else { if(Position == 1 && -Deviation > SpreadES+THRESHOLD/2) return Position = 0; if(Position == 2 && Deviation > SpreadSPY+THRESHOLD/2) return Position = 0; } return -1; }


The traderHFT function is called from a certain framework (it is not considered in the article), which receives quotes and sends orders. The current best prices for buying and selling by ES and SPY from the top of the order book are used as parameters (it is assumed that the price of SPY is multiplied by ten so that both assets are on the same scale). The function returns a code that tells the framework to open or close positions, or do nothing. The variable Arbitrage represents the average price difference between SPY and ES. Its average (ArbMean) is filtered by a slow exponential moving average, and Deviation from the average is also filtered by a fast moving average to prevent reactions to quotes outside the desired range. The variable Position denotes the machine state, which can take the value of long, short and nothing. The threshold for entering or exiting a position is set at 40 cents. This is the only adjustable system parameter. If the strategy were intended for real trading, it would also be necessary to optimize the threshold value using several months of data on ES and SPY.



Such a minimalist system is not difficult to translate into assembler, or even programmed in an FPGA chip. However, this is not necessary: ​​even if you use the compiler of the Zorro framework compiler (developed by the author of the article), the tradeHFT function is executed in just 750 nanoseconds. If you use a more advanced compiler like Microsoft VC ++, this value can be reduced to 650 nanoseconds. Since the time between two ES quotes is 10 microseconds or more, the speed C is quite sufficient.



In the course of our HFT experiment, two questions are required. First, is there really a difference in the prices of the two instruments sufficient to extract arbitrage profit? Secondly, at what maximum delay will the system still work?



Data



For backtesting HFT-system data, which can usually be obtained from brokers for free, will not work. It is necessary to fork out for the purchase of data on the order book in the necessary resolution or BBO (Best Bid and Offer) data, with the time stamps of the exchange included. Without information about the time at which the quotation was received at the stock exchange, it is impossible to determine the maximum delay.



Some companies record all the quotes coming from the exchanges, and then sell this data. Each of them has its own data format, so for the beginning they will have to lead to a common format. This example uses the following target data format: 



 typedef struct T1 // single tick { double time; // time stamp, OLE DATE format float fVal; // positive = ask price, negative = bid price } T1;


One of the companies that monitor the situation on the CME exchange provides data in CSV format with many additional fields, most of which are not needed for the problem to be solved. All quotes per day are stored in a single CSV file. Below is a script for extracting ES data from it for December 2016 and converting it into datas of T1 quotes: 

 ////////////////////////////////////////////////////// // Convert price history from Nanotick BBO to .t1 ////////////////////////////////////////////////////// #define STARTDAY 20161004 #define ENDDAY 20161014 string InName = "History\\CME.%08d-%08d.E.BBO-C.310.ES.csv"; // name of a day file string OutName = "History\\ES_201610.t1"; string Code = "ESZ"; // December contract symbol string Format = "2,,%Y%m%d,%H:%M:%S,,,s,,,s,i,,"; // Nanotick csv format void main() { int N,Row,Record,Records; for(N = STARTDAY; N <= ENDDAY; N++) { string FileName = strf(InName,N,N+1); if(!file_date(FileName)) continue; Records = dataParse(1,Format,FileName); // read BBO data printf("\n%d rows read",Records); dataNew(2,Records,2); // create T1 dataset for(Record = 0,Row = 0; Record < Records; Record++) { if(!strstr(Code,dataStr(1,Record,1))) continue; // select only records with correct symbol T1* t1 = dataStr(2,Row,0); // store record in T1 format float Price = 0.01 * dataInt(1,Record,3); // price in cents if(Price < 1000) continue; // no valid price string AskBid = dataStr(1,Record,2); if(AskBid[0] == 'B') // negative price for Bid Price = -Price; t1->fVal = Price; t1->time = dataVar(1,Record,0) + 1./24.; // add 1 hour Chicago-NY time difference Row++; } printf(", %d stored",Row); dataAppend(3,2,0,Row); // append dataset if(!wait(0)) return; } dataSave(3,OutName); // store complete dataset }


The script first parses the CSV into an intermediate binary dataset, which is then converted into the target format T1. Since time stamps are stamped in Chicago time, you need to add one more hour to convert them during New York.



The company tracking the New York Stock Exchange delivers the data in a highly compressed specially NxCore Tape format, it needs to be converted to the second T1 list using a special plug-in: 



 ////////////////////////////////////////////////////// // Convert price history from Nanex .nx2 to .t1 ////////////////////////////////////////////////////// #define STARTDAY 20161004 #define ENDDAY 20161014 #define BUFFER 10000 string InName = "History\\%8d.GS.nx2"; // name of a single day tape string OutName = "History\\SPY_201610.t1"; string Code = "eSPY"; int Row,Rows; typedef struct QUOTE { char Name[24]; var Time,Price,Size; } QUOTE; int callback(QUOTE *Quote) { if(!strstr(Quote->Name,Code)) return 1; T1* t1 = dataStr(1,Row,0); // store record in T1 format t1->time = Quote->Time; t1->fVal = Quote->Price; Row++; Rows++; if(Row >= BUFFER) { // dataset full? Row = 0; dataAppend(2,1); // append to dataset 2 } return 1; } void main() { dataNew(1,BUFFER,2); // create a small dataset login(1); // open the NxCore plugin int N; for(N = STARTDAY; N <= ENDDAY; N++) { string FileName = strf(InName,N); if(!file_date(FileName)) continue; printf("\n%s..",FileName); Row = Rows = 0; // initialize global variables brokerCommand(SET_HISTORY,FileName); // parse the tape dataAppend(2,1,0,Row); // append the rest to dataset 2 printf("\n%d rows stored",Rows); if(!wait(0)) return; // abort when [Stop] was hit } dataSave(2,OutName); // store complete dataset }


The Callback function is called by any quote in the source file, but most of the data is not needed, so only SPY (“eSPY”) quotes are filtered out.



Proof of market inefficiency



Having obtained data from two sources, we can now compare ES and SPY prices in high resolution. Here is a typical ten-second sample of price curves:



image



SPY (black) vs. ES (red), October 5, 2017, 10:01:25 - 10: 01.35



The resolution here is one millisecond. ES drawn in dollar units, SPY - in tenths. Charts prices are “ask” prices (asking price). It seems that prices are strongly correlated even at such a small interval. ES is a little behind.



The opportunity for arbitration arises at the site in the center - at about 10:01:30, ES responded to the changes a little slower, but stronger. The reason could be some kind of event like a sharp jump in prices of one of the stocks included in the S & P 500 index. For several milliseconds, the difference between ES-SPY exceeded the bid-ask spread of two assets (usually it is 25 cents for ES and 1-4 cents for SPY ). Ideally, you could sell ES and buy SPY. Thus, we have confirmed the theory of the existence of market inefficiency, which opens up opportunities for earnings.



Script for drawing graphs in high resolution: 



 #define ES_HISTORY "ES_201610.t1" #define SPY_HISTORY "SPY_201610.t1" #define TIMEFORMAT "%Y%m%d %H:%M:%S" #define FACTOR 10 #define OFFSET 3.575 void main() { var StartTime = wdatef(TIMEFORMAT,"20161005 10:01:25"), EndTime = wdatef(TIMEFORMAT,"20161005 10:01:35"); MaxBars = 10000; BarPeriod = 0.001/60.; // 1 ms plot resolution Outlier = 1.002; // filter out 0.2% outliers assetList("HFT.csv"); dataLoad(1,ES_HISTORY,2); dataLoad(2,SPY_HISTORY,2); int RowES=0, RowSPY=0; while(Bar < MaxBars) { var TimeES = dataVar(1,RowES,0), PriceES = dataVar(1,RowES,1), TimeSPY = dataVar(2,RowSPY,0), PriceSPY = dataVar(2,RowSPY,1); if(TimeES < TimeSPY) RowES++; else RowSPY++; if(min(TimeES,TimeSPY) < StartTime) continue; if(max(TimeES,TimeSPY) > EndTime) break; if(TimeES < TimeSPY) { asset("ES"); priceQuote(TimeES,PriceES); } else { asset("SPY"); priceQuote(TimeSPY,PriceSPY); } asset("ES"); if(AssetBar > 0) plot("ES",AskPrice+OFFSET,LINE,RED); asset("SPY"); if(AssetBar > 0) plot("SPY",FACTOR*AskPrice,LINE,BLACK); } }


First, the script reads two files with historical data that we created earlier, and then parses them line by line.



System testing



To backtest the resulting HFT system, you need to slightly modify the script, and call the tradeHFT function in a loop: 



 #define LATENCY 4.0 // milliseconds function main() { var StartTime = wdatef(TIMEFORMAT,"20161005 09:30:00"), EndTime = wdatef(TIMEFORMAT,"20161005 15:30:00"); MaxBars = 200000; BarPeriod = 0.1/60.; // 100 ms bars Outlier = 1.002; assetList("HFT.csv"); dataLoad(1,ES_HISTORY,2); dataLoad(2,SPY_HISTORY,2); int RowES=0, RowSPY=0; EntryDelay = LATENCY/1000.; Hedge = 2; Fill = 8; // HFT fill mode; Slippage = 0; Lots = 100; while(Bar < MaxBars) { var TimeES = dataVar(1,RowES,0), PriceES = dataVar(1,RowES,1), TimeSPY = dataVar(2,RowSPY,0), PriceSPY = dataVar(2,RowSPY,1); if(TimeES < TimeSPY) RowES++; else RowSPY++; if(min(TimeES,TimeSPY) < StartTime) continue; if(max(TimeES,TimeSPY) > EndTime) break; if(TimeES < TimeSPY) { asset("ES"); priceQuote(TimeES,PriceES); } else { asset("SPY"); priceQuote(TimeSPY,FACTOR*PriceSPY); } asset("ES"); if(!AssetBar) continue; var AskES = AskPrice, BidES = AskPrice-Spread; asset("SPY"); if(!AssetBar) continue; var AskSPY = AskPrice, BidSPY = AskPrice-Spread; int Order = tradeHFT(AskSPY,BidSPY,AskES,BidES); switch(Order) { case 1: asset("ES"); enterLong(); asset("SPY"); enterShort(); break; case 2: asset("ES"); enterShort(); asset("SPY"); enterLong(); break; case 0: asset("ES"); exitLong(); exitShort(); asset("SPY"); exitLong(); exitShort(); break; } } printf("\nProfit %.2f at NY Time %s", Equity,strdate(TIMEFORMAT,dataVar(1,RowES,0))); }


The script starts backtesting for one trading day from 9:30 am to 3:30 pm New York. In essence, the HFT function is simply called with ES and SPY prices, and then the code is executed to switch states. It opens positions on one hundred units of each asset (2 contracts for ES and 1000 for SPY). The delay is set using the variable EntryDelay. In HFT mode (Fill = 8), the transaction is at the last price after the delay time. This allows the simulation to be closer to real conditions.



The table below shows the profit from the simulation with different values ​​of delay:

Delay0.5 ms4.0 ms6.0 ms10 ms
Profit / day+ $ 793+ $ 273+ $ 205- $ 15


As you can see, the arbitration strategy of ES-SPY can earn $ 800 a day - with an unrealistically small delay of 500 microseconds. Unfortunately, with 700 miles between the NYSE and the CME, you will need a time machine (or some kind of quantum teleportation tool) to achieve this result. The server in Warren, Ohio, with a delay of 4 ms will bring about $ 300 per day. If the server is a bit away from the high-speed channel between New York and Chicago, the profit will be $ 200. If the infrastructure for trade is even further - say, in Nashville - then nothing will be earned.



Even $ 300 per day will result in an annual income of $ 75,000. But to achieve this result, in addition to hardware and software, a lot of money is also needed. The SPY contract costs $ 250, 100 units for trading will result in 100 * $ 2500 + 100 * 10 * $ 250 = half a million dollars in trading volume. So the annual return on investment will not exceed 15%. The results, however, can be improved by adding more pairs of financial instruments for arbitration.



findings





Other materials on finance and stock market from ITI Capital :



Source: https://habr.com/ru/post/358928/

More articles:



All Articles