How-to: how to choose a programming language to create a trading robot

One of the most common questions asked by people just beginning to be interested in algorithmic trading is “Which programming language is best suited for this?”. Of course, the short answer is that there is no "best" option. When choosing a tool, you should take into account the parameters of the trading strategy, the necessary performance, modularity, development methodology and requirements for fault tolerance. In this article we will talk about the main components of the architecture of an algorithmic trading system and how each of them influences the choice of a programming language.

Translator's note : We very often encounter skepticism about algorithmic trading. There is an opinion that this is a solid speculation, which bears exceptional harm, and it’s for a technical specialist to do this, to put it mildly, not comme il faut. Anticipating some of the questions in the comments, we would like to immediately give a link to the material , in which much attention is paid to describing what types of traders exist in stock markets and why EACH of them has some benefit at a certain point in time, as well as a topic in which The more general theme of the appointment of the exchanges themselves is affected And here you can read about the experience of such a trade, which allowed a person with knowledge in programming to earn half a million dollars ( first part , second part .) Happy reading!
')
First of all, we will look at the main elements of an algorithmic trading system, such as analytical tools, portfolio optimizer, risk manager and, in fact, the trading engine. Then we will touch on the features of various trading strategies and how the choice of one of them influences the development of the entire system. In particular, we will discuss the estimated frequency (speed) and trading volume.

After you have chosen a trading strategy, you need to design the architecture of the entire system. This includes the choice of hardware, the operating system (or several systems) and resistance to rare but potentially catastrophic events. When designing the architecture, due attention should also be paid to performance - and, both, the speed of the analytical tools of the system and the trading engine itself.

What does the trading system do?

Before choosing the “best” programming language in which our robot, earning millions, will work, it is necessary to decide on the requirements for this language. Will the system be based solely on the execution of tasks or will we also need risk management or a portfolio designer module? Do I need a high-speed back-testing module to work? For most strategies, trading systems can be divided into two categories: research and generating signals.



Research strategies focus on performance testing on historical data. Testing on data collected in the past is called backtesting. The computing power of the backtesting module is influenced by the amount of data and the algorithmic complexity of the strategy. In optimizing the optimization of the speed of research strategies, the speed of the processor and the number of its cores often act as limiting factors.

If we talk about the generation of trading signals, then the algorithm must understand when to buy or sell and send the corresponding orders (most often through the broker system) to the market. Some strategies require a high level of performance. Such factors as the width of the data channel and the delay introduced by the broker and exchange system (latency) limit the speed of the strategy.

Thus, depending on which strategy you need, and the choice of programming language for its implementation may vary.

Type, liquidity and scope of strategy

The type of trading strategy will affect all its subsequent device. It is necessary to assess the markets in which you plan to trade, the ability to connect external data providers, as well as the frequency of operations performed by the algorithm, and their volume. Important factors will be finding a balance between ease of development and performance optimization, as well as hardware, including servers that need to be placed in brokerage or stock exchange data centers, and additional equipment you may need (GPU, FPGA, etc.).

To trade low-liquid shares in the US markets will need to use completely different technologies than in the case of high-frequency statistical arbitrage strategy in the futures market. Before you start choosing your own programming language, you should choose the data providers with which your trading strategy will work.



It is necessary to analyze the existing connectivity to the supplier systems, the structure of any API, the speed of data delivery, and the possibilities for its storage in case of failures. Organization of access to several such systems at the same time will be a wise decision, which will also require separate study, since each data provider has its own technological requirements (symbols of tickers of exchange instruments, etc.).

The estimated trading frequency will have a decisive influence on how the technological stack of your system will be implemented. Strategies that need to be updated more than once a minute will require more resources for their work.

In the case of strategies that require tick data, it is necessary to develop the entire system according to the performance driven design methodology. HDF5 or kdb + is often used for these tasks.

To process the excess amounts of data required for HFT applications, it is necessary to use both an optimized backtester and a trading engine. The main candidates for the role of a programming language in such a system would be C / C ++ (perhaps, in some places, Assembler). High-frequency strategies often require additional equipment, such as programmable matrices (FPGA), as well as placing servers as close as possible to the exchange core and tuning the network interfaces of the servers themselves.

Research systems

When creating systems of this kind, it is often necessary to resort to interactive development and automation of scripts. The first concept takes place in an IDE like Visual Studio, MatLab or R Studio. Automating scenarios is associated with a large volume of calculations for various parameters and data points. Considering all this, it is necessary to choose a language that provides excellent opportunities for testing the code, and also allows you to achieve acceptable performance when calculating strategies for different parameters.

At this stage, IDEs such as Microsoft Visual C ++ / C # are often used, which includes a variety of tools for debugging, code completion, and working with the entire project stack (ORM, LINQ); MatLab, which is designed specifically for problems of numerical linear algebra and vector operations; R Studio, which is implemented using the statistical language R; Eclipse IDE for Linux Java and C ++ and semi-proprietary IDEs like Enthought Canopy for Python, which includes a variety of data analysis libraries (NumPy, SciPy, scikit-learn and pandas).

For the numerical backtesting, all the mentioned tools are suitable, although due to the fact that the code will run “in the background”, graphic IDEs are not necessarily used. At this stage, you should first think about the speed of the system. Compiled languages ​​(like C ++) are useful when the number of parameters during backtesting is very large. In this case, you should always carefully approach each step in the design, since your system may initially not be as fast. In the case of interpretable languages ​​like Python, back-testing often uses high-performance libraries (NumPy / pandas).

The choice of language for implementing the backtesting module will be determined by the specific needs of your algorithm and the number of libraries available for this language (more on this below). Do not forget, however, that the language used for the back-tester and research environment may differ from the means chosen for the modules of the portfolio designer, risk management and the trading engine.

Portfolio Designer and Risk Management

Many algorithmic traders often underestimate the importance of portfolio constructor and risk management. This is a big mistake, because these funds will allow you to save your money on the exchange. With their help, you can not only reduce the number of risky transactions, but also minimize the cost of trading, reducing transaction costs.

Sophisticated implementations of these components can have a significant impact on the quality and sustainability of profitability. Without them, it is very difficult to create a stable strategy, because the presence of a collection mechanism and a risk manager make it easy to modify the trading system.

The objective of the portfolio designer module is to schedule a set of potentially profitable deals and make those that will bring the greatest benefit — many factors are analyzed (for example, volatility, asset class, and the sector of the company whose shares are traded). In accordance with this, the available capital is distributed among various exchange instruments.

Portfolio design is often reduced to the problem of linear algebra (like matrix factorization), which means that the performance of the mechanism largely depends on the efficiency of the implementation in the system of linear algebra tools. Popular libraries include uBLAS, LAPACK and NAG for C ++. MatLab has extensive capabilities in terms of matrix operations. In Python, NumPy / SciPy is used for such calculations. In order for the system to maintain a high-quality and balanced portfolio, you will need a compiled (and well-optimized) library for working with matrices.

Another extremely important part of any algorithmic trading system is the risk management module. Risk can take many forms: increased volatility (for some strategies, though it is even desirable), increased correlations between asset classes, server outages, so-called “black swans” (events that cannot be predicted) and undetected bugs in the code of the trading program - and this is only a small part of the possible problems.

The risk management module attempts to “anticipate” the consequences of most of the above risks. Very often, for this purpose, statistical analysis is used (for example, Monte-Carlo stress tests). In such calculations, parallelism plays an important role, and, in general, performance problems can be solved by simply increasing the computing power.

Trading engine

The task of the trading engine of the system is to obtain filtered trading signals from the modules of the portfolio designer and risk management, generating trade orders based on them, which are then sent to the brokerage trading system. In the case of “ordinary” private traders, this will most likely require an API or a connection using the FIX protocol. Accordingly, to select a language, it is necessary to analyze the quality of the API itself, the presence / absence of software shells for working with it, the expected frequency of operations and the expected “slippage” between the moment the order is sent to the broker’s system and its appearance in the core of the exchange trading system.

The “quality” of the API is made up of several elements: the quality of the documentation, the performance that the interface provides, whether you need separate software to work with it, or you can establish a connection without a GUI, etc.

Most brokerage APIs have interfaces in C ++ and / or Java. Usually, around each such tool there is a community of broker client users who help develop it and create wrappers for C #, Python, R, Excel and MatLab. It is necessary, however, to remember that any additional plugin may contain various errors, so they must always be carefully tested and ensure that developers are engaged in supporting their creation. It is best to see how often there have been updates in recent months.

Frequency of trading operations is the most important element of the trading engine algorithm. A robot can send hundreds of orders per minute, so system performance is extremely important. If the system is not implemented very well, then the inevitable occurrence of a significant slippage between the price, when the order had to be placed and that for which it was actually fulfilled. This can dramatically tell on profitability.

Languages ​​with static typing (see below) like C ++ / Java are usually best suited for writing a trading engine, but using them raises questions about development time, ease of testing and code support. On the other hand, dynamically typed languages ​​such as Python and Perl are now “fast enough”. Make sure that all components of your system are designed using a modular approach that makes it easy to remove and add new elements to the system over time.

Architecture planning and development process

We have already discussed the components of the trading system, the importance of the parameters of the frequency of trading operations and their volume, but so far we have not touched on infrastructure issues. An independent private trader or employee of a small HFT company or fund is likely to face many challenges — alpha model analysis, risk management and execution parameters, and the final deployment of the system — all to be done on your own. All these are important points, so before headlong into the discussion of programming languages, it would be nice to discuss the optimal system architecture.

Separation of interests

One of the most important tasks in creating a trading robot is “division of interests” or, in the language of software development, the separation of various aspects of a trading system into modular components.

Such a division into components will help in the future to change / replace / add new modules to the system to improve performance, reliability or facilitate its maintenance, without having to check all the dependencies and whether “there is something broken” elsewhere. For trading systems, this approach is the best practice. For systems that operate "at medium speeds" its implementation is highly desirable. In the case of HFT-systems, some rules may have to be ignored in order to achieve even higher speed, but in general, it is necessary to adhere to this approach.

Creating a component map of an algorithmic trading system is a topic that deserves a separate article. However, the optimal approach here is to implement individual components for historical and real market information, data storage, access API, backtesting module, strategy parameters, portfolio designer, as well as risk management module and the trading engine itself.

For example, if problems with the efficiency of working with the data warehouse are detected (even after optimization work), such a module can be easily replaced without almost having to rewrite anything in the data reception or API access components.

Another advantage of the modular scheme is that it allows using different programming languages ​​in different parts of the system. There is no need for tight binding to a particular tool if the communication method of the system components is independent. For example, they can communicate via TCP / IP, ZeroMQ, or other protocols.

A concrete example: the backtesting system can be written for C ++ to achieve high performance, while the portfolio manager and trading engine can be written in Python using SciPy and IBPy.

Performance thoughts

Performance is important for almost any trading strategy. The higher the frequency of the trading system, the more important this factor plays. “Performance” means a lot of things, including the speed of the algorithm, network latency, communication channel, data input / output, multi-threading / concurrency and scaling. Separate books are devoted to each of these aspects, therefore we will only touch them a little. Now we will discuss the architecture and specific programming languages ​​in terms of their impact on the overall system performance.

Donal Knut, one of the fathers of what we call Computer Science, said a very wise thing: "The root of all evil is premature optimization." This is true almost always, but not in the case of the development of the HFT-trading algorithm! If you are interested in creating a less high-frequency strategy, then the general approach in your case will be to build the system in the simplest way and start optimizing it only when bottlenecks are detected.

To identify them, various profiling tools are used. You can create profiles both in MS Windows and Linux environments. For this there is a whole bunch of various tools. Now, as agreed, we will discuss specific programming languages ​​in the context of performance.

C ++, Java, Python, R and MatLab - each of them has high-performance libraries (both internal and external) for basic data sets and algorithmic work. C ++ comes bundled with the Standard Template Librar, and Python contains NumPy / SciPy. In these libraries, you can find standard math problems, and writing your own implementation is a way that can rarely be called profitable.

An exception is the case when you need unique equipment and you use an algorithm that works with some proprietary extensions (such as custom caches). At the same time, it must be remembered that the invention of the wheel often takes time, which can be spent with much greater benefit in developing and optimizing all parts of the trading system. Development time is priceless, especially if you create your system alone.

For a trading engine, there is often a problem of delays since the tools for analyzing the market are usually located on the same machine. Delays can occur at any step of the execution process: the database is being accessed (disk / network delays), it is necessary to generate trading signals (OS or kernel delays), send orders to the exchange (delay of communication channels) and must be processed by the core of the exchange trading system ( stock delays).

To create an effective HFT system, you will have to deal with optimization at the kernel level and optimization of data transfer processes.

Another useful developer tool for high-speed robots is caching. The main idea of ​​caching is to store frequently requested information in such a way that it can be obtained without extra resources. In web development, for example, caching can be used when loading data from a relational database on disk into memory. All subsequent requests to this data will no longer need to be sent to the database, due to which it is possible to significantly improve system performance.

For online trading, caching can also be a very useful thing. For example, the current state of the portfolio can be saved in the cache and kept there until the tools are “rebalanced” in it, which will avoid the need to generate a list of purchased and sold assets anew each time the algorithm is triggered - it can be simply updated. Such an operation requires significant processor and I / O resources.

Unfortunately, caching is not a problem with its tools. Reloading data in the cache due to the changeable nature of cache storage may also require a lot of infrastructure resources. Another problem is the domino effect, in which, at high load, multiple copies of the cache start to be generated by mistake, which leads to a series of failures.

Dynamic memory allocation is an expensive operation. Therefore, high-performance trading applications should be well able to work with memory and be able to allocate and take it at all stages of the program flow. In newer programming languages ​​such as Java, C #, or Pythong, there is an automatic garbage collection, thanks to which memory allocation or release occurs dynamically.

This tool is extremely useful in development because it reduces the number of errors and improves the readability of the code. However, for some HFT-systems it is still better not to use standard means for working with memory, but to implement your own. For example, in Java, with some tuning of the garbage collector and heap configuration, you can improve the performance of HFT strategies.

In C ++, there is no native garbage collector that is why it is necessary to control the allocation and freeing of memory during the implementation of objects. This, of course, makes it more likely that errors will occur, but it also allows you to better control objects and heaps in specific applications. When choosing a programming language, take a closer look at how the garbage collection works in it, and whether it is possible to somehow optimize the operation of this mechanism for specific scenarios.

Many operations in algorithmic trading can be paralleled, that is, they can be made so that various program operations are executed simultaneously. The so-called "stunningly parallel" algorithms include steps that can be performed completely independently of other steps. Specific statistical operations, such as Monte Carlo simulations, are good examples of such parallelized algorithms, since each probability and course of events at its occurrence can be calculated without knowing the other possible paths of the situation.

Other algorithms only partially lend themselves to parallelization. Algorithms of this type include modeling in hydrodynamics, where the computational domain can be divided into separate domains, but they must still be linked to each other. The parallelizable algorithms obey the Amdahl law, which introduces the theoretical upper limit to increasing the performance of the parallelized algorithm in the case of N separate processes (for example, on the processor core or in the stream).

Parallelization has become an important element of optimization, since the processor clock speeds have not increased recently, and new processors contain more and more cores that can perform parallel computing. The development of graphics hardware (especially for video games) has led to an improvement in the GPU, which now contain hundreds of "cores" for performing multiple simultaneous operations. And the price of such a GPU has become much more acceptable. High-level frameworks like Nvidia's CUDA are widely used in science and finance.

Typically, such GPU devices are suitable only for research tasks, but there are some (including programmable FPGAs) that are used directly for HFT. Currently, the overwhelming majority of modern programming languages ​​support multithreading to one degree or another, which will allow you, for example, to optimize the backtester so that it uses processes independent of each other.

Scaling in software development means the ability of the system to cope with ever-increasing loads in the form of more requests, more CPU usage and more allocated memory. In algorithmic trading, the strategy is "scaled", it can work with a large amount of capital and still consistently give a positive result. The stack of trading technology is scalable if it can handle large volumes and cope with the increased latency without the occurrence of bottlenecks.

Of course, systems should be designed so that they can be scaled, but it is rather difficult to predict the occurrence of problems and bottlenecks. Strict logging, profiling and monitoring will make the system more scalable. Some programming languages ​​are often described as "unscalable." In fact, those who say so simply “do not know how to cook them.” The entire technology stack can be unscalable, but not the language itself. Naturally, some languages ​​have better performance than others in specific cases, but one cannot say that some language is “better” than another in all senses.

As we said above, it is necessary to divide interests. In order for the system to cope with the “spikes” (as they call sudden volatility, which causes a large number of transactions), it is useful to create a “message queue architecture”. This means that there is a message queue between the components of the trading system, so that the system hangs if a certain component cannot process many requests.

Iron and operating systems

The hardware on which your trading system runs can have a significant effect on the profitability of the algorithm. This does not even concern exclusively high-frequency merchants - bad servers can fail at any time, and it will not matter how many transactions your robot makes if it cannot do one but very important operation due to the fault of the hardware. Therefore, the choice of hardware for the trading system is extremely important. Usually, the choice is between the user's own computer, a remote server, a cloud-based virtual machine or a server on colocation (in the data center of the exchange or broker).

It is clear that the desktop version is the easiest and cheapest thanks largely to the existence of a large number of user-friendly operating systems (Windows, Mac OS, Ubuntu). But they have significant drawbacks. One of the main ones is that each new OS upgrade will require patching of the trading robot, plus the computer will have to be restarted periodically, which is also not very good. In addition, the computational resources of the personal machine are spent on maintaining the GUI, and in fact they could be spent on increasing the performance of the trading system!



In addition, work at home or in the office is fraught with problems with uptime and Internet connection. The main advantage of a desktop system is the fact that additional computing power for it can be purchased for an amount much less than is required to upgrade a server of similar speed.

Dedicated server or cloud machine will come to you more expensive than a desktop computer, but also allow you to organize a much more redundant infrastructure - including automated data backups, the ability to configure uptime monitoring and speed. They will be more difficult to administer, since this will require at least a remote connection.

For Windows-based servers, RDP will most likely be used, and in Unix-based systems it is impossible to do without SSH - there is no way to get away from the command line at all, which makes some development tools like Excel or MatLab inapplicable due to their inability to work without a graphical interface.

A colocation server means simply that you place your server as close as possible to the stock exchange's core — to its data center, or to a broker's data center , which is located in the same local area network with the exchange system. For some HFT strategies, this is the only acceptable solution, despite the fact that it is the most expensive.

The final aspect to consider when choosing software and a programming language is platform independence. Is it necessary to execute code on different OS? Or is the code designed to run on a specific processor architecture - for example, x86 / x64, - or can it be run on ARM RISC processors? Answers to these questions will depend directly on the expected frequency and type of trading strategy.

Resilience and Testing

, . ( ), , - (, -), . . , , , .

, 50% .

. , , .

— , C++ Java, Python . , pdb — . Microsoft Visual C++ IDE GUI, Linux C++ gdb.

. — TTD Test Driven Development, , , , .

— , . C++ Boost, Java JUnit. Python , . .

. . — , , . — — , , .

Windows Linux , -, . — .

, , . : , , — .

— , , . , , - . (email, SMS, ) .

, , , , , .

— , . : 1) - () ( ), ? 2) «» ( ) ? .

— . , , «» .

. — .

Language selection

, . .

. , , . C++ Java — . « » . , , Python, Perl JavaScript

, , , , . — , . , . , TDD .

Open source ?

, , . . , , , , .



Microsoft .NET ( Visual C++, Visual C#) MathWorks MatLab . .

, . .NET C++, C# VB, Microsoft SQL Server ( LINQ). MatLab ( ), .

. — , - ( Microsoft Visual Studio ). Microsoft , - . , Visual Studio WIndows, Linux-. MatLab , .

— . , , , , .

open source-. Linux, MySQL/PostgreSQL, Python, R, C++ Java , . , «» . Python R , , ( , ).

— . Python R . , console based , .

, ( ) , HFT , . Python R , , , . , .

open source , , . Linux- , . Python R . , , C++ , .

, , . , .

? . C++, Java Python , HTTP, , , .

C++ STL (Standard Template Library), . Python ( ), . R , MatLab ( , , ).

C++ Boost, , STL. Boost TR1, C++11, - .

Python NumPy/SciPy/Pandas, — . , MySQL++ (MySQL/C++), JDBC (Java/MatLab), MySQLdb (MySQL/Python) psychopg2 (PostgreSQL/Python). Python «» R RPy!

, API ( SmartCOM — . .). API C++ Java, C# Python.

Conclusion

, , (-) — , . , , , , , , , .

, «» , . — , .

Note : . Thanks for attention!