I have an algorithm trader with my mother: we are looking for free frameworks for testing hypotheses and launching in combat mode (Python)
Let's say you had something to do with the stock market before. Or, not having one, they got carried away with the hot (but in recent months already noticeably chilled ....) topic of cryptocurrencies. Also suppose that you went even further and decided that “manual control” of flights is already inefficient and it would be necessary to automate your bright ideas and turn the monkey work into something more technological. Exactly at this moment the questions that I would like to discuss in the article begin, namely: is there a ready solution for backtesting of trading ideas (free is desirable), where to get historical data (ideally for free), and what to do with it all then those. What are the solutions for the combat launch of automated trading systems successfully tested on backtest? Note one and two: the article is written for Python-based libraries and systems, I can not judge how things are with accessibility for other languages; priority is given to foreign markets and / or cryptocurrencies, and I don’t try to judge about the applicability to the stock market of the Russian Federation.
This material is not an exhaustive guide or review, but rather a proposal to exchange working ideas on the basis of a practical search for a normal working assembly for personal use. Faced with the need to automate even uncomplicated trading strategies (momentum trade, trend following, etc.), a number of problems immediately arose. Let's point by point - what do we need to:
Now is the time to make two important digressions regarding the “introductory” with which we work in this article. The first is the choice of language: many will say that C # for creating trading robots / automated trading systems is much more common and faster than Python. I can not disagree, but we must take into account the second important point - the difference between algorithmic trading and automation of trading systems. From conversations with quantitative trading practitioners and simply from the theoretical materials found, it can be concluded that algorithmic trading (often used as a synonym for “High-Frequency Trading (HFT)) is tactics of working with orders, with data flows to different exchanges, based on speed access to them (and dependent on milliseconds), and there exclusively C # can cope with the tasks. At the same time, for tasks that are pursued by the “home” algo-trader, Python is most likely suitable, because we aim at automating short- and medium-term speculative trading systems, where portfolio turnover does not reach 100% per day, and the number of orders and requirements to the speed of their execution do not exceed some reasonable threshold, separating high-frequency trading and short / medium-term automated trading.
')
How are the trading systems evaluated, how are the trading algorithms written, and so on - this is all now beyond the scope, otherwise the article does not fit. Let's look at a specific example - I have a trading system based on the one described in the book „Stocks on the Move: Beating the Market with the Hedge Fund Momentum Strategies“. Trading strategy is written in Python and does not include any supercomplex elements. The first thing you need is an opportunity to test the strategy on historical data. I have no task to discuss the choice of broker, I use the services of Interactive brokers and they already have a native API for Python. However, the possibility of backtesting even with paid subscriptions to the data there is not provided. Here is what I managed to find for backtesting:
1) Everyone knows - Quantopian.com
Only Python 2.7 is supported, there is a possibility to build and test trading systems for free, there is free historical data for use (US equity & futures), it is available online, local installation is impossible. I will not write a long review, I will immediately identify problem areas that I managed to find even at a basic level: since they removed the ability to link to a brokerage account, there is no possibility to use developed and tested trading systems for live trading. This immediately imposes limitations (some of them will help to solve zipline-live, see below), since At the core of Quantopian is the zipline library and, moreover, there are some functions that work only inside their system, so when you want to transfer your samopisny and proven system to the same native API from the broker, you will have to rewrite half of it to make it work, and also decide what to do with zipline-based code fragments. A minus (for some systems) can also be considered a ban on dynamic tickers, but for the time being we will leave this thesis without details.
The main plus of Quantopian (IMHO) (in addition to the free system for tests and historical data) is a very active community, a lot of posts on analytics, and on the nuances of building systems. Important: the historical data available there takes into account some of the problems (you can work not only with live companies, but also receive lists of indices of past years, to take into account fallen out, bankrupt, etc.), splits are taken into account and there is even (!) Total Return datasets, that is, you can use the indicators of profitability, taking into account dividends.
2) He really wants to be heard by everyone, but he still does not come out - Backtrader.com
Python 2.7, 3.2-3.6 is supported, available only for use on a local machine.
It seems like there is integration with brokers and it will be possible to run systems using this library into battle with minimal integration efforts. To be honest, it’s very doubtful, I haven’t seen examples of the living, the community is almost dead - there is silence in the forum. The author of its library is actively promoting Quora and other resources. There are no historical data, so you have to either buy or search for free sources - but there the problems of "survivors" begin, dividends (taken into account or not in the data), splits and other things. I will be glad to real reviews about the practice of using this library.
3) QuantConnect - quantconnect.com
This is an online platform (supporting Python, respectively), offering both data and backtesting and integration with popular brokers. But at the same time, let's say in order to be able to live trading through Interactive Brokers at least you have to pay $ 20 per month for a subscription to QuantConnect, and this does not take into account subscriptions to data from a broker and so on.
My subjective opinion is that the platform is not very convenient for permanent use, somehow everything is too cluttered and inconvenient, plus a fee, plus initially everything is written in C # and the wrapper is simply made so that you can use Python. So, having tried several strategies there, I refused to use it further. Although we must pay tribute - the community is less active there, there is a lot of material on the forums, and problems can be quickly resolved. I would be glad if you share real reviews - maybe I just have a user experience did not work out.
4) QuantRocket - quantrocket.com
Judging by the description - it's just a "song." There is everything - both online and local installation, back-testing, and historical data that can be collected from a broker or provided by themselves (for $ 9 a month), and immediately integrated with brokers, and the integration is so good that they promise that they will not rewrite anything you have to ... And even promise that if you used to Quantopian and you have written some things on the zipline, they will help you with the transition to their platform and the internal library (Moonshot), which will allow you to implement live trading through a broker with minimal effort ... Everything sounds perfect if not for one o "but" - at least $ 30 per month for the opportunity to potestit like everything really. I did not dare, because there are months that do not require your participation in the market (for example, all entry filters in the market show “sit still”) and then a fee of $ 30 for the opportunity to test the hypotheses seems too unjustified. But I would appreciate real feedback - who tried it, what are the pros and cons and so on. Another embarrassing moment - binding to the platform and the library, which is used only on it, is fraught with an increase in "costs of switching", i.e. tomorrow, instead of $ 30, they will put all $ 150 for a simple tariff, and you are tied to their unique Moonshot library and deployed through their platform ... As they say, your eggs are in one basket ...)
5) Zipline & Zipline-live (http://www.zipline.io, www.zipline-live.io )
It is installed locally, works with Python 2.7 and 3.5 for Zipline and only 2.7 for zipline-live.
We got to the cherished bunch, which, according to my observations, is most actively used in the community of fans of automated trading systems at home.
What we have is: the zipline library, which is based on the Quantopian engine (see item 1) and its younger brother, zipline-live, which supports integration with Interactive Brokers and allows you to run algorithms created using zipline in production with minimal modifications. Briefly about the history of the issue - when a couple of years ago Quantopian decided to abandon live trading, the community became agitated, the waters foamed, and the fruit of collective efforts came to light (including the Quantopian team that promised to support the development of a live library on a disinterested basis). After that, Zipline-live became, in fact, the only (?) Convenient way for free (and without registration :) to adapt systems tested on Quantopian (or locally on a zipline) to real trading through Interactive brokers (with which there is integration, but without a number of some important and not so much zipline functions). What can be said about this bundle:
6) QsTrader - github.com/mhallsmoore/qstrader
This is a library from the creators of the quantstart portal, a certain community and a set of educational materials, lectures, and so on. Installed locally, using Python 3 and higher. The possibility of real trading through a broker is not, but they promise to add. In many ways, it relies on the usual Data Science Python stack - pandas, numpy, scipy, etc., so you have to install everything at once. I did not manage to get objective estimates of the prevalence of this library and I will tell you frankly - I did not try it in practice, because judging by the update history, the work on it is inactive, functions are not added, and therefore for a complete, even if it’s a “home” system It is suitable, but if there are results of its use and a personal opinion is formed - share, suddenly this solution is better than others.
7) Next, I will list the libraries that I managed to find, but I can’t say a lot about them. Most often, they are supported solely by the authors of the blogs that created them, and they do not have the ability to integrate with the broker, which means it’s just back testing without the ability to be used in battle, without community and with a slow pace of development, i.e. does not answer all the tasks.
- bt (Backtesting for Python)
- pysystem trade
Summing up the interim results, I share the subjective conclusions:
PS In conclusion, I want to note that the importance of choosing a platform / library is not only a convenience, but also the “scalability” of systems. Let's say I developed something intelligible and tested on a zipline. Then I decided to implement it without a backtest with any broker through its API - and here begins a million additional details that you don’t take into account in the sandbox mode - but how orders are executed, and how your system works in different trading modes, and if there is a gap / post-trade super-political session after the quarterly report, and so on ... Therefore, in my subjective opinion, it is important to choose such an assembly so that the code is working for the backtest and for implementing the trading system in battle, then it will work out the flaws, h to improve the quality of both.
This material is not an exhaustive guide or review, but rather a proposal to exchange working ideas on the basis of a practical search for a normal working assembly for personal use. Faced with the need to automate even uncomplicated trading strategies (momentum trade, trend following, etc.), a number of problems immediately arose. Let's point by point - what do we need to:
- Formulate a hypothesis and algorithmize it
- Test the hypothesis on historical data. Super important note: you must also understand what historical data fall into the category of “suitable” for use in backtesting trading systems. I will not undertake to write a separate post about it (although this requires that, but there is a lot of material on this topic in the public domain), I’ll point out some key problems with historical data: are companies that have dropped out of indices or bankrupt, are considered and how profitability is considered - taking into account dividends (total return) or not. I am very sorry for the brevity, but this post is not about that yet, and this is not an exhaustive set of data requirements.
- If the test is successful, then how to transfer the hypothesis to a real trading account, complementing it with a system of risk management, order management, rebalancing and evaluation of the result.
Now is the time to make two important digressions regarding the “introductory” with which we work in this article. The first is the choice of language: many will say that C # for creating trading robots / automated trading systems is much more common and faster than Python. I can not disagree, but we must take into account the second important point - the difference between algorithmic trading and automation of trading systems. From conversations with quantitative trading practitioners and simply from the theoretical materials found, it can be concluded that algorithmic trading (often used as a synonym for “High-Frequency Trading (HFT)) is tactics of working with orders, with data flows to different exchanges, based on speed access to them (and dependent on milliseconds), and there exclusively C # can cope with the tasks. At the same time, for tasks that are pursued by the “home” algo-trader, Python is most likely suitable, because we aim at automating short- and medium-term speculative trading systems, where portfolio turnover does not reach 100% per day, and the number of orders and requirements to the speed of their execution do not exceed some reasonable threshold, separating high-frequency trading and short / medium-term automated trading.
')
How are the trading systems evaluated, how are the trading algorithms written, and so on - this is all now beyond the scope, otherwise the article does not fit. Let's look at a specific example - I have a trading system based on the one described in the book „Stocks on the Move: Beating the Market with the Hedge Fund Momentum Strategies“. Trading strategy is written in Python and does not include any supercomplex elements. The first thing you need is an opportunity to test the strategy on historical data. I have no task to discuss the choice of broker, I use the services of Interactive brokers and they already have a native API for Python. However, the possibility of backtesting even with paid subscriptions to the data there is not provided. Here is what I managed to find for backtesting:
1) Everyone knows - Quantopian.com
Only Python 2.7 is supported, there is a possibility to build and test trading systems for free, there is free historical data for use (US equity & futures), it is available online, local installation is impossible. I will not write a long review, I will immediately identify problem areas that I managed to find even at a basic level: since they removed the ability to link to a brokerage account, there is no possibility to use developed and tested trading systems for live trading. This immediately imposes limitations (some of them will help to solve zipline-live, see below), since At the core of Quantopian is the zipline library and, moreover, there are some functions that work only inside their system, so when you want to transfer your samopisny and proven system to the same native API from the broker, you will have to rewrite half of it to make it work, and also decide what to do with zipline-based code fragments. A minus (for some systems) can also be considered a ban on dynamic tickers, but for the time being we will leave this thesis without details.
The main plus of Quantopian (IMHO) (in addition to the free system for tests and historical data) is a very active community, a lot of posts on analytics, and on the nuances of building systems. Important: the historical data available there takes into account some of the problems (you can work not only with live companies, but also receive lists of indices of past years, to take into account fallen out, bankrupt, etc.), splits are taken into account and there is even (!) Total Return datasets, that is, you can use the indicators of profitability, taking into account dividends.
2) He really wants to be heard by everyone, but he still does not come out - Backtrader.com
Python 2.7, 3.2-3.6 is supported, available only for use on a local machine.
It seems like there is integration with brokers and it will be possible to run systems using this library into battle with minimal integration efforts. To be honest, it’s very doubtful, I haven’t seen examples of the living, the community is almost dead - there is silence in the forum. The author of its library is actively promoting Quora and other resources. There are no historical data, so you have to either buy or search for free sources - but there the problems of "survivors" begin, dividends (taken into account or not in the data), splits and other things. I will be glad to real reviews about the practice of using this library.
3) QuantConnect - quantconnect.com
This is an online platform (supporting Python, respectively), offering both data and backtesting and integration with popular brokers. But at the same time, let's say in order to be able to live trading through Interactive Brokers at least you have to pay $ 20 per month for a subscription to QuantConnect, and this does not take into account subscriptions to data from a broker and so on.
My subjective opinion is that the platform is not very convenient for permanent use, somehow everything is too cluttered and inconvenient, plus a fee, plus initially everything is written in C # and the wrapper is simply made so that you can use Python. So, having tried several strategies there, I refused to use it further. Although we must pay tribute - the community is less active there, there is a lot of material on the forums, and problems can be quickly resolved. I would be glad if you share real reviews - maybe I just have a user experience did not work out.
4) QuantRocket - quantrocket.com
Judging by the description - it's just a "song." There is everything - both online and local installation, back-testing, and historical data that can be collected from a broker or provided by themselves (for $ 9 a month), and immediately integrated with brokers, and the integration is so good that they promise that they will not rewrite anything you have to ... And even promise that if you used to Quantopian and you have written some things on the zipline, they will help you with the transition to their platform and the internal library (Moonshot), which will allow you to implement live trading through a broker with minimal effort ... Everything sounds perfect if not for one o "but" - at least $ 30 per month for the opportunity to potestit like everything really. I did not dare, because there are months that do not require your participation in the market (for example, all entry filters in the market show “sit still”) and then a fee of $ 30 for the opportunity to test the hypotheses seems too unjustified. But I would appreciate real feedback - who tried it, what are the pros and cons and so on. Another embarrassing moment - binding to the platform and the library, which is used only on it, is fraught with an increase in "costs of switching", i.e. tomorrow, instead of $ 30, they will put all $ 150 for a simple tariff, and you are tied to their unique Moonshot library and deployed through their platform ... As they say, your eggs are in one basket ...)
5) Zipline & Zipline-live (http://www.zipline.io, www.zipline-live.io )
It is installed locally, works with Python 2.7 and 3.5 for Zipline and only 2.7 for zipline-live.
We got to the cherished bunch, which, according to my observations, is most actively used in the community of fans of automated trading systems at home.
What we have is: the zipline library, which is based on the Quantopian engine (see item 1) and its younger brother, zipline-live, which supports integration with Interactive Brokers and allows you to run algorithms created using zipline in production with minimal modifications. Briefly about the history of the issue - when a couple of years ago Quantopian decided to abandon live trading, the community became agitated, the waters foamed, and the fruit of collective efforts came to light (including the Quantopian team that promised to support the development of a live library on a disinterested basis). After that, Zipline-live became, in fact, the only (?) Convenient way for free (and without registration :) to adapt systems tested on Quantopian (or locally on a zipline) to real trading through Interactive brokers (with which there is integration, but without a number of some important and not so much zipline functions). What can be said about this bundle:
- zipline locally suits you if there is a reliable source of historical data or you just have quandl datasets, which are built into the library for free. At the same time, it is necessary to take into account that the library does not support online data broadcasting, and you will have to download them and then do an upload for backtesting ( here 's more on this for all)
- In theory, zipline-live is suitable if the integration with IB is implemented normally, and the algorithms transferred from Quantopian will not need to be significantly processed and circumcised due to unrealized functions in zipline-live.
6) QsTrader - github.com/mhallsmoore/qstrader
This is a library from the creators of the quantstart portal, a certain community and a set of educational materials, lectures, and so on. Installed locally, using Python 3 and higher. The possibility of real trading through a broker is not, but they promise to add. In many ways, it relies on the usual Data Science Python stack - pandas, numpy, scipy, etc., so you have to install everything at once. I did not manage to get objective estimates of the prevalence of this library and I will tell you frankly - I did not try it in practice, because judging by the update history, the work on it is inactive, functions are not added, and therefore for a complete, even if it’s a “home” system It is suitable, but if there are results of its use and a personal opinion is formed - share, suddenly this solution is better than others.
7) Next, I will list the libraries that I managed to find, but I can’t say a lot about them. Most often, they are supported solely by the authors of the blogs that created them, and they do not have the ability to integrate with the broker, which means it’s just back testing without the ability to be used in battle, without community and with a slow pace of development, i.e. does not answer all the tasks.
- bt (Backtesting for Python)
- pysystem trade
Summing up the interim results, I share the subjective conclusions:
- If you don’t mind using online platforms, outdated Python 2.7, and your target tools in strategies are stocks / futures on US sites (and IB broker), then Quantopian + zipline-live using convenient which you will receive free historical data (± corresponding to the quality criteria), a free IDE for testing hypotheses, as well as a local library that allows you to integrate your work with the brokerage account and launch the system into combat flight taking into account all and a combination of limits that have been disclosed hereinabove. Personally, I chose this option for you because of the convenience, free base functionality, and a very active community. In the next post I will tell you how easy (or not easy) it turned out to be testing and transferring the momentum stocks strategy from Quantopian online to zipline-live on a local machine, as well as how well the integration with the IB brokerage account in this library has been worked out, as well as sharing the details of using native Python API from IB.
- If the priority is local installation, full control over the system and does not bother to download historical data in the form of data bundle's for testing hypotheses, and there is also a source of this data or is free from quandl, then a possible solution would be a Zipline + Zipline-live combination, which will allow to you, strategies tested on a local machine on historical data (your own or free ones) to implement via IB and all this is free of charge within the framework of the functionality of the designated libraries and the broker's Python API.
- If you can’t or don’t want to open an account with Interactive brokers, then a lot of the above list will suit you, because QuantConnect, for example, works with other brokers that also support FX / Crypto trading, and so on. And if the tasks in principle of live trading through a broker are not worth doing, then you can try all the described libraries / backtesting platforms and choose the one that answers your tasks most fully.
PS In conclusion, I want to note that the importance of choosing a platform / library is not only a convenience, but also the “scalability” of systems. Let's say I developed something intelligible and tested on a zipline. Then I decided to implement it without a backtest with any broker through its API - and here begins a million additional details that you don’t take into account in the sandbox mode - but how orders are executed, and how your system works in different trading modes, and if there is a gap / post-trade super-political session after the quarterly report, and so on ... Therefore, in my subjective opinion, it is important to choose such an assembly so that the code is working for the backtest and for implementing the trading system in battle, then it will work out the flaws, h to improve the quality of both.
Source: https://habr.com/ru/post/434244/
All Articles