Exchange Zodiac: What algorithms and tools are used to predict the movement of stock prices

Today we will talk about technical analysis tools that are used to predict the behavior of stock indexes. Our task was not to gather all the technological ways of forecasting prices on stock markets in one pile and describe in detail. For each of them you can find enough detailed information in our blog. But a small cheat sheet would be quite helpful.

A truly effective exchange strategy can only be created using most of the tools in the complex. Moreover, the strategy itself involves several steps, including the collection and processing of data, the construction of the algorithm, debugging and checking in real time. And for each of them you can use different methods and mathematical models.
')

Mathematical models of price prediction

The turnover of algorithmic trading on large stock exchanges today reaches, according to some data, 70%. In this case, it is not just about getting ahead of competitors in the transaction, but also being able to predict the price movement. This can be done, for example, with the help of a mathematical formula that takes into account the hidden liquidity of the market with a given liquidity of applications for the purchase and sale. The "exhaustion" of the queue of applications for the purchase or sale may indicate an early price move

A change occurs when at one of the price levels all orders for purchase or sale disappear, and there is the next level of bid and ask prices.

In one of our previous materials, we considered a formula that allows us to calculate the probability that the queue of ask requests will be depleted earlier than the queue of bid requests.

The formula for calculating the probability of a price increase was as follows:

where H is the hidden liquidity of the market, that is, transactions that are unknown to the general public (for example, transactions of large financial organizations that are concluded outside the stock exchanges).

The analysis procedure itself is as follows:

To begin with, the collected data is divided by exchanges, one trading day is analyzed at a time.
Quotes for bid and asc values are arranged by deciles (). For each such set, the frequency of price increase is calculated.
Count the number of occurrences of each value.
An analysis of the conformity of the model using the method of least squares .

It is necessary to clearly realize that stock indices, in addition to purely economic factors, are affected by a mass of extraneous things. Therefore, a mathematical model for such non-linear, unpredictable series will be extremely difficult to create. But it is quite possible to use it as a basis for more complex strategies.

Machine Learning and Big Data

Machine learning is probably the most sought after and promising area of complex financial calculations. In our blog, he, as well as, in general, work with big data, is devoted to a series of materials (we wrote about this, for example, here and here ).

The process of machine learning itself consists of several steps: from the selection of mathematical and software tools, the collection of input data, to the development of predictions and testing. The easiest way is to create a model based on historical data using machine learning, test it and then use it to generate forecasts of future price movements.

The easiest way to understand the work of this model is with a specific example. Here in this article, in some detail, successively describes the successful experience of using machine learning strategies.

The model involves the creation of a framework that simulates trading, which should as accurately as possible recreate the behavior of the real market. It contains a training data set that allows the system to learn from them. Then, an algorithm is created or selected that is responsible for predicting price movements and organizing trades. You can integrate ready-made algorithms. For example, hidden Markov Models, artificial neural networks, boosting algorithm, naive Bayes classifier, support vector machine, decision tree, analysis of variance and many others.

Further actions will depend on the algorithm used (for specific examples, see the links above). Usually, this is followed by the selection, creation and optimization of indicators that will be involved in the forecast. They are, by and large, tied to raising or lowering prices. Based on the curves of changes in indicators, you can create a formula for more accurate price prediction. To test the resulting algorithm can be on historical data.

Another interesting approach to using machine learning to predict stock prices is its application to financial analysts' forecasts. A similar mechanism works like this:
At the entrance of the system, the opinions of stock market experts are presented (just their opinion, which is not necessarily true), and then, based on their forecasts, predictions of possible price movements are made time after time. At each iteration, the weight of the expert, whose prediction turned out to be true, rises, while for those who made a mistake, on the contrary, it decreases.

Such a weighing technique based on expert opinions can be viewed as a hybrid approach combining fundamental and technical analysis — experts make predictions based on fundamental analysis, and the algorithm subsequently uses them to generate their own forecasts using technical analysis methods.

Adaptive Filtering Algorithm

The adaptive filtering algorithm is widely used in radio electronics as a digital data processing system. Without going into details, the adaptive filter is a self-learning system aimed at achieving the maximum correspondence of the analyzed data at the output to the real state of affairs.

Adaptive filter block diagram for signal prediction

The essence of the method is that we can quickly and clearly respond to changes in the input data to obtain accurate predictions. In practice, adaptive algorithms are implemented by two classical methods - the gradient method and least squares (LMS and RLS).

At one time, the LMS filter was successfully used to predict traffic in wireless networks. Brazilian scientists came up with the idea to try out this algorithm in stock trading. They created a module to predict the price movement of one of the companies. For this, an adaptive digital FIR filter with 100 real coefficients was used. RLS with a forgetting factor of 0.98 was used as an adaptation algorithm. The simulation was carried out on the MATLAB platform.

Tests with different parameters showed that the use of an adaptive filter allows you to achieve a profit of an average of 7% of the investment.

Genetic algorithms

Another recent trend in the field of algorithmic trading is genetic algorithms. These are search algorithms used in systems where the exact relationship of elements is unknown or completely absent.

How it works: the task is set, formalized in such a way that a solution coded as a vector of genes (“genotype”) appears at the output. Genes are any objects, numbers, bits. Further, a set of genotypes of the initial “population” is randomly created, which are estimated using a special fitness function. As a result, each genotype is assigned the value of "fitness" - it determines how well it solves the problem.

In our blog, we wrote about the work of scientists from the Islamic Azad University, which deals with predicting the behavior of stock indices through a combination of genetic algorithm methods, neural networks and data mining using reference vectors.

In this case, data mining is responsible for collecting information and organizing data in the classification model. Genetic algorithm configures the system. In order to optimize it, each gene is considered as a vector, and the corresponding optimization algorithm applies an intermediate recombination mechanism to it. Predictions are generated via the support vector machine (a special case of machine learning). The accuracy of the predictions for the NASDAQ generated by the system was 74.4%.

News Analysis

The fact that news can seriously affect the stock market today will not surprise anyone. Examples of how this or that event (sometimes fake) "brought down" the market appear with enviable regularity - sometimes the creators of such fakes then experience problems with the law . But few have been able to turn economic news manipulation into real art.

In 2015, 62-year-old Scotsman Alan Craig created two fake twitter accounts of analytical companies and posted news about the problems of companies traded on the stock exchange. The graph shows the rise in prices of one of these companies after the publication of fake news and their fall after its refutation.

Analysis systems appear on the market, using publications in the media and social networks to complete transactions. Work is underway to create systems that will be able to create articles on their own, based on data feeds, and then posting them to the network to provoke those who do not have complete information to buy or sell assets.

Back in 2013, researchers from Warwick Business School published the results of an experiment in which the Google search engine and, in particular, the Google Trends service were used as a tool for predicting trends in the stock market.

It allows you to work with information about search queries, ranked by popularity. Researchers have suggested that there is a correlation between the increase in the number of search queries on certain political and economic topics and significant events in stock markets.

Obviously, before making a decision, people try to find out as much as possible through a search engine. Information about search inquiries about topics that may affect stock prices may indicate a reversal of the market trend - if ordinary people already, rather than professional analysts, are interested in business on the stock exchange, this is a sure sign that a trend is about to reverse.

The investment game simulator created as part of an experiment by scientists has shown impressive results. For example, the most reliable for the United States was the word "debt." By tracking markets only by it, scientists increased their hypothetical securities portfolio by 326% over seven years. When modeling the standard trading strategy, which did not take into account the frequency of search queries, they managed to achieve an increase of only 16%.