How not to give the algorithm to sell the bank
Hi, Habr! Our team in Moscow is developing an internal algorithmic trading platform. Today we would like to talk about the mechanisms that we add to our architecture to protect against possible failures.
The non-zero probability of errors in the code, even after the most thorough testing and review code, is a fact that must be accepted and taken for granted. Therefore, when designing an architecture, it is always worthwhile to put in place protection mechanisms that will allow the system to function or be completed without causing harm to itself, when it begins to behave not in accordance with expectations. This is especially important in the financial sector, where we work.
Three years ago, everyone had a story about the Knight Capital Group. As a result of the “successful” upgrade of their system, they lost about $ 460 million due to the fact that their trading system set up and bought 397 million shares of different companies at non-market prices. The report on the investigation of this event should probably be on the table of every COO of any financial company - as a reminder, which may lead to an insufficient level of technical development of the processes in the company and the lack of automatic protection systems.
The architecture of any trading system must have in one form or another a subsystem to control the financial risks from trading. The case of KCG in our domestic jargon can be classified as “out of control strategy”. When designing you need to understand that this is only one of the possibilities that can happen with your system. In addition to technological risks, there is, of course, a large set of various “human errors” that can be the result of carelessness or be deliberate, caused by the desire for personal enrichment of people who will use your system. But in this article we only want to discuss possible protection mechanisms against technical risks associated with cases where the algorithm gets out of control and does not behave as planned.
')
In our platform, trading strategies (or, in other words, trading algorithms) are launched within a certain container. In the container are the 'circuit breaker' components that stand in the way from the trading algorithm to the outside world. The closest analogy from the physical world is “fuses”. The main purpose of these 'circuit breakers' is to make an automatic decision to turn off a strategy in the event of the rules that they contain. They have two states: “closed” - they hold all messages between the strategy and the outside world, and “open” - when they block any new requests (aka orders) from the strategy to the stock exchange. In this case, in any state, they always transmit messages from the stock exchange to the strategy.
Returning to the case of KCG: they had different systems for monitoring, but finding and deciding whether to disable the “broken” subsystems took the support team more than 45 minutes. In the conditions of high-frequency trading during this time, the modern trading system is able to sell and buy all your assets hundreds of times. Therefore, the decision to stop the "suspicious" algorithm should be made automatically.
The container in which the algorithm is launched must ensure that the strategy cannot bypass this protection. You can also add from practice that the development of strategies and the 'circuit breaker' of components should be handled by different teams.
Each 'circuit breaker' is a kind of simple rule that should limit the freedom of action of a controlled strategy. A typical rule might sound like this: “A strategy can send no more than 100 orders to the market at any time.” As soon as the strategy tries to send 101 orders to the market, the circuit breaker will go to the open state and will cease to transmit new orders from the strategy further to the market.
As soon as any rule is triggered, the following chain of events is triggered: a) the strategy receives a message that the 'circuit breaker' has passed to the open state and is obliged to complete its work; b) all active orders are removed from the markets that have been supplied by this strategy; c) the trader receives a notification to his trading terminal about an error in the algo-strategy and its forced stop; d) the same message is received by the support team, which should immediately begin an investigation of the incident.
Let's see which, in our opinion, 'circuit breakers' should be present in any trading system:
In general, this is the minimum set of rules that must be present in any system, in our opinion. But this list, of course, goes on and on.
It is also important to note again that the presence of a 'circuit breaker' on your system does not guarantee the absence of problems. This is only one of the lines of defense that you must build within your trading platform. Bugs can sneak inside the algo container and the circuit circuit breaker components. How we deal with the technical risks of these possible errors, we will tell in the following articles if you are interested in this.
The non-zero probability of errors in the code, even after the most thorough testing and review code, is a fact that must be accepted and taken for granted. Therefore, when designing an architecture, it is always worthwhile to put in place protection mechanisms that will allow the system to function or be completed without causing harm to itself, when it begins to behave not in accordance with expectations. This is especially important in the financial sector, where we work.
Three years ago, everyone had a story about the Knight Capital Group. As a result of the “successful” upgrade of their system, they lost about $ 460 million due to the fact that their trading system set up and bought 397 million shares of different companies at non-market prices. The report on the investigation of this event should probably be on the table of every COO of any financial company - as a reminder, which may lead to an insufficient level of technical development of the processes in the company and the lack of automatic protection systems.
The architecture of any trading system must have in one form or another a subsystem to control the financial risks from trading. The case of KCG in our domestic jargon can be classified as “out of control strategy”. When designing you need to understand that this is only one of the possibilities that can happen with your system. In addition to technological risks, there is, of course, a large set of various “human errors” that can be the result of carelessness or be deliberate, caused by the desire for personal enrichment of people who will use your system. But in this article we only want to discuss possible protection mechanisms against technical risks associated with cases where the algorithm gets out of control and does not behave as planned.
')
In our platform, trading strategies (or, in other words, trading algorithms) are launched within a certain container. In the container are the 'circuit breaker' components that stand in the way from the trading algorithm to the outside world. The closest analogy from the physical world is “fuses”. The main purpose of these 'circuit breakers' is to make an automatic decision to turn off a strategy in the event of the rules that they contain. They have two states: “closed” - they hold all messages between the strategy and the outside world, and “open” - when they block any new requests (aka orders) from the strategy to the stock exchange. In this case, in any state, they always transmit messages from the stock exchange to the strategy.
Returning to the case of KCG: they had different systems for monitoring, but finding and deciding whether to disable the “broken” subsystems took the support team more than 45 minutes. In the conditions of high-frequency trading during this time, the modern trading system is able to sell and buy all your assets hundreds of times. Therefore, the decision to stop the "suspicious" algorithm should be made automatically.
The container in which the algorithm is launched must ensure that the strategy cannot bypass this protection. You can also add from practice that the development of strategies and the 'circuit breaker' of components should be handled by different teams.
Each 'circuit breaker' is a kind of simple rule that should limit the freedom of action of a controlled strategy. A typical rule might sound like this: “A strategy can send no more than 100 orders to the market at any time.” As soon as the strategy tries to send 101 orders to the market, the circuit breaker will go to the open state and will cease to transmit new orders from the strategy further to the market.
As soon as any rule is triggered, the following chain of events is triggered: a) the strategy receives a message that the 'circuit breaker' has passed to the open state and is obliged to complete its work; b) all active orders are removed from the markets that have been supplied by this strategy; c) the trader receives a notification to his trading terminal about an error in the algo-strategy and its forced stop; d) the same message is received by the support team, which should immediately begin an investigation of the incident.
Let's see which, in our opinion, 'circuit breakers' should be present in any trading system:
- The maximum number of orders sent to the market - for any strategy there is a reasonable number of orders that it can send during its lifetime. If this number is exceeded, then something went wrong.
- The maximum number of orders sent to the market within a certain period of time - no market likes when it is spammed. Even if your strategy does not cause direct damage to your company, you can be punished by the exchange, because, for example, your strategy can put, cancel and re-set a warrant to buy or sell some paper. The exchanges do not like such orders, because they create a load, but do not lead to real transactions.
- The maximum position in the market that a strategy can open - the strategy may not exceed the maximum number of active orders and the maximum size of each individual order, but it should always have the maximum total size of all open and active orders that it can put on the market. If it exceeds this limit, then this is a sign that the strategy goes beyond the risk that it has defined.
- Maximum number of open active orders on the market - the strategy may not exceed the maximum allowable risk for open positions, but too many active orders on the market may be a signal that something is wrong;
- The maximum delay time for a response from the market / verification of receipt of confirmations for orders sent to the market - the algorithms send orders to the market, each order has its own life cycle. When a strategy sends a new order to the market, it must be confirmed by the other party in accordance with the messaging protocols with the exchange. This rule is responsible for verifying that the world behaves in accordance with the expectations of the algorithm. A strategy cannot work if it sends its orders to the market and does not receive the results of the execution of its orders. In this case, the error may not be inside the strategy, but in the environment. But in any case, trading should be stopped until the reasons for delays and errors in messages are clarified.
- Too good to be true - sometimes the algorithm may try to buy or sell a tool at a price that is too good to be real. Usually, for the implementation of such checks, you must have a source of prices that you take as the average for trading this asset on the market. In case the strategy tries to buy / sell an asset at prices that go beyond the corridor designated by you, this again means that something suspicious is happening and you need to stop trading until you can say for sure what caused it.
- Fat fingers - this check limits the size of the order that the strategy can put in one order at a time to the market. The check is most likely intended to protect against traders if they launch a strategy with some very large orders to buy or sell assets.
- Dead-man switch - any algorithms run by traders who are ultimately responsible for the financial result. The main rule is that the person should always watch the algorithm, follow the open position, financial result. He decides that the algorithm is working within a given program or something goes wrong. This test is intended to deal more with human negligence or forgetfulness, which may result in large financial losses for your bank. In our case, if the trader does not perform any active actions on the computer (keystrokes, mouse clicks) for a long time, a warning window is displayed. If it does not react, then the UI closes the active compounds with the algo container. And the algo-container, already seeing that the session with the UI has closed, stops its strategies that were associated with this session (strategies from several different traders can be launched in the algo-container).
In general, this is the minimum set of rules that must be present in any system, in our opinion. But this list, of course, goes on and on.
It is also important to note again that the presence of a 'circuit breaker' on your system does not guarantee the absence of problems. This is only one of the lines of defense that you must build within your trading platform. Bugs can sneak inside the algo container and the circuit circuit breaker components. How we deal with the technical risks of these possible errors, we will tell in the following articles if you are interested in this.
Source: https://habr.com/ru/post/330008/
All Articles