How does the exchange test itself?
We all watched the December events. High market volatility, sharp movements in currency rates, stock prices and futures on them were accompanied by another factor - the very high activity of the exchange's clients. Under these conditions, the key task of the exchange is to ensure the continuity of trading and the maximum resiliency of the systems. We coped with these tasks. And how are we preparing for peak loads? Together with market participants, we perform load testing on a regular basis.
Load testing is an opportunity, both for the exchange and for market participants, to determine the maximum permissible loads on the existing technological infrastructure. The main components of the infrastructure are software and hardware, communication channels, architectural solutions, management processes, support and modernization.
On ordinary days, trading volumes, activity of participants, the number of transactions / orders - the level of market data flows and its intensity - change quite smoothly, within well predictable limits. But sometimes, there are bursts of activity on the market that disrupt the normal course of events. For example, with the release of important economic news, the number of applications increases by several times, as the market demonstrated in the fall of 2014. In such cases of abnormal loads, together with the bidders, we must have an adequate margin for the performance of the trading system and the capacity of the communication channels.
')
The main objectives of testing are formulated as follows:
• Simulate peak loads and their influence on the software and hardware complex of the exchange and the participant.
• Confirm the ability of the entire complex to process data flow in intensity that exceeds average market indicators by 2-3 times. That is, to ensure the "margin of safety" of the entire system by 2-3 times.
• Align the entire “exchange participant” infrastructure, including the “exchange participant”, “exchange provider” and “participant provider” interfaces.
Load testing allows you to estimate the approximate amount of data that bidders will receive from us over the coming year. Brokers, like the stock exchange, based on the analysis of test results, are able to prepare their infrastructure for design loads in a timely manner, adjust further work to optimize the network and trade architecture, and get answers to the questions: “Where are the bottlenecks? What infrastructure components need upgrading? What is the budget for it to allocate? It is to the solution of the last question that the testing time is tied - autumn.
Load, schedule, script
The load during testing is created by our own robotic systems created specifically for testing the trading platforms of the exchange. These simulators are able to create a load profile that corresponds to the real trading, both in the ratio of the bid and bid orders, and in the fluctuations of the price band and other market parameters. In reality, more than 90% of orders are removed by the clients themselves within milliseconds - this is the reality of algorithmic trading, but the load on the system from the order that does not lead to a transaction does not disappear anywhere. “This is how robots torture us with their garbage, no profit, and everyone has to modernize the infrastructure because of them,” our IT scoffs.
Similar machines are being prepared for testing by brokers and vendors of trading software. This year, the load ratio was 50 to 50 between the exchange and bidders. In total, more than 70 professional market participants and vendors took part in testing this year.
The total duration of testing is 4 hours - in which the schedule of the real trading day fully complies with a break in the clearing session. Target numbers of orders for each market exceed the current average daily values ​​of 1.5-2 times. Three markets are tested on two technology platforms with the maximum number of applications per session:
• Stock Market (ASTS) - 30 000 000
• Foreign exchange market (ASTS) - 30 000 000
• Derivatives market (SPECTRA) - 60 000 000
Before testing, all access servers are connected to the trading platforms. And as the load increases, servers that are unable to cope with it are registered in the test.
During testing, the load is created 2-3 times higher than the peak frequency of applications of the real market in seconds intervals, but not reaching the limits of the trading platforms. Direct measurement of the marginal performance of the platforms and their subsystems is carried out briefly at the final stage of the test, so that the results obtained by measuring the delays are not distorted by the resulting queues for processing applications.
In addition to the trading platforms themselves, services and subsystems that are critical to loads and actively used by trading participants and their clients are tested:
• Index servers.
• Website and related information services.
• FIX udp multicast marketdata servers.
• FIX transactional servers.
In testing, switching to backup systems and communication channels is simulated, which is also important for bidders. They can work out these procedures and, if necessary, localize bottlenecks on their side.
Important
Exchange - a huge conglomerate of networks with complex architecture. Currently, the hardware and software complex of the exchange is distributed between two data centers.
• Main - M1 - located on Warsaw highway;
• Reserve - in the Stock Exchange building in Kislovsky Lane.
We understand how it works ...
There are 6 typical connection schemes to the exchange (shown in Fig. “Connection diagram”) and participants of all schemes are tested. With such a diversity, the main part of the market participants (brokers, banks, providers, vendors) is connected according to a universal scheme . 7 authorized telecom operators specifically for the exchange organized private networks (L3 VPN), to which, together with the participants, the main and backup data centers are connected.
Each bidder is recommended to use a fault-tolerant connection to the exchange - at least two communication channels, which is necessary for the purpose of reservation and load redistribution to prevent traffic congestion during periods of peak activity of the trading session.
One of the “bottlenecks” of the technological infrastructure detected, including during the testing process, is the “last mile” of the participant’s connection to the operator’s network. The recommended minimum bandwidth of a dedicated communication channel without using FAST UDP multicast marketdata technology is 10 Mbps. Using FAST UDP multicast marketdata - 30 Mbit / s for the stock market, 20 Mbit / s for the currency market and 15 Mbit / s for the urgent market (in case of receiving data from several markets, the numbers are summed up).
From the point of view of load testing and processing of results, the most representative for us are the data on the output switches from the exchange to the colocation zone, when customers receive data directly from the exchange, without using a telecom operator. Customers put their equipment (servers) in the data center of the exchange. The servers are located next to the hardware complex, and the connection goes directly to the trading platforms with network cables at a speed of 100 Mbit / s, 1 Gbit / s or 10 Gbit / s.
It is here that the struggle begins for every microsecond in data transfer speed. On the collocation, as a rule, there are servers of high-frequency traders (HFT - high frequency trading, they are also GTA - hyperactive vending machines), for which it is important to receive / send data at maximum speed and with minimal spread. Measurements on collocations allow us to most accurately measure all delays, eliminating the influence of networks independent of us.
General scheme of load testing
The general scheme of load testing is as follows:
In these areas, we measure the main indicators:
- Disposal of local networks: utilization of the processor on a network device. We measure in order to check the capacity of our own local network: traffic must be free to go from one point to another. In the trading network all connections at 10 Gb / s.
- CPU utilization : the load level of network devices, which should not exceed 60-70% for their normal operation. Excess results in the switch failing to cope with the volume of traffic, which leads to packet loss.
- Latency : the time between receiving a network message with an order for placing / withdrawing / changing an application on any switch in the path between the access server and the client and receiving a response to the order. Measurements on different switches allow you to evaluate all the components of the delays, including communication channels and software components of the complex
- Jitter : statistical variation of delay values ​​(latency). Persistence latency - an important indicator for all categories of clients. As a rule, maximum response times are measured for the best 50% (median), 80%, 90%, 99% of orders. The ratio of these values ​​gives a good idea of ​​the spread of the values ​​of delays.
What is measured?
In our arsenal there are several measurement tools.
- Adopted by us generally accepted solutions, for example, Network Node Management from HP, with which the utilization of communication channels is measured.
- Internal means of event logging with microsecond accuracy in our software, as well as the good old tcpdump.
“In 2014, for the first time, we used the capabilities of the Corvil precision measuring system - to control the parameters of latency and jitter in a collocation zone sensitive to microsecond delays. This system is used, including the world's largest stock exchanges such as the NASDAQ and the NYSE.
See the example of uploading data from Corvil below.
What happened - a brief summary
Detailed results of testing all exchange systems, access servers, gateways, etc. can be found in the detailed report , but for clarity, we present some numbers:
Stock market
Currency market
Load testing was characterized by high activity in frequency and number of transactions, 25 times higher than typical values ​​for normal trading.
Derivatives Market
The introduction in July 2014 of a new version of TKS SPECTRA with optimized matching, as well as the allocation of a separate data replication branch with accelerated distribution, increased the performance of the TKS core to 52,000 transactions per second and stabilized the latency RTT application. In the course of load testing, according to plan, clearing and intermediate clearing were carried out, and, despite the large volumes of orders and transactions, both procedures were carried out on a regular basis within the planned time interval.
It should be noted that both systems (ASTS and SPECTRA) do not have technological limitations on the number of orders and transactions per trading day, the figures achieved are due to the time and testing program.
We want to thank customers, telecom operators, vendors and all specialists for their active participation in testing, which makes the test results more accurate and fair.
Load testing is an opportunity, both for the exchange and for market participants, to determine the maximum permissible loads on the existing technological infrastructure. The main components of the infrastructure are software and hardware, communication channels, architectural solutions, management processes, support and modernization.
On ordinary days, trading volumes, activity of participants, the number of transactions / orders - the level of market data flows and its intensity - change quite smoothly, within well predictable limits. But sometimes, there are bursts of activity on the market that disrupt the normal course of events. For example, with the release of important economic news, the number of applications increases by several times, as the market demonstrated in the fall of 2014. In such cases of abnormal loads, together with the bidders, we must have an adequate margin for the performance of the trading system and the capacity of the communication channels.
')
The main objectives of testing are formulated as follows:
• Simulate peak loads and their influence on the software and hardware complex of the exchange and the participant.
• Confirm the ability of the entire complex to process data flow in intensity that exceeds average market indicators by 2-3 times. That is, to ensure the "margin of safety" of the entire system by 2-3 times.
• Align the entire “exchange participant” infrastructure, including the “exchange participant”, “exchange provider” and “participant provider” interfaces.
Load testing allows you to estimate the approximate amount of data that bidders will receive from us over the coming year. Brokers, like the stock exchange, based on the analysis of test results, are able to prepare their infrastructure for design loads in a timely manner, adjust further work to optimize the network and trade architecture, and get answers to the questions: “Where are the bottlenecks? What infrastructure components need upgrading? What is the budget for it to allocate? It is to the solution of the last question that the testing time is tied - autumn.
Load, schedule, script
The load during testing is created by our own robotic systems created specifically for testing the trading platforms of the exchange. These simulators are able to create a load profile that corresponds to the real trading, both in the ratio of the bid and bid orders, and in the fluctuations of the price band and other market parameters. In reality, more than 90% of orders are removed by the clients themselves within milliseconds - this is the reality of algorithmic trading, but the load on the system from the order that does not lead to a transaction does not disappear anywhere. “This is how robots torture us with their garbage, no profit, and everyone has to modernize the infrastructure because of them,” our IT scoffs.
Similar machines are being prepared for testing by brokers and vendors of trading software. This year, the load ratio was 50 to 50 between the exchange and bidders. In total, more than 70 professional market participants and vendors took part in testing this year.
The total duration of testing is 4 hours - in which the schedule of the real trading day fully complies with a break in the clearing session. Target numbers of orders for each market exceed the current average daily values ​​of 1.5-2 times. Three markets are tested on two technology platforms with the maximum number of applications per session:
• Stock Market (ASTS) - 30 000 000
• Foreign exchange market (ASTS) - 30 000 000
• Derivatives market (SPECTRA) - 60 000 000
Before testing, all access servers are connected to the trading platforms. And as the load increases, servers that are unable to cope with it are registered in the test.
During testing, the load is created 2-3 times higher than the peak frequency of applications of the real market in seconds intervals, but not reaching the limits of the trading platforms. Direct measurement of the marginal performance of the platforms and their subsystems is carried out briefly at the final stage of the test, so that the results obtained by measuring the delays are not distorted by the resulting queues for processing applications.
In addition to the trading platforms themselves, services and subsystems that are critical to loads and actively used by trading participants and their clients are tested:
• Index servers.
• Website and related information services.
• FIX udp multicast marketdata servers.
• FIX transactional servers.
In testing, switching to backup systems and communication channels is simulated, which is also important for bidders. They can work out these procedures and, if necessary, localize bottlenecks on their side.
Important
Exchange - a huge conglomerate of networks with complex architecture. Currently, the hardware and software complex of the exchange is distributed between two data centers.
• Main - M1 - located on Warsaw highway;
• Reserve - in the Stock Exchange building in Kislovsky Lane.
We understand how it works ...
There are 6 typical connection schemes to the exchange (shown in Fig. “Connection diagram”) and participants of all schemes are tested. With such a diversity, the main part of the market participants (brokers, banks, providers, vendors) is connected according to a universal scheme . 7 authorized telecom operators specifically for the exchange organized private networks (L3 VPN), to which, together with the participants, the main and backup data centers are connected.
Each bidder is recommended to use a fault-tolerant connection to the exchange - at least two communication channels, which is necessary for the purpose of reservation and load redistribution to prevent traffic congestion during periods of peak activity of the trading session.
One of the “bottlenecks” of the technological infrastructure detected, including during the testing process, is the “last mile” of the participant’s connection to the operator’s network. The recommended minimum bandwidth of a dedicated communication channel without using FAST UDP multicast marketdata technology is 10 Mbps. Using FAST UDP multicast marketdata - 30 Mbit / s for the stock market, 20 Mbit / s for the currency market and 15 Mbit / s for the urgent market (in case of receiving data from several markets, the numbers are summed up).
From the point of view of load testing and processing of results, the most representative for us are the data on the output switches from the exchange to the colocation zone, when customers receive data directly from the exchange, without using a telecom operator. Customers put their equipment (servers) in the data center of the exchange. The servers are located next to the hardware complex, and the connection goes directly to the trading platforms with network cables at a speed of 100 Mbit / s, 1 Gbit / s or 10 Gbit / s.
It is here that the struggle begins for every microsecond in data transfer speed. On the collocation, as a rule, there are servers of high-frequency traders (HFT - high frequency trading, they are also GTA - hyperactive vending machines), for which it is important to receive / send data at maximum speed and with minimal spread. Measurements on collocations allow us to most accurately measure all delays, eliminating the influence of networks independent of us.
General scheme of load testing
The general scheme of load testing is as follows:
In these areas, we measure the main indicators:
- Disposal of local networks: utilization of the processor on a network device. We measure in order to check the capacity of our own local network: traffic must be free to go from one point to another. In the trading network all connections at 10 Gb / s.
- CPU utilization : the load level of network devices, which should not exceed 60-70% for their normal operation. Excess results in the switch failing to cope with the volume of traffic, which leads to packet loss.
- Latency : the time between receiving a network message with an order for placing / withdrawing / changing an application on any switch in the path between the access server and the client and receiving a response to the order. Measurements on different switches allow you to evaluate all the components of the delays, including communication channels and software components of the complex
- Jitter : statistical variation of delay values ​​(latency). Persistence latency - an important indicator for all categories of clients. As a rule, maximum response times are measured for the best 50% (median), 80%, 90%, 99% of orders. The ratio of these values ​​gives a good idea of ​​the spread of the values ​​of delays.
What is measured?
In our arsenal there are several measurement tools.
- Adopted by us generally accepted solutions, for example, Network Node Management from HP, with which the utilization of communication channels is measured.
- Internal means of event logging with microsecond accuracy in our software, as well as the good old tcpdump.
“In 2014, for the first time, we used the capabilities of the Corvil precision measuring system - to control the parameters of latency and jitter in a collocation zone sensitive to microsecond delays. This system is used, including the world's largest stock exchanges such as the NASDAQ and the NYSE.
See the example of uploading data from Corvil below.
What happened - a brief summary
Detailed results of testing all exchange systems, access servers, gateways, etc. can be found in the detailed report , but for clarity, we present some numbers:
Stock market
| Values ​​achieved (pcs.) | Maximum processing speed (units per second) | |
|---|---|---|
| Applications | 40 599 733 | 9621 |
| Deals | 2,992,713 | 1318 |
| Transactions | 80 537 469 | 19,100 |
Currency market
Load testing was characterized by high activity in frequency and number of transactions, 25 times higher than typical values ​​for normal trading.
| Values ​​achieved (pcs.) | Maximum processing speed (units per second) | |
|---|---|---|
| Applications | 32,994,049 | 5700 |
| Deals | 747 539 | 700 |
| Transactions | 65,568,622 | 11324 |
Derivatives Market
The introduction in July 2014 of a new version of TKS SPECTRA with optimized matching, as well as the allocation of a separate data replication branch with accelerated distribution, increased the performance of the TKS core to 52,000 transactions per second and stabilized the latency RTT application. In the course of load testing, according to plan, clearing and intermediate clearing were carried out, and, despite the large volumes of orders and transactions, both procedures were carried out on a regular basis within the planned time interval.
| Values ​​achieved (pcs.) | Maximum processing speed (units per second) | |
|---|---|---|
| Applications | 63,815,090 | 36,000 |
| Deals | 3,787,105 | 4,953 |
It should be noted that both systems (ASTS and SPECTRA) do not have technological limitations on the number of orders and transactions per trading day, the figures achieved are due to the time and testing program.
We want to thank customers, telecom operators, vendors and all specialists for their active participation in testing, which makes the test results more accurate and fair.
Source: https://habr.com/ru/post/248083/
All Articles