Highload: check ERP for strength. How it was
Greetings reader!
In this article I will tell how we tested the performance of our platform .
We planned to test the performance from the very beginning of the development of a new version of the system, since we implemented many optimizations. However, they were postponed due to high employment, until one day I was informed that I had managed to agree on testing on Oracle Exadata Database Machine X2-2.
Here is a server
inspires.
The machine is so powerful that for our tests did not need to use it all. Actually, we were interested in the question of whether the application has become more productive than the previous version.
The previous version worked and is working for one of the clients , we took its parameters as the basic ones (the number of users, goods in a warehouse, warehouses, offices, cash desks, etc.). In the tested configuration, the processes repeated the processes for this client almost completely (except for some specific functions), so we took statistics on which user roles generate the main load.
Under the load, we understand the creation of documents, records in directories, search directories and the like activity. The list of roles did not come as a surprise, but it was considered necessary to carry out a formal procedure for checking intuition. We considered roles in the history of changes and using FGA (a very convenient functionality, we used to separate access in the new platform) - again, a nice addition to the main purpose of the functionality.
So, here are these roles:
Of course, there are a lot of analysts and heads of departments and departments who consider all sorts of reports all day, but these reports are calculated on a StandBy server, and do not create loads on Primary, so they decided to exclude this kind of workload from the script.
In a real situation, most of the orders are created through the site, and not by people, and at the same time there are slightly less operations. We decided that the evaluation from above was important for us, and did not separately model the script for the site.
Generated a corresponding number of warehouses, offices, goods, managers, cash desks and other facilities, built the distribution of sales of goods (if you count in pieces, the distribution almost exactly coincided with the 80/20 formula - 80% of sales made 20% of the range).
It was important to reproduce the transaction load as accurately as possible - the performance of ERP systems and their scalability is limited only by blocking. The business processes are formulated in such a way that it is impossible to refuse blocking, and, accordingly, the only way is to reduce the blocking time. Determine how we coped with this task and was the purpose of this test.
')
The work of the roles was emulated inside application servers, by launching one stream per virtual user.
To begin with, it was necessary to test the work of 4000 users - for comparison with the real profile. The Oracle Database server was launched in a virtual environment on 4 cores and 8 megabytes of memory. Application servers were running on the same server as virtual machines - 2 pieces, each with 2 cores and 2 gigabytes of memory. To our joy, the server was easy to maintain, the execution time was within one second, the CPU load was about 60-70% and 50% IO, i.e. There were practically no locks, and iron was used quite effectively.
The results shown exceeded the real client profile in all respects.
In addition, the entire machine was available and we decided to continue, increasing the load by 2000 users for each run (one run took 2-3 hours, 2 runs worked all night).
Experienced to establish that one application server (2 cores, 2 GB of memory) can withstand 3,000 users.
As a result, we reached the figure of 18,000 users. Attempts were made to launch for 20,000 users, however, the results were unstable. The instability is explained by the stochastic scenario, and, as a result, by the emergence of instability points, when hit by which the system fell into a block, from which it could not get out. It should be noted that with an increase in the number of users, we kept the user profile and other frequency characteristics - the number of offices, goods, warehouses, cash desks, and so on. Those. virtual company traded the same product through the same offices, without opening new ones! I repeat, the real estimate “from above” was important for us, so we didn’t suck out the frequency response from the finger. As soon as we have a client for 18,000 users, we will remove his profile and try to multiply the users.
This article provides a somewhat lyrical description of the testing process; a formal and complete description can be found on our website. Recently, a new version of Oracle Exadata Database Machine X5-2 has appeared, and we are going to test it. To improve performance, we’ll try to use features like in-memory database, column storage and some others.
The results will tell in our blog. Do not switch.
In this article I will tell how we tested the performance of our platform .
We planned to test the performance from the very beginning of the development of a new version of the system, since we implemented many optimizations. However, they were postponed due to high employment, until one day I was informed that I had managed to agree on testing on Oracle Exadata Database Machine X2-2.
Here is a server
inspires.The machine is so powerful that for our tests did not need to use it all. Actually, we were interested in the question of whether the application has become more productive than the previous version.
The previous version worked and is working for one of the clients , we took its parameters as the basic ones (the number of users, goods in a warehouse, warehouses, offices, cash desks, etc.). In the tested configuration, the processes repeated the processes for this client almost completely (except for some specific functions), so we took statistics on which user roles generate the main load.
Under the load, we understand the creation of documents, records in directories, search directories and the like activity. The list of roles did not come as a surprise, but it was considered necessary to carry out a formal procedure for checking intuition. We considered roles in the history of changes and using FGA (a very convenient functionality, we used to separate access in the new platform) - again, a nice addition to the main purpose of the functionality.
So, here are these roles:
- Purchasing Manager
- Storekeeper, stock taker
- Creating applications for replenishment of the warehouse
- Storekeeper Relocation Set
- Sales Manager
- Cashier
- Storekeeper, set order in stock
- Storekeeper sending and receiving inter-warehouse transportation
Of course, there are a lot of analysts and heads of departments and departments who consider all sorts of reports all day, but these reports are calculated on a StandBy server, and do not create loads on Primary, so they decided to exclude this kind of workload from the script.
In a real situation, most of the orders are created through the site, and not by people, and at the same time there are slightly less operations. We decided that the evaluation from above was important for us, and did not separately model the script for the site.
Generated a corresponding number of warehouses, offices, goods, managers, cash desks and other facilities, built the distribution of sales of goods (if you count in pieces, the distribution almost exactly coincided with the 80/20 formula - 80% of sales made 20% of the range).
It was important to reproduce the transaction load as accurately as possible - the performance of ERP systems and their scalability is limited only by blocking. The business processes are formulated in such a way that it is impossible to refuse blocking, and, accordingly, the only way is to reduce the blocking time. Determine how we coped with this task and was the purpose of this test.
')
The work of the roles was emulated inside application servers, by launching one stream per virtual user.
To begin with, it was necessary to test the work of 4000 users - for comparison with the real profile. The Oracle Database server was launched in a virtual environment on 4 cores and 8 megabytes of memory. Application servers were running on the same server as virtual machines - 2 pieces, each with 2 cores and 2 gigabytes of memory. To our joy, the server was easy to maintain, the execution time was within one second, the CPU load was about 60-70% and 50% IO, i.e. There were practically no locks, and iron was used quite effectively.
The results shown exceeded the real client profile in all respects.
In addition, the entire machine was available and we decided to continue, increasing the load by 2000 users for each run (one run took 2-3 hours, 2 runs worked all night).
Experienced to establish that one application server (2 cores, 2 GB of memory) can withstand 3,000 users.
As a result, we reached the figure of 18,000 users. Attempts were made to launch for 20,000 users, however, the results were unstable. The instability is explained by the stochastic scenario, and, as a result, by the emergence of instability points, when hit by which the system fell into a block, from which it could not get out. It should be noted that with an increase in the number of users, we kept the user profile and other frequency characteristics - the number of offices, goods, warehouses, cash desks, and so on. Those. virtual company traded the same product through the same offices, without opening new ones! I repeat, the real estimate “from above” was important for us, so we didn’t suck out the frequency response from the finger. As soon as we have a client for 18,000 users, we will remove his profile and try to multiply the users.
Finally
This article provides a somewhat lyrical description of the testing process; a formal and complete description can be found on our website. Recently, a new version of Oracle Exadata Database Machine X5-2 has appeared, and we are going to test it. To improve performance, we’ll try to use features like in-memory database, column storage and some others.
The results will tell in our blog. Do not switch.
Source: https://habr.com/ru/post/250439/
All Articles