Rally: Testing OpenStack with Tempest has become easy (easier)
Author: Andrey Kurilin
In today's world, when building cloud services, it’s not possible to limit the installation of the platform; you have to upgrade components, integrate new hardware, scale, and make many changes to the settings. But how to stay confident that after all these changes, your cloud will continue to work properly and serve the benefit of the empire to customers?
At this stage, Tempest is used for these tasks (the official OpenStack component for functional testing). Tempest tests are automatically run for each new change in almost all OpenStack projects, which reduces the likelihood of incorrect changes (broken, curves, in other words, changes that can break anything in your infrastructure).
')
However, when using Tempest, users often experience some inconvenience. Using Tempest reminds me of learning to assembler at a university. Writing and executing assembler instructions directly caused a lot of inconvenience, even if it was impossible to correct my mistakes without pain and suffering due to possible memory shifts. When we were offered the opportunity to use Turbo Pascal as an assembly language runtime environment, we were immensely happy, the process of studying and developing applications in Assembler was greatly simplified. I feel similar sensations when, instead of suffering with the Tempest configuration, I can test my cloud with a couple of simple commands.
In most cases, using Tempest is difficult and time consuming. In addition, Tempest does not support a range of load testing functionality, such as load modeling and presenting results in graphical form.
Rally can solve these problems - the OpenStack project, which is capable of much, in particular: create environments for verification, test performance, as well as benchmarking OpenStack (including using Tempest).
If we talk about Rally + Tempest, Rally automates the process of installing, configuring and running Tempest tests. Unlike Tempest, which must be installed and configured separately on each cluster, once installed, the Rally service can check a huge number of clouds — all you need to do is add information about your clusters in Rally, and you can easily switch between them later. The benchmarking mechanism Rally is able to automatically perform tests, simulating user load. The results of both verification and benchmarks are stored in the Rally database. You can view them immediately, or after a certain period, to compare how changes in the configuration of your cloud affect its operation.
Next, we will look at how to use Rally and Tempest, but before moving on to the most interesting, let's list the main reasons for integrating Rally with Tempest:
1. Most OpenStack projects use CI / CD systems that are based on Tempest to automatically test new changes. Based on this, if there is an opportunity to make life easier for developers, namely, to allow them to use Tempest more actively and conveniently, then we must do it!
2. Load testing with Tempest tests:
1. Within the framework of Tempest, many tests have already been developed, so there is no need to duplicate them in Rally. The less code duplication, the better.
2. Verification should not be limited to testing without load. Rally can simulate the load for tempest. In other words, the benchmarking mechanism in Rally can run Tempest tests with different numbers of (simulated) active users.
3. Debbuging Tempest and OpenStack. When a race condition is detected in the gates (“gates”), developers will be able to run the same test (or set of tests) under load in their environment, which will make it easier to identify the source of such errors.
You can talk a lot about the coolness and perspective of Rally, but you will never know how good Rally is until you experience it in action. So, you are probably ready to move from “lyricism” to practice. Then let's get started!
It is not surprising, but to use Rally it is necessary to start downloading it ( github repository ), and then run the install_rally.sh installation script.
After successful installation, you should add the cloud to Rally, which will be tested. This operation can be performed in two ways. One of the options for cluster initialization in Rally is using your cloud's OpenRC. For example, like this:
The second way is to describe the cloud in the json file , which should be passed to the 'rally deployment create' using the –file argument:
By running the `rally deployment list` command, you can see a list of all initialized“ deployments ”.
Using the `rally deployment check` you can check the correctness of the added deployment, as well as see which services are available in your Rally cloud:
As mentioned earlier, Rally can simultaneously work with a huge amount of clouds, therefore for each cloud its own Tempest is customized. In order to reduce the time it takes to clone the Tempest repository, Rally clones it once and then uses the local repository, customizing it for each cloud.
Tempest preparation (repository cloning and installation of a virtual environment with all dependencies) is performed by the `rally-manage tempest install` command.
Starting automatic cloud checking (in other words, cloud verification) is implemented using the `rally verify start` command, it only needs one argument: the name of the test set. If this argument is not specified, smoke tests will be executed (by default). The result of this command is similar to the result issued by Tempest.
Valid test case names are full, smoke, baremetal, compute, data_processing, identity, image, network, object_storage, orchestration, telemetry, and volume.
The results of all verifications are stored in the Rally database, so that you can view and compare them at any time. Lists of running checks will be displayed on the screen with the command `rally verify list`.
You can view the results of a specific verification with two commands: `rally verify show` and` rally verify detailed`. The latter complements the output of the first command with traceback tests that ended with an error.
In addition to installing, configuring, and running Tempest, Rally can simulate user load using Tempest tests. The results of load testing are also saved and can later be displayed.
Run the benchmark and simulate the load (from 2 to 10 users) for the following test scenario: tempest.api.identity.admin.v3.test_domains.DomainsTestXML.test_create_update_delete_domain.
After completing the test, using the command `rally task plot2html`, we will generate an HTML file with the results. Information and examples of how the input data is correctly specified in the JSON / YAML files used to define this task can be found in the Rally repository (in the rally / doc / samples / tasks directory) or on the OpenStack wiki page.
When using clouds in production, you will have to deal with constant changes. The question arises: how to check every change? Tempest is a good, but cumbersome solution. Rally, among other things, can simplify its use.
Rally provides a mechanism for load testing OpenStack, and now, thanks to integration with Tempest, the library of test scripts has become huge. Will your patch speed up any operations? Do you have a fast cloud like your competitors? Why not find out with Rally ?!
Original article in English .
In today's world, when building cloud services, it’s not possible to limit the installation of the platform; you have to upgrade components, integrate new hardware, scale, and make many changes to the settings. But how to stay confident that after all these changes, your cloud will continue to work properly and serve the benefit of the empire to customers?
At this stage, Tempest is used for these tasks (the official OpenStack component for functional testing). Tempest tests are automatically run for each new change in almost all OpenStack projects, which reduces the likelihood of incorrect changes (broken, curves, in other words, changes that can break anything in your infrastructure).
')
However, when using Tempest, users often experience some inconvenience. Using Tempest reminds me of learning to assembler at a university. Writing and executing assembler instructions directly caused a lot of inconvenience, even if it was impossible to correct my mistakes without pain and suffering due to possible memory shifts. When we were offered the opportunity to use Turbo Pascal as an assembly language runtime environment, we were immensely happy, the process of studying and developing applications in Assembler was greatly simplified. I feel similar sensations when, instead of suffering with the Tempest configuration, I can test my cloud with a couple of simple commands.
In most cases, using Tempest is difficult and time consuming. In addition, Tempest does not support a range of load testing functionality, such as load modeling and presenting results in graphical form.
Rally can solve these problems - the OpenStack project, which is capable of much, in particular: create environments for verification, test performance, as well as benchmarking OpenStack (including using Tempest).
If we talk about Rally + Tempest, Rally automates the process of installing, configuring and running Tempest tests. Unlike Tempest, which must be installed and configured separately on each cluster, once installed, the Rally service can check a huge number of clouds — all you need to do is add information about your clusters in Rally, and you can easily switch between them later. The benchmarking mechanism Rally is able to automatically perform tests, simulating user load. The results of both verification and benchmarks are stored in the Rally database. You can view them immediately, or after a certain period, to compare how changes in the configuration of your cloud affect its operation.
Next, we will look at how to use Rally and Tempest, but before moving on to the most interesting, let's list the main reasons for integrating Rally with Tempest:
1. Most OpenStack projects use CI / CD systems that are based on Tempest to automatically test new changes. Based on this, if there is an opportunity to make life easier for developers, namely, to allow them to use Tempest more actively and conveniently, then we must do it!
2. Load testing with Tempest tests:
1. Within the framework of Tempest, many tests have already been developed, so there is no need to duplicate them in Rally. The less code duplication, the better.
2. Verification should not be limited to testing without load. Rally can simulate the load for tempest. In other words, the benchmarking mechanism in Rally can run Tempest tests with different numbers of (simulated) active users.
3. Debbuging Tempest and OpenStack. When a race condition is detected in the gates (“gates”), developers will be able to run the same test (or set of tests) under load in their environment, which will make it easier to identify the source of such errors.
You can talk a lot about the coolness and perspective of Rally, but you will never know how good Rally is until you experience it in action. So, you are probably ready to move from “lyricism” to practice. Then let's get started!
Install Rally
It is not surprising, but to use Rally it is necessary to start downloading it ( github repository ), and then run the install_rally.sh installation script.
Adding your cloud to Rally
After successful installation, you should add the cloud to Rally, which will be tested. This operation can be performed in two ways. One of the options for cluster initialization in Rally is using your cloud's OpenRC. For example, like this:
The second way is to describe the cloud in the json file , which should be passed to the 'rally deployment create' using the –file argument:
By running the `rally deployment list` command, you can see a list of all initialized“ deployments ”.
Check your cloud
Using the `rally deployment check` you can check the correctness of the added deployment, as well as see which services are available in your Rally cloud:
Tempest preparation for your cloud
As mentioned earlier, Rally can simultaneously work with a huge amount of clouds, therefore for each cloud its own Tempest is customized. In order to reduce the time it takes to clone the Tempest repository, Rally clones it once and then uses the local repository, customizing it for each cloud.
Tempest preparation (repository cloning and installation of a virtual environment with all dependencies) is performed by the `rally-manage tempest install` command.
Running tempest
Starting automatic cloud checking (in other words, cloud verification) is implemented using the `rally verify start` command, it only needs one argument: the name of the test set. If this argument is not specified, smoke tests will be executed (by default). The result of this command is similar to the result issued by Tempest.
Valid test case names are full, smoke, baremetal, compute, data_processing, identity, image, network, object_storage, orchestration, telemetry, and volume.
Listing all Tempest results
The results of all verifications are stored in the Rally database, so that you can view and compare them at any time. Lists of running checks will be displayed on the screen with the command `rally verify list`.
View the results of specific verification
You can view the results of a specific verification with two commands: `rally verify show` and` rally verify detailed`. The latter complements the output of the first command with traceback tests that ended with an error.
Tempest Benchmarking
In addition to installing, configuring, and running Tempest, Rally can simulate user load using Tempest tests. The results of load testing are also saved and can later be displayed.
Run the benchmark and simulate the load (from 2 to 10 users) for the following test scenario: tempest.api.identity.admin.v3.test_domains.DomainsTestXML.test_create_update_delete_domain.
After completing the test, using the command `rally task plot2html`, we will generate an HTML file with the results. Information and examples of how the input data is correctly specified in the JSON / YAML files used to define this task can be found in the Rally repository (in the rally / doc / samples / tasks directory) or on the OpenStack wiki page.
With 2 active users:
With 6 active users:
With 10 active users:
Conclusion
When using clouds in production, you will have to deal with constant changes. The question arises: how to check every change? Tempest is a good, but cumbersome solution. Rally, among other things, can simplify its use.
Rally provides a mechanism for load testing OpenStack, and now, thanks to integration with Tempest, the library of test scripts has become huge. Will your patch speed up any operations? Do you have a fast cloud like your competitors? Why not find out with Rally ?!
Original article in English .
Source: https://habr.com/ru/post/229331/
All Articles