Unified test runner and analysis of test data
Jan Drugal has been creating development tools in Unity for 2 years. Recently, his team has grown significantly, and with it the test suites, the number of unstable, slow tests and unrecoverable errors. We publish a translation of an article on how Yang and his team solve these problems.
We in Unity have a set of the directions of testing (fig. 1) and sets of tests:
')
• Runtime tests allow you to test the Unity runtime public API on all supported platforms.
• Integration tests cover what is easy to miss when testing runtime, such as Unity editor functionality or integration with components such as Cache Server and Bug Reporter.
• Native C ++ tests are designed to directly test C ++ code without using scripts.
• Graphic tests are responsible for rendering functions and allow you to compare the resulting images with the reference.
• And many others (performance tests, downloads, IMGUI, etc.).
Figure 1. Test directions in Unity
At the top level, tests are grouped in several ways. Further separation depends on the platform, frequency, execution time and other criteria. This structure gives us a huge number of control points. But we will talk about them later.
To simplify this system, about a year ago we began the development of a universal test performer Unified Test Runner (UTR). In fact, this is a single entry point (Fig. 2), which provides a universal interface for all performers and test areas. Thus, any set of tests can be run directly from the command line.
Figure 2. Unified Test Runner as an entry point for all tests
All test artifacts are stored in one place and grouped according to the same rules. In addition, the UTR provides other options:
• filters can be applied to all tests: testfilter = TestName;
• identical test reports are generated for all sets.
Initially, we used UTR for local tests, but then decided to apply to our build environment. The idea was to run the same tests from local computers and when building releases. In other words, so that in case of an error, it can be recreated locally.
Gradually, the UTR has become a single entry point for all tests in Unity. And then we decided to use it for another task - to collect test data from local computers and from the build environment. At the end of each test, the UTR sends the results to a special web service. This is how our solution for analyzing test data, called Hoarder, appeared. Its task is to collect data on test execution, save them, provide access to them and display cumulative testing statistics with the possibility of in-depth analysis of the results of individual tests (Fig. 3).
Figure 3. Assembly agents and users upload data to the Hoarder web service, where they are processed by the analytic application.
In the process of data processing, we discovered a lot of interesting things and as a result made some important decisions. Which ones - read the next article.
We in Unity have a set of the directions of testing (fig. 1) and sets of tests:
')
• Runtime tests allow you to test the Unity runtime public API on all supported platforms.
• Integration tests cover what is easy to miss when testing runtime, such as Unity editor functionality or integration with components such as Cache Server and Bug Reporter.
• Native C ++ tests are designed to directly test C ++ code without using scripts.
• Graphic tests are responsible for rendering functions and allow you to compare the resulting images with the reference.
• And many others (performance tests, downloads, IMGUI, etc.).
Figure 1. Test directions in Unity
At the top level, tests are grouped in several ways. Further separation depends on the platform, frequency, execution time and other criteria. This structure gives us a huge number of control points. But we will talk about them later.
To simplify this system, about a year ago we began the development of a universal test performer Unified Test Runner (UTR). In fact, this is a single entry point (Fig. 2), which provides a universal interface for all performers and test areas. Thus, any set of tests can be run directly from the command line.
Figure 2. Unified Test Runner as an entry point for all tests
All test artifacts are stored in one place and grouped according to the same rules. In addition, the UTR provides other options:
• filters can be applied to all tests: testfilter = TestName;
• identical test reports are generated for all sets.
Initially, we used UTR for local tests, but then decided to apply to our build environment. The idea was to run the same tests from local computers and when building releases. In other words, so that in case of an error, it can be recreated locally.
Gradually, the UTR has become a single entry point for all tests in Unity. And then we decided to use it for another task - to collect test data from local computers and from the build environment. At the end of each test, the UTR sends the results to a special web service. This is how our solution for analyzing test data, called Hoarder, appeared. Its task is to collect data on test execution, save them, provide access to them and display cumulative testing statistics with the possibility of in-depth analysis of the results of individual tests (Fig. 3).
Figure 3. Assembly agents and users upload data to the Hoarder web service, where they are processed by the analytic application.
In the process of data processing, we discovered a lot of interesting things and as a result made some important decisions. Which ones - read the next article.
Source: https://habr.com/ru/post/276531/
All Articles