Parallel execution of tests based on context using Visual Studio 2015 Update 1
Visual Studio 2015 Update 1 includes a number of improvements and fixes. In this publication, we will focus on two improvements that can significantly improve overall software development performance.
The test core of Visual Studio 2015 Update 1 is able to execute test builds in parallel, which significantly improves the testing performance. By default, this mode is disabled, so you need to enable it.
The system runs parallel test builds on the number of cores you specify - up to all of the cores of your computer. Of course, with a single test project, this does not give any advantages. This solution is intended for projects of significant scale. It does not depend on the type of test platform used and works with MSTest, NUnit 2 , NUnit 3 and XUnit . This mechanism is completely independent of the underlying platform supporting concurrency.
In order to enable this feature, you need to use the runsettings file that you want to include in Visual Studio. A simple way to create a valid runsettings file is to use one of the templates . Please note that you must use templates of version 3.1 or higher. After installing this extension element, you will have three templates; If you need parallelism, then use the Parallel template.
')
Do the following:
select Solution, right-click and select Add / New item;
on the screen you will see a list similar to the one below;
select Parallel; You will see an instance of it in the Solution Items tab of your solution.
That's all you need to enable parallel test execution!
Do not forget to activate it in the Visual Studio Test menu, for which follow these steps:
(1) Go to the Select Test Settings file from the main menu Test / Test settings.
(2) Then select the desired Runsettings file, in this example I called it Parallel2.runsettings.
(3) Make sure that it is selected and checked in the same menu.
Options:
The contents of the runsettings file look like this:
Here we can change the value of only one element - MaxCpuCount, which determines the number of processes running in parallel. A value of 0 indicates that the maximum number of processes will be executed, it is limited only by the number of computer cores. A value of 1 indicates that the processes will be executed sequentially and only one process will be active (this value is set by default). Any other values ​​determine the maximum number of processes executed in parallel (or the number of used cores of your computer).
Why don't we turn on code checking?
You may have noticed that the description of the Parallel file indicates that the code coverage check is not included. This means that when activating its settings, the code coverage check will not be performed. The other two patterns are arranged in the same way, since code coverage does not provide any advantages.
Activating a code coverage check degrades performance because the code coverage check is performed after each test and significantly increases the total testing time and does not provide any useful results.
Disabling code coverage does not mean that you cannot execute it from Team Explorer. On the contrary, you can do it! Oddly enough, the results of this test for code coverage are not used in the Test Explorer. In the other two templates, only the background execution of the check for code coverage is activated, the results of which are saved in a file located in the Test Results folder.
This file does not give you any advantages in Visual Studio, but is useful if you want to use these checks to cover code in another program, for example, NDepend .
One of the advantages of this section is that it not only includes background scanning for code coverage, but also configures filtering of verification data for code coverage that may be useful to you. These filters also function when performing code coverage analysis from selected tests (“Analyze code coverage from selected tests”) in the Test Explorer.
The article referenced above contains a set of options for checking for code coverage. Instead, use the CompleteRunSettings template, which includes both these and other settings. The XML provided in this article is irrelevant.
In the CompleteRunsettings template, the value of the MaxCpuCount parameter is 0; This means that the number of processes executed in parallel will be the maximum possible.
The CoverageNoParallel template is almost identical to the previous one: it contains the same fields and values, with the exception of MaxCpuCount, whose value is 1. This means that tests will be performed sequentially.
An article about a more correct setup of unit tests has been published on MSDN , however this publication does not contain as detailed information as the above link. However, this article describes all the fields.
I created a simple project for parallel execution of tests. It consists of four test projects, each of which contains one test run for 5 seconds. You can download the project source code from here and perform these tests yourself.
(1) Perform tests without any settings. In this case, the tests are performed sequentially, and the execution time is 26 seconds. Each test is performed for 5 seconds, plus a delay of 6 seconds.
(2) Running tests using CodeCoverageNoParallel is the same as using the old runsettings file. In addition to tests, a check is performed to cover the code, but without concurrency. The total time rises to 31 seconds.
(3) When using a full set of settings, including CodeCoverage and Parallel, test execution time is reduced to 18 seconds.
(4) When using only Parallel without checking for code coverage, test execution time is reduced to 12 seconds.
Thus, option (4) more than doubles productivity compared to option (1). This is a good result!
The computer on which these tests were performed was equipped with 8 cores, but other programs did not allow all these cores to be used for testing. The increase in test performance is not a multiple of the number of computer cores, but a double increase is a good and very tangible achievement in everyday work.
Parallel execution on the build server
Does the described scheme work on the build server? Yes: In the assembly definition, you can specify which runsettingfile file will be used when executing the build server. But ask yourself another question: do you really need it? The answer depends on how many build servers you use, what their settings are and how many checks and fixations you perform, i.e. from the load on the build servers. Usually, one build agent is executed on each computer core. If during the day you use a large number of assembly definitions and constantly perform fixations and checks, then the load on the assembly agents will be great and additional parallel execution of tests will not cause a significant effect, since the tests will “deprive” other assemblies of access to computer cores. In addition, there may be additional costs that neutralize productivity growth.
On the other hand, if the computer used for the assembly is not heavily loaded and the number of fixations is small, why not increase its performance by running tests in parallel?
There are other ways to execute tests in parallel, one of which is described in this article .
Running tests with context is one of the new features in Update 1. The new Test Explorer only tests assemblies in whose code it detects changes.
This means that if you work with a code fragment in a specific module of your system, the code is used in several modules. When you make changes in one of the modules, only the code you modified participates in the assembly. This is the usual incremental build that is used by default. The novelty lies in the fact that during testing only tests are performed that check the modified code. Obviously, this significantly improves the testing performance of large systems.
In order to use this function, you need to activate the “Run tests after build” item.
This can be done in Test Explorer (1) or in the main menu Test / Test Settings / Run Tests after Build (Test settings / Run tests after build) (2).
The results are shown in the following screen shot:
(1) Activate "Run tests after build" ("Run tests after build").
(2) Make changes to the code
(3) Select Build or use the appropriate keyboard shortcut.
(4) Only the necessary tests will be performed, and the remaining tests will be displayed in muted green. The previous time they were executed and displayed in green, but this time they are not executed, because the code they are checking has not changed.
This process will help you significantly improve performance when testing large projects and speed up the transition to editing code after testing the next build. Agree that this is an important advantage for the developer!
Parallel test execution
The test core of Visual Studio 2015 Update 1 is able to execute test builds in parallel, which significantly improves the testing performance. By default, this mode is disabled, so you need to enable it.
The system runs parallel test builds on the number of cores you specify - up to all of the cores of your computer. Of course, with a single test project, this does not give any advantages. This solution is intended for projects of significant scale. It does not depend on the type of test platform used and works with MSTest, NUnit 2 , NUnit 3 and XUnit . This mechanism is completely independent of the underlying platform supporting concurrency.
In order to enable this feature, you need to use the runsettings file that you want to include in Visual Studio. A simple way to create a valid runsettings file is to use one of the templates . Please note that you must use templates of version 3.1 or higher. After installing this extension element, you will have three templates; If you need parallelism, then use the Parallel template.
')
Do the following:
select Solution, right-click and select Add / New item;
on the screen you will see a list similar to the one below;
select Parallel; You will see an instance of it in the Solution Items tab of your solution.
That's all you need to enable parallel test execution!
Do not forget to activate it in the Visual Studio Test menu, for which follow these steps:
(1) Go to the Select Test Settings file from the main menu Test / Test settings.
(2) Then select the desired Runsettings file, in this example I called it Parallel2.runsettings.
(3) Make sure that it is selected and checked in the same menu.
Options:
The contents of the runsettings file look like this:
Here we can change the value of only one element - MaxCpuCount, which determines the number of processes running in parallel. A value of 0 indicates that the maximum number of processes will be executed, it is limited only by the number of computer cores. A value of 1 indicates that the processes will be executed sequentially and only one process will be active (this value is set by default). Any other values ​​determine the maximum number of processes executed in parallel (or the number of used cores of your computer).
Why don't we turn on code checking?
You may have noticed that the description of the Parallel file indicates that the code coverage check is not included. This means that when activating its settings, the code coverage check will not be performed. The other two patterns are arranged in the same way, since code coverage does not provide any advantages.
Activating a code coverage check degrades performance because the code coverage check is performed after each test and significantly increases the total testing time and does not provide any useful results.
Disabling code coverage does not mean that you cannot execute it from Team Explorer. On the contrary, you can do it! Oddly enough, the results of this test for code coverage are not used in the Test Explorer. In the other two templates, only the background execution of the check for code coverage is activated, the results of which are saved in a file located in the Test Results folder.
This file does not give you any advantages in Visual Studio, but is useful if you want to use these checks to cover code in another program, for example, NDepend .
One of the advantages of this section is that it not only includes background scanning for code coverage, but also configures filtering of verification data for code coverage that may be useful to you. These filters also function when performing code coverage analysis from selected tests (“Analyze code coverage from selected tests”) in the Test Explorer.
The article referenced above contains a set of options for checking for code coverage. Instead, use the CompleteRunSettings template, which includes both these and other settings. The XML provided in this article is irrelevant.
In the CompleteRunsettings template, the value of the MaxCpuCount parameter is 0; This means that the number of processes executed in parallel will be the maximum possible.
The CoverageNoParallel template is almost identical to the previous one: it contains the same fields and values, with the exception of MaxCpuCount, whose value is 1. This means that tests will be performed sequentially.
An article about a more correct setup of unit tests has been published on MSDN , however this publication does not contain as detailed information as the above link. However, this article describes all the fields.
Test project for parallel execution
I created a simple project for parallel execution of tests. It consists of four test projects, each of which contains one test run for 5 seconds. You can download the project source code from here and perform these tests yourself.
(1) Perform tests without any settings. In this case, the tests are performed sequentially, and the execution time is 26 seconds. Each test is performed for 5 seconds, plus a delay of 6 seconds.
(2) Running tests using CodeCoverageNoParallel is the same as using the old runsettings file. In addition to tests, a check is performed to cover the code, but without concurrency. The total time rises to 31 seconds.
(3) When using a full set of settings, including CodeCoverage and Parallel, test execution time is reduced to 18 seconds.
(4) When using only Parallel without checking for code coverage, test execution time is reduced to 12 seconds.
Thus, option (4) more than doubles productivity compared to option (1). This is a good result!
The computer on which these tests were performed was equipped with 8 cores, but other programs did not allow all these cores to be used for testing. The increase in test performance is not a multiple of the number of computer cores, but a double increase is a good and very tangible achievement in everyday work.
Parallel execution on the build server
Does the described scheme work on the build server? Yes: In the assembly definition, you can specify which runsettingfile file will be used when executing the build server. But ask yourself another question: do you really need it? The answer depends on how many build servers you use, what their settings are and how many checks and fixations you perform, i.e. from the load on the build servers. Usually, one build agent is executed on each computer core. If during the day you use a large number of assembly definitions and constantly perform fixations and checks, then the load on the assembly agents will be great and additional parallel execution of tests will not cause a significant effect, since the tests will “deprive” other assemblies of access to computer cores. In addition, there may be additional costs that neutralize productivity growth.
On the other hand, if the computer used for the assembly is not heavily loaded and the number of fixations is small, why not increase its performance by running tests in parallel?
There are other ways to execute tests in parallel, one of which is described in this article .
Test execution with context
Running tests with context is one of the new features in Update 1. The new Test Explorer only tests assemblies in whose code it detects changes.
This means that if you work with a code fragment in a specific module of your system, the code is used in several modules. When you make changes in one of the modules, only the code you modified participates in the assembly. This is the usual incremental build that is used by default. The novelty lies in the fact that during testing only tests are performed that check the modified code. Obviously, this significantly improves the testing performance of large systems.
In order to use this function, you need to activate the “Run tests after build” item.
This can be done in Test Explorer (1) or in the main menu Test / Test Settings / Run Tests after Build (Test settings / Run tests after build) (2).
The results are shown in the following screen shot:
(1) Activate "Run tests after build" ("Run tests after build").
(2) Make changes to the code
(3) Select Build or use the appropriate keyboard shortcut.
(4) Only the necessary tests will be performed, and the remaining tests will be displayed in muted green. The previous time they were executed and displayed in green, but this time they are not executed, because the code they are checking has not changed.
This process will help you significantly improve performance when testing large projects and speed up the transition to editing code after testing the next build. Agree that this is an important advantage for the developer!
Source: https://habr.com/ru/post/277583/
All Articles