How we automate testing with release management - Part 2

In the first part of this article, I fully described the test automation setup process using RM. In the second (and last) part of the article I will talk about some design solutions, problems that we encountered when setting up the system, and ways to overcome these problems.

One or more agent pools?

Question : how to redirect the release definition to the appropriate agent computer, i.e. on a computer with the resources needed by this release definition?
Solution : when we started to create release definitions, it became obvious that it was necessary to direct each release definition to the appropriate agent, since the requirements of all test suites were different. At first, we created a separate agent pool for each release definition, but managing a large number of agent pools turned out to be difficult. Ultimately, we followed the advice of Chris Patterson from the build group and developed a system that used a single agent pool called RMAgentPool. All agents of this pool differed from each other by the user's capabilities. Each release definition and assembly definition is now performed by the appropriate agent using the RmCdpCapability feature. For example, a computer that is prepared for RM.CDP.TfsOnPrem has the ability RmCdpCapability = TfsOnPrem:


RM.CDP.TfsOnPrem RD directs its execution to this agent using the requirement RmCdpCapability = TfsOnPrem:

JIT debugging

Question : how to debug intermittent crashes when log files contain insufficient data? By the time developers find a problem, the next test run removes the “error state” from the test computer.
Solution : when the test suite behavior is unpredictable, we include the “Pause Agent on test failure” task in the issue definition:

')
This task suspends release when an error occurs in an unpredictable test. In this situation, the developer can remotely connect to the agent's computer, the state of which will be exactly the same as his state at the time of the error. Please note that this is only possible if the value of the environment wait time is 0 (there is no waiting time).
The Pause task takes the following arguments (highlighted in the illustration above):
-AlternateCredentialsUserName $ (release.alternateusername) -AlternateCredentialsPassword $ (release.alternatepassword) -ReleaseId $ (Release.ReleaseId) -TaskName $ (TaskToDebugName) -NumberOfSeconds $ (TaskToDebugTeteb
The key argument here is $ (TaskToDebugName): it has the value "Run RMCDPOnPrem tests" for RM.CDP.TfsOnPrem, i.e. the name of the task in which intermittent failures occur. All RM.CDP. * Release definitions include the same task with the same arguments, but the value of $ (TaskToDebugName) in each release definition corresponds to the task whose alternate failures we want to debug.
The source code for this task can be found here (instead of “YourAccount”, you should substitute the account name and use it).

How to perform tests in administrator mode on the agent's computer?

Question : even if the agent is running with administrator rights, it performs the task in a mode other than administrator mode. This makes it impossible to perform a task that must be performed in administrator mode (for example, installing a service on an agent computer).
Solution : we solved this problem using the “Powershell on Target Machines” task (“Powershell on target computers”) and remote access to the current agent's computer as an administrator. For example, we set the TFS service to RM.CDP.RmEqTfs as follows:

Tests that can not reuse the task VsTest

Question : some tests were created relatively long ago (we will call them “obsolete”) and are not compatible with the VsTest task. How in such situations to take advantage of the preparation of reports that provide tests that are compatible with the task VsTest?
Solution : to solve this problem, we have taken the following actions:

1. We performed obsolete tests using a batch script or powershell task (whichever is more convenient) and determined the location of the output log file.
2. We added the VsTest task, which analyzed the output log file detected in the previous step and determined whether the test was successful.

Our definition of the RM.CDP.DevFabricUpgrade release is as follows: the main test is performed using the powershell script RunAOConfigTest.ps1.


It then runs the VsTest task, which analyzes the log file and checks if there are any errors in it.
Note: until recently, the test code RM.CDP.DevFabricUpgrade, compatible with VsTest, simply placed the text “Expected True, but found False” (“True expected, but False detected”) into the log file, which seriously complicated debugging. The developers were forced to open the output log file RunAOConfigTest.ps1, which in itself was time consuming because the file was very large, and look for a real error in it, as shown in the screenshot below:


We recently changed this code to more thoroughly analyze the output log file RunAOConfigTest.ps1. Now, when testing fails, more detailed error information is displayed:

How to eliminate bottlenecks when performing tests?

Question : the implementation of some definitions of release takes a very long time, which leads to the appearance of the queue and significantly increases the processing time. What can be improved in this situation?
Solution : since most of our tests are designed for service deployment and test execution performed on an agent PC, we can simply install additional hardware (with the right RmCdpCapability capability) on the problematic release definitions.

How to perform user interface tests?

Question : how to perform user interface tests on the agent's computer?
Solution : a quick and primitive way to solve this problem is to start the agent interactively (and not in service mode), because in service mode the agent cannot interact with the desktop computer. The disadvantage of this method is that when the agent runs in interactive mode, the automatic update function is not supported — to update the agent, you must log in to the agent’s computer and restart the agent.
A more automated approach is to start the agent in service mode and use the tasks of the “Visual Studio Test Agent Deployment” (“Deployment of the test agent Visual Studio”) and “Visual Studio Test using Test Agent” (“Run a Visual Studio test using a test agent "), respectively. We use a method similar to the “Powershell on Target Machines” task described above (“Powershell on Target Computers”), in which we log in remotely to the current agent's computer with administrator rights. These tasks are performed on the agent by a testing service configured to interact with the desktop computer.
In order to use this method, we first create a group of suitable computers in the Test section:


Then we use this group of computers in the two tasks of the VS test agent mentioned above. In the VsTest Agent Deployment task, you need to select the "Interactive Process":


At the end, the “Vs Test using Test Agent” task (“Test Vs using a test agent”) performs tests using our group of computers, using the same filters as in the standard VsTest task:


It is worth noting that in the near future this process will become more consistent; we will no longer need to create computer groups for test agent tasks and it will be sufficient to specify the computer name in the $ parameter (Agent.MachineName).

What variables are available at runtime?

Question : What variables are available at runtime?
Solution : Variables that are available at runtime and that can be used in the release definition are listed at the beginning of the following logs:

Is it possible to scale RM according to the needs of our group?

Question : As we approach the end of the sprint, the number of checks in our group increases exponentially, and we have doubts that RMO will cope with these loads. Performing 15 checks per day at the release stage is common practice and implies 15 * 9 = ~ 135 deployments per day (since each check activates nine release definitions).
Solution : At first, the RMO service showed some instability under high loads, but in the second half of 2015 we eliminated it using a series of stress tests. Now RMO easily handles the tasks of our group. The screenshot below shows about 20 issues of RM.CDP.DevFabircUpgrade, made in the last 24 hours, which corresponds to 20 * 9 = 180 releases of our group for the same period of time.

Conclusion

I hope that this publication will help you solve most of the problems encountered in test automation with RM.