About research testing in Microsoft Test Manager 2012
A couple of days ago there was an article about research testing , and I would like to continue the topic with a description of one of the tools that support the process of such testing.
What, in fact, we expect from such a tool, if in research testing we have neither a scenario, nor a plan, nor clear criteria for evaluating the correctness of the system's behavior?
In my opinion, such a tool should:
Actually, the best option in this sense would be to support research testing in an integrated tool for managing requirements, tests and defects. I want to tell you about one of these tools - Microsoft Test Manager 2012 .
In 2012, the MTM version introduced support for research testing. I see the following ways to use this functionality:
For the demonstration, I will use a ready- to-boot virtual machine . There are more detailed instructions and usage scenarios.
Launch MTM, select the project and open the test plan. There may be several test plans in the project, therefore requirements, test scripts, defects, etc. stored in a shared TFS repository, and a test plan allows you to organize work with some of them.
Our test plan may be completely empty, but we will consider the scenario when we have several requirements and we added them to the plan (the “Add requirements” button is active when the root folder or subfolder, Suite is selected). When we add a requirement to the plan, a folder is created for it, into which all test scripts related to the requirement are automatically added. Here we can add other scripts (the “Add” button) or create new ones (the “New” button). Hereinafter, I use the term “Requirement”, although its specific implementation in TFS may differ depending on the template chosen. It will be “User Story” for Agile, “BacklogItem” for SCRUM, “Requirement” or some other type from the category of requirements for different templates.
')
Let's pass to research tests. To do this, switch to the “Test” tab and then open the “Do Exploratory Testing” item.
Here we have two key opportunities - to launch a research testing session (“Explore”) and to launch a research testing session for the selected requirement from the list included in the current plan (“Explore work item”).
The difference between these two options is quite simple: if we select the test requirement, then the artifacts created afterwards (test results, defects, test scripts) will be automatically tied to this requirement.
Choose the option "Explore work item". A test snap-in window will open. This tool will allow us to collect information on the progress of research testing. When we are ready, press the "Start" button, our tool is activated.
The right side of the desktop remains free to work with the tested Web or Desktop application, and in our snap-in we can leave comments during the testing, add screenshots, control the recording of actions. In the process of testing can also be recorded:
- video of what is happening on the screen
- voice comments
- actions (actions) performed in the user interface
- debug info Intellitrace
- other data collected by the Diagnostics Adapters (see below for their configuration)
So, we have a web application and we test its behavior.
The user opened the site, chose the product, added it to the cart, entered the promo code, clicked the miracle Go button and nothing happened. It seems to me that this is a defect, so I write in the comments “No reaction to the Go button” and attach a screenshot (“Add screenshot”).
At first glance, uninformative. Unfortunately, this is how testers often document defects. When they start writing a normal description, they are renamed Quality Specialists (QA Engineer).
However, let's see what modern tools can do.
Using the “Create Bug” button create a new error report:
And here we understand why MTM is needed in research testing:
The defect is practically documented itself:
- a sequence of actions is automatically recorded (steps 1-7)
- formed the description of the defect from the entered comments and screenshots
- attached collected system information
- Intellitrace log attached
- attached video recording what happened on the screen
Now the quality specialist does not need to remember what he did in order to get an error. All moves are recorded. It will be very easy for the developer to reproduce the sequence of steps and it is very difficult to reject an error with the words “and it works for me” while looking at the video.
One of the testing practices states that before correcting a found error, you should create a test that checks for the presence of this error.
The function "Create test case" allows you to automatically create a test script based on information about the actions performed.
Actions that were performed during the session are added as test steps. It remains, if necessary, to correct their description and add information about the expected result in the column “Expected Result”.
As a result, we can not only conduct research testing and document errors, but also use this tool to quickly describe test scenarios .
And if at the time of the start of the testing session we chose the requirement, then the errors found and the test script will be tied to the requirement. However, such links can always be added manually.
Information about research testing sessions can be saved even if no defects were found.
You can see them on the View Exploratory Test Sessions tab
You can open a detailed session description and view the collected data.
The same data (Test Result) is also collected by the system during other types of tests.
Let's briefly review the settings of the parameters of the information collected during testing.
In MTM, we can create and use several test configurations, including not only local profiles, but also profiles for complex test environments that include several machines, which allows, for example, to collect protocols of what happened on the application servers, on the client workstation, and on other machines included in the test environment.
To configure, switch from the Testing Center mode to the Lab Center and open the Test Settings tab
Choose one of the configurations and see its settings for one of the roles of the machines included in the test environments:
Several basic data collection adapters are available:
The last point requires clarification. The system does recognize which buttons were clicked, which links were opened, which fields were filled in, etc. Moreover, this mechanism is used in several places: in the tools for describing automatic user interface tests (Coded UI Test), in the system of partial automation of manual tests, and also during research testing.
Windows Forms 2.0, Win32, WPF, Silverlight, Web and HTML5 applications are supported via IE8-10 and some others.
There are also unsupported technologies - Flash / JAVA, SAP, Lotus Notes. A number of technologies are supported through additional extensions.
Research testing is one type of testing. And it should be used as an addition to existing unit testing procedures, automatic, functional, ergonomic, load testing, security testing, etc.
The tools used must provide a unified test execution environment, store the results and ensure connectivity with all project entities — requirements, defects, and test scenarios.
What, in fact, we expect from such a tool, if in research testing we have neither a scenario, nor a plan, nor clear criteria for evaluating the correctness of the system's behavior?
Tool requirements
In my opinion, such a tool should:
- Be integrated with a bug tracking system so that defects can be made as they are detected.
- Automatically document the defect found. This is important when the test goes not according to the scenario, but in an arbitrary sequence, which is impossible to keep in mind.
- Provide the ability to repeat a research test sequence.
- To be integrated with the requirements management system - in order, if possible, to bind detected defects to the requirements
- Be integrated with a test management system to:
- conduct all types of testing in a single environment
- create new test scenarios based on research tests
Actually, the best option in this sense would be to support research testing in an integrated tool for managing requirements, tests and defects. I want to tell you about one of these tools - Microsoft Test Manager 2012 .
In 2012, the MTM version introduced support for research testing. I see the following ways to use this functionality:
- Conducting research testing in addition to scenario tests
- Testing in the absence of testing scripts
- Rapid creation of new test scenarios through research testing sessions
For the demonstration, I will use a ready- to-boot virtual machine . There are more detailed instructions and usage scenarios.
Test plan
Launch MTM, select the project and open the test plan. There may be several test plans in the project, therefore requirements, test scripts, defects, etc. stored in a shared TFS repository, and a test plan allows you to organize work with some of them.
Our test plan may be completely empty, but we will consider the scenario when we have several requirements and we added them to the plan (the “Add requirements” button is active when the root folder or subfolder, Suite is selected). When we add a requirement to the plan, a folder is created for it, into which all test scripts related to the requirement are automatically added. Here we can add other scripts (the “Add” button) or create new ones (the “New” button). Hereinafter, I use the term “Requirement”, although its specific implementation in TFS may differ depending on the template chosen. It will be “User Story” for Agile, “BacklogItem” for SCRUM, “Requirement” or some other type from the category of requirements for different templates.
')
Research Testing Session
Let's pass to research tests. To do this, switch to the “Test” tab and then open the “Do Exploratory Testing” item.
Here we have two key opportunities - to launch a research testing session (“Explore”) and to launch a research testing session for the selected requirement from the list included in the current plan (“Explore work item”).
The difference between these two options is quite simple: if we select the test requirement, then the artifacts created afterwards (test results, defects, test scripts) will be automatically tied to this requirement.
Choose the option "Explore work item". A test snap-in window will open. This tool will allow us to collect information on the progress of research testing. When we are ready, press the "Start" button, our tool is activated.
The right side of the desktop remains free to work with the tested Web or Desktop application, and in our snap-in we can leave comments during the testing, add screenshots, control the recording of actions. In the process of testing can also be recorded:
- video of what is happening on the screen
- voice comments
- actions (actions) performed in the user interface
- debug info Intellitrace
- other data collected by the Diagnostics Adapters (see below for their configuration)
So, we have a web application and we test its behavior.
The user opened the site, chose the product, added it to the cart, entered the promo code, clicked the miracle Go button and nothing happened. It seems to me that this is a defect, so I write in the comments “No reaction to the Go button” and attach a screenshot (“Add screenshot”).
At first glance, uninformative. Unfortunately, this is how testers often document defects. When they start writing a normal description, they are renamed Quality Specialists (QA Engineer).
However, let's see what modern tools can do.
Create an error report.
Using the “Create Bug” button create a new error report:
And here we understand why MTM is needed in research testing:
The defect is practically documented itself:
- a sequence of actions is automatically recorded (steps 1-7)
- formed the description of the defect from the entered comments and screenshots
- attached collected system information
- Intellitrace log attached
- attached video recording what happened on the screen
Now the quality specialist does not need to remember what he did in order to get an error. All moves are recorded. It will be very easy for the developer to reproduce the sequence of steps and it is very difficult to reject an error with the words “and it works for me” while looking at the video.
Creating a test script
One of the testing practices states that before correcting a found error, you should create a test that checks for the presence of this error.
The function "Create test case" allows you to automatically create a test script based on information about the actions performed.
Actions that were performed during the session are added as test steps. It remains, if necessary, to correct their description and add information about the expected result in the column “Expected Result”.
As a result, we can not only conduct research testing and document errors, but also use this tool to quickly describe test scenarios .
And if at the time of the start of the testing session we chose the requirement, then the errors found and the test script will be tied to the requirement. However, such links can always be added manually.
Saved research testing sessions
Information about research testing sessions can be saved even if no defects were found.
You can see them on the View Exploratory Test Sessions tab
You can open a detailed session description and view the collected data.
The same data (Test Result) is also collected by the system during other types of tests.
Configure Information Collected
Let's briefly review the settings of the parameters of the information collected during testing.
In MTM, we can create and use several test configurations, including not only local profiles, but also profiles for complex test environments that include several machines, which allows, for example, to collect protocols of what happened on the application servers, on the client workstation, and on other machines included in the test environment.
To configure, switch from the Testing Center mode to the Lab Center and open the Test Settings tab
Choose one of the configurations and see its settings for one of the roles of the machines included in the test environments:
Several basic data collection adapters are available:
- IntelliTrace - collecting historical application execution protocol. Articles on Habré.
- EventLog - collecting events from the Windows event log
- System Information - collection of operating system configuration data
- Test Impact - collection of data about executable lines of application code during a test. Used to identify tests that could be affected by new changes in the version control system (a good topic for an article, right?)
- Video Recorder - recording what is happening on the screen in a video file. There are settings for bit rate and quality level. You can also write sound.
- Action Log - record user interface operations.
The last point requires clarification. The system does recognize which buttons were clicked, which links were opened, which fields were filled in, etc. Moreover, this mechanism is used in several places: in the tools for describing automatic user interface tests (Coded UI Test), in the system of partial automation of manual tests, and also during research testing.
Windows Forms 2.0, Win32, WPF, Silverlight, Web and HTML5 applications are supported via IE8-10 and some others.
There are also unsupported technologies - Flash / JAVA, SAP, Lotus Notes. A number of technologies are supported through additional extensions.
Conclusion
Research testing is one type of testing. And it should be used as an addition to existing unit testing procedures, automatic, functional, ergonomic, load testing, security testing, etc.
The tools used must provide a unified test execution environment, store the results and ensure connectivity with all project entities — requirements, defects, and test scenarios.
Source: https://habr.com/ru/post/148598/
All Articles