Test Case Management Tool: how to make the right choice and not regret it

The head of the QA-division of Redmadrobot Ilya Gorshkov tells how he chose the tools for working with test cases.

In the process of working on any complex software system, a large number of project documentation is created. In most cases, its structure is almost the same - these are the requirements for a system at various levels, a description of its architecture, program code, QA-team documentation, project plans, reports, etc.

Today I will talk about the artifacts of the Redmadrobot QA team with whom we work during projects. These are test strategies, test plans, test runs and test cases, traceability matrix, bug reports, performance and quality metrics, various test results reports, etc. All of them have a certain hierarchy, are created in a certain sequence and require periodic updating.
')
There are several ways to solve the problem of creating a unified system for working with all QA artifacts. For example, select Excel and Google Docs, conduct everything in a bug tracker (using Test Case as the Issue Type) or use one of the test case management tools integrated with the company's bug tracker. We at Redmadrobot have chosen the third option, based on the specifics of the projects, the volume of tasks, the types of QA documentation and the types of testing we use, the number of test cases developed and executed by manual and various projects in operation simultaneously.

The next step for us was the selection of a suitable test case management tool. It is very important to approach this choice as responsibly as possible, since the cost of an error when choosing such a tool for a company is quite high. The QA-team at Redmadrobot went to the final decision in several stages and began with the development of criteria that the tool we need to meet.

We introduced the following criteria:

• company bug tracker integration (JIRA)
• storage and the ability to edit test cases, including the import of test cases created by us earlier
• easy tracking of test case coverage
• convenient formation of Test Runs / Test Suites and user-friendly interface
• storing all information on Test Development and Test Execution in one place and creating a single space for the QA-team to work
• ability to create a traceability matrix
• the ability to distribute tasks and assign them to specific engineers
• ease of receiving reports, metrics, statistics
• ease of installation, implementation, support

What to choose from:

After working out the criteria, we considered several of the most popular tools on the market that matched our expectations. Part of the bodies was discarded upon closer examination, for the remaining ones we issued an evaluation-license and continued the study. As a result, the list was reduced to three cases where pilot projects were conducted:

1. Zephyr
2. TestRail
3. Meliora

At the end of the pilots, there was a clear understanding of the possibilities and convenience of work in each of them, the features of each tool became clear, their applicability and usability. Here are the high-level characteristics of each tool:

Zephyr (1 user - $ 30) - the distinctive feature of this tool is that it is an Atlassian product, which means it must be perfectly compatible with JIRA, but the problem is that it is valid for the JIRA Server, and we are still using OnDemand, which at the pilot stage had a lot of problems. In addition to this drawback, Zephyr is inconvenient to use: adding test cases for a long time and not so simple, a lot of unnecessary actions when creating plans and runs.

Meliora (1 user - $ 25) - you also need to switch to JIRA Server + by itself. Meliora is a rather cumbersome tool that artificially complicates most simple tasks, including its own bug tracker.

TestRail (1 user - $ 20) - simple and convenient tool. The main plus is that customization is possible in almost everything, and any actions are intuitive. There is the ability to import / export test cases.

After analyzing the results of the pilots and feedback for each tula, we decided to concentrate on TestRail, which includes and allows:

• Creating / storing / editing test scripts, managing test plans, running test cycles, recording test results.
• A clear description of the test scenarios, their review, correlation with the requirements, division into areas - all this allows us to evaluate both the completeness of the test coverage of the functional and the necessary material for the entire project team.
• Creating reports on completely different criteria: Defect Summary, Comparison for Cases, test results for projects / components / milestones, etc.
• Opportunities for complete customization of the “working dashboard”, as well as conveniently getting the status of a QA-team for different periods of time (helps when creating a weekly / monthly QA report).
• Easy integration with JIRA.
• Reasonable price.
• Great support.
• Easy and convenient way to import tests from Excel.

The ability to easily import previously created test cases was a very important criterion for us. When we first started developing the QA practice in Redmadrobot, there was no talk about the Test Case Management Tool, and used Excel to work with tests. But we immediately approached the issue of using Excel with the groundwork for the future and developed a clear test format, realizing that after a while we will “feed” them in some form. When we launched TestRail into commercial operation, we were able to export “as is” several thousand test cases, which greatly reduced efforts to implement the tool.

Below, I will examine in detail the capabilities of TestRail for various QA activities:

Test Development:

• creating test plans / suits / test cases;
• convenient storage, updating and organization;
• import / export ability to edit;
• easily customizable set of any test attributes;
• Requirements Traceability.

Test Execution:

• milestones (according to the quality criteria in the company);
• compiling test runs;
• creating defects from test runs;
• distribution of tasks;
• convenient integration with JIRA.

Test Management:

• activity management;
• distribution of roles;
• assignment of tasks and execution control.

Reporting:

• testing progress;
• test results in the form of ready-made reports;
• project statistics;
• variety of report options;
• team performance metrics.

What is the result:

We made the final choice in August. In September, a large part of QA activities for projects were transferred to TestRail. We continue to transfer there all new projects. For all the time, I have never regretted my choice. We collected several metrics that confirmed our assumptions about the correctness of choice and the positive effect of the introduction. We were able to quickly teach engineers to effectively use the tools. In the near future, we will finish work on the implementation of Requirements Traceability for all projects and will develop the use of TestRail further.