Reading a great article is difficult ... Testing a complex product is easy
In today's IT industry, many still wonder: “why do we need a separate test department?”.
What is Petersburg autumn? Rain, Rosenbaum, melancholy ... Is it all ?! No, this is also a new academic year, which in many regions of our country began with Dnevnik.ru. Over the summer, we managed to achieve some success: the number of project users exceeded 7,000,000, we became the only Russian company on the list of World Economic Forum Technology Pioneers 2014 , and the Republic of North Ossetia-Alania completely switched to electronic journals from Diary. ru On this lyrical introduction will end and move on to the essence of this post.
In today's IT industry, many still wonder: “why do we need a separate test department?”. As one of the participants of the test team of our project, I declare responsibly: this is an erroneous opinion. Organizing a QA department is not easy, but worth it.
In this case, you need to clearly understand for yourself:
- what is considered a quality criterion?
- what metrics to use?
- with what tools to test?
- how to build interaction with developers / analysts, etc.
')
By and large, reinvent the wheel is not worth it - these practices are described on millions of Internet pages, successfully used in many corporations whose experience can be learned. However, we should not forget the simple truth - everywhere there is a specificity. For example, in Dnevnik.ru, these are strongly interconnected components and CI (Continuous Integration), and simply a high level of integration of related areas. How best to test in this case?
Let's start with a simple one: any information can never 100% automatically “pop up” in your (undoubtedly very clever) head. QA-shnik, first of all, it is necessary to memorize and later run test scenarios that are checked from release to release - “didn’t it break anything?”. As we all know - regression testing. But now it is not about that, about the types - welcome to protesting.ru (a good, by the way, resource).
So, for writing and storing test cases (“Test case is an artifact describing a set of steps, specific conditions and parameters necessary to check the implementation of the test function or its part”) you can use a lot of tools: Excel, notepad, google-docs , sheets of paper, after all. But is it efficient and convenient? For one person - maybe. For the team, where the status of the project “you should have known already yesterday” is definitely not.
For a long time there have been many heavy Test Management programs that allow you to fully manage the test process: working with test plans, test cycles, requisitions, test cases, etc. I worked with different tulas - paid, free, self-hand-written ... For many years I was convinced that in this matter it is better not to skimp on money: either get a quality tool, or hire a team and write “perfect for a company, with the requirements that You need".
So, firstly: it became clear that the existing checklists do not cope with the success of testing. They are not enough, there is little information in them, they are not transparent, it’s difficult to provide results, they are difficult to maintain, there is not enough time to write new ones. Secondly: in the current situation, we needed a tool that was rather easy to understand and would allow us to effectively manage the test management scenario: “requirements -> test cases -> review -> test suite -> test cycle -> results”.
- convenient user interface
- ease of installation, implementation and support
- convenient writing and storage of test scripts
- connection with JIRA
- effective management of the following scenario: "requirements -> test cases -> review -> test suite -> test cycle -> results"
- convenient selection of test cases from different areas to run in one test run
- ease of obtaining transparent reports on the launch of test suites
- the possibility in the future to associate with auto-testing and launch of written auto-tests
- Microsoft Test Lab ( how to in Russian )
- Zephyr
- Test Link
- Testia Tarantula
- Test Rail
PS: warning questions - we didn’t consider HP Quality Center , because 1) it’s unreasonably expensive 2) it’s not very “friendly” either, especially to those who have never worked with test-cases, and writing yours is resource costs and lack of time ...
I will try to meet a couple of key points:
"Well, very cool thing, everything," isn't it? However, if you have never encountered its installation and implementation, you should not have optimistic forecasts. To put it mildly, if you have at your disposal a couple of free months - welcome. Install, configure, there is a nice bonus - the full how to in Russian. For us, the main advantages were the license and easy setup of communication with auto tests.
Total: there was no time to introduce and teach it to use.
A nice, modern, colorful tool integrated with JIRA. And yet - have you seen how to start test cycles there and write test cases? For editing - a separate window, adding steps and results - only line by line and numbered. Many unnecessary actions to "create-save" script.
Total: If we have one project for half a year before us, gently and leisurely drive in test cases, in the end everything will be quite nice and transparent. The option also did not suit us.
Very simple tool that allows you to do the most basic work of the tester - test cases + test cycles. And that's all. Design - for C grade, convenience - too. It seems to me that Test Link can be put at home in order to just learn how to insert test cases and pass test runs with Passed / Blocked / Failed. The option, of course, is not bad, especially free. Disadvantages: non-obviousness with auto-tests, imperfect writing of test cases.
Total: Okay, let's leave for consideration.
Free Finnish toolkit, which is easy to implement in the QA team. Pros - quite well allows you to manage the test process: test cases, test wounds, reports. True structure: test case -> test suite. There are quite funny moments, such as “tags” in test cases, by which you can find the set you need (for example, you need negative test cases only for a specific functionality that affects this and this -> initially set in all scenarios the correct set of tags, from here it is easy to make a sample in a separate test run). Convenience is not bad, for auto tests you can customize ... Cons: reporting is lame, customization is problematic, it is not connected with JIRA.
Total: Leave as one of the options for consideration, why not?
In my opinion, it is one of the easiest, but at the same time very large-scale existing bodies. The main advantage is that customization is possible in almost everything. Ranging from design to logic. Direct integration with JIRA. Effective recording of test cases and splitting them into the necessary suites. As one of the drawbacks - until recently, Test Rail had a poorly implemented reporting system: poor results on test injuries, lack of customization of reports, etc. However, Gurock & Co took up this business and by the 3rd version of the tool the guys did the work on the bugs. Reporting started playing with new colors: if you want, a report on mylston, if you want, only for defects, if you want, we will set up a regular e-mail with sending a report on testing a strictly defined functionality, etc. Well done! The result - this tool met all (!) Of my requirements, listed a little higher.
• This is an effective management of test cases, plans and test cycles (creating / storing / editing test scenarios, managing test plans, launching test cycles, recording test results)
• This is a significant improvement in the quality of testing (a clear description of test scenarios, their review, correlation with the requirements, division into areas - all this allows us to evaluate both the completeness of the coverage of the functional tests and the necessary material for the entire project team)
• This is the receipt of test results in real time (creating reports on completely different criteria: Defect Summary, Comparison for Cases, test results for projects / components / milestown, etc.)
• This is an easy organization and convenient tracking of the testing department’s load (opportunities for full customization of the “working dashboard”, as well as convenient receipt of the work status of the QA department)
1. Put the TestRail on the Windows Server 2008 R2 virtual machine with the MSSQL database
2. Set up AD authentication
3. Set up a full connection with JIRA
To begin with, each logged in TestRail user needs to configure a direct access to JIRA. To do this, we need a couple of custom variables jira_user + jira_password (welcome to: docs.gurock.com/testrail-integration/defects-plugins-variables ). After creating them in the JIRA_Rest plugin (Administration -> Integration -> JIRA_Rest), set the following:
[connection]
address = http: // jira / ***
user =% jira_user%
password =% jira_password%
Hooray. Now each user has the opportunity in My Settings to set login / password from JIRA.
Further, in JIRA_Rest we register both the connection directly from the established JIRA, and subsequently the connections of all the existing JIRA fields: whether it is the main field or custom (welcome to: docs.gurock.com/testrail-integration/tools-jira-rest ).
Direct connection with JIRA (example):
Reference View Url: your-server / jira / browse /% id%
Reference Add Url: your-server / jira / secure / CreateIssue ! Default.jspa
Links to main fields (example):
[push.fields]
summary = on
project = on
issuetype = on
component = on
assignee = on
priority = on
affects_version = on
fix_version = off
estimate = off
labels = off
environment = off
description = on
Links to custom fields (example):
[push.fields]
customfield_XXXXX = on
[push.field.customfield_XXXXX]
label = Customer
size = compact
type = dropdown
required = true
As a result, when fixing a bug, you can safely use the Test Rail toolkit, and at the same time do not waste time going into the next window with JIRA. Plus? You know, when running large test cycles and setting test results, this is a definite plus, since those same tens of seconds finally add up to precious minutes and simply convenience of work, which is important.
And again: When generating a test report, we usually output after the results a list of known defects that block the passage of the test case. So, imagine such a list, and when you hover the cursor over a defect in this report, a popup pops up with JIRA about the ticket - Summary, Description, etc. Just brilliant! Everything is done for you.
4. After the main tech. settings, you can safely create projects linked to milestones and test suites containing various sections with test cases, as well as customize the dashboard exclusively according to your desires and criteria
We all know that when creating test cases for a functional, for the beginning, it is desirable to create a test plan that clearly describes the equivalence classes, sections and splitting at least into positive-negative scenarios. TestRail allows you to create such a plan as effectively as possible and subsequently expand it to full-fledged scenarios.
A little more detail: create a test suite. Then inside it we can add both sections and test cases inside these sections. We have the opportunity, without going to the inner essence of the case, to create templates (= check-list / test plan), enter only Summary and press Enter after entering. It looks like this:
Type the name (test / group / equivalence class), press Enter -> and you are already on the next case with a cursor in the input field. Thus, as quickly as possible, you can make the necessary plan, which was often written in Word, Excel, or even not even created. These developments should go through the teams of analysts / developers -> and by the results of the review it will be absolutely clear whether you need to add something or not. Do not forget about the fact that this is just a plan, and in order not to get confused with test cases that have already been filled out completely, we introduced an additional. Only Title field, which defaults to Yes.
This field can be corrected to No, only by writing the test case itself in the edit form (you just need to tick “Only Title” = No).
Any custom fields that you enter can also be put on the dashboards like all the others by selecting them in the displayed columns (pressing the “tags” on the main page of writing test cases in the line listing the field names: ID, Title so so on).
5. All internal fields of the test case are also customizable.
In the initial configuration of Test Rail, a classic and absolutely necessary set of internal fields of the test scenario is presented: ID, Priority, Initial Conditions, View, Steps, Expected Result, Reference to requirements or Implementation task in JIRA, etc. You have the right to add any field and customize it as you want: whether it is a string, checkbox, dropdown - it does not matter. Everything is in your hands and heads! What did we do:
- Added the Data for test-case field, in which for the test from one equivalence class simply enumerate any data with which the test should be performed. It can be anything: from the roles of performing, ending with listing the input lines.
- Added ID Auto and Auto fields . ID Auto - a unique ID for the auto-test. Auto - checkbox Yes / No, which shows whether the test case is automated or not.
This was done so that later when setting up Test Rail for automation, you could immediately use these parameters to run the necessary auto tests and generate the necessary report on them (http://www.gurock.com/testrail/tour/5/ - Automated tests and API).
6. For any test run, you can choose absolutely any set of test cases.
Test Rail allows you to create test cycles for any criteria. This is a very important feature of the tool, as it is often necessary to get rid of strictly defined scenarios (selected by priorities, by milestones, by name, by part of the name, but in fact, everything depends on the goals set). So, when assigning Test Run, we have the opportunity to choose “our” suite by filter. The general view is as follows:
But, let's say, we choose by name, which is “equal or contains something”:
With all this, the test suite can be selected to run completely.
7. Customizable reporting system
Any test run can be presented in the form of a clear report: “what-where-when-why”. It is possible to select any criteria, in the same way as when creating a test cycle. Among other things, it makes sense to set up the distribution of various reports for the necessary time and interested recipients.
Check-lists, ad-hoc testing is good, however, without clear procedures, the product often reveals weak spots, which eventually increase and drag the company down. Having both a set of test cases and a coverage matrix (at least componentwise) - this can be avoided.
I have listed far from all the possibilities of TestRail and cited as an example only a few options for using it. Nevertheless, it is obvious that such flexible customization will allow many people to customize this tool for their development process. Among other things, the tool allows you to completely transparently monitor the QA department, which, firstly, saves everyone from the understatement of "what they are testing," and, secondly, it establishes a clear interaction between the teams and gives an understanding of "why and how to test" .
Choose your tool, test, control the process! Good luck in all your endeavors!
The author of the article: Vasily Petukhov, head of the department of analysis and quality management Diary.ru
Lyrical introduction
What is Petersburg autumn? Rain, Rosenbaum, melancholy ... Is it all ?! No, this is also a new academic year, which in many regions of our country began with Dnevnik.ru. Over the summer, we managed to achieve some success: the number of project users exceeded 7,000,000, we became the only Russian company on the list of World Economic Forum Technology Pioneers 2014 , and the Republic of North Ossetia-Alania completely switched to electronic journals from Diary. ru On this lyrical introduction will end and move on to the essence of this post.
Introduction to the topic
In today's IT industry, many still wonder: “why do we need a separate test department?”. As one of the participants of the test team of our project, I declare responsibly: this is an erroneous opinion. Organizing a QA department is not easy, but worth it.
In this case, you need to clearly understand for yourself:
- what is considered a quality criterion?
- what metrics to use?
- with what tools to test?
- how to build interaction with developers / analysts, etc.
')
By and large, reinvent the wheel is not worth it - these practices are described on millions of Internet pages, successfully used in many corporations whose experience can be learned. However, we should not forget the simple truth - everywhere there is a specificity. For example, in Dnevnik.ru, these are strongly interconnected components and CI (Continuous Integration), and simply a high level of integration of related areas. How best to test in this case?
Let's start with a simple one: any information can never 100% automatically “pop up” in your (undoubtedly very clever) head. QA-shnik, first of all, it is necessary to memorize and later run test scenarios that are checked from release to release - “didn’t it break anything?”. As we all know - regression testing. But now it is not about that, about the types - welcome to protesting.ru (a good, by the way, resource).
So, for writing and storing test cases (“Test case is an artifact describing a set of steps, specific conditions and parameters necessary to check the implementation of the test function or its part”) you can use a lot of tools: Excel, notepad, google-docs , sheets of paper, after all. But is it efficient and convenient? For one person - maybe. For the team, where the status of the project “you should have known already yesterday” is definitely not.
Preparatory work
For a long time there have been many heavy Test Management programs that allow you to fully manage the test process: working with test plans, test cycles, requisitions, test cases, etc. I worked with different tulas - paid, free, self-hand-written ... For many years I was convinced that in this matter it is better not to skimp on money: either get a quality tool, or hire a team and write “perfect for a company, with the requirements that You need".
So, firstly: it became clear that the existing checklists do not cope with the success of testing. They are not enough, there is little information in them, they are not transparent, it’s difficult to provide results, they are difficult to maintain, there is not enough time to write new ones. Secondly: in the current situation, we needed a tool that was rather easy to understand and would allow us to effectively manage the test management scenario: “requirements -> test cases -> review -> test suite -> test cycle -> results”.
Summarizing all the above, the requirements for the tool were as follows:
- convenient user interface
- ease of installation, implementation and support
- convenient writing and storage of test scripts
- connection with JIRA
- effective management of the following scenario: "requirements -> test cases -> review -> test suite -> test cycle -> results"
- convenient selection of test cases from different areas to run in one test run
- ease of obtaining transparent reports on the launch of test suites
- the possibility in the future to associate with auto-testing and launch of written auto-tests
So, what was considered:
- Microsoft Test Lab ( how to in Russian )
- Zephyr
- Test Link
- Testia Tarantula
- Test Rail
PS: warning questions - we didn’t consider HP Quality Center , because 1) it’s unreasonably expensive 2) it’s not very “friendly” either, especially to those who have never worked with test-cases, and writing yours is resource costs and lack of time ...
I will try to meet a couple of key points:
Microsoft Test Lab
"Well, very cool thing, everything," isn't it? However, if you have never encountered its installation and implementation, you should not have optimistic forecasts. To put it mildly, if you have at your disposal a couple of free months - welcome. Install, configure, there is a nice bonus - the full how to in Russian. For us, the main advantages were the license and easy setup of communication with auto tests.
Total: there was no time to introduce and teach it to use.
Zephyr
A nice, modern, colorful tool integrated with JIRA. And yet - have you seen how to start test cycles there and write test cases? For editing - a separate window, adding steps and results - only line by line and numbered. Many unnecessary actions to "create-save" script.
Total: If we have one project for half a year before us, gently and leisurely drive in test cases, in the end everything will be quite nice and transparent. The option also did not suit us.
Test link
Very simple tool that allows you to do the most basic work of the tester - test cases + test cycles. And that's all. Design - for C grade, convenience - too. It seems to me that Test Link can be put at home in order to just learn how to insert test cases and pass test runs with Passed / Blocked / Failed. The option, of course, is not bad, especially free. Disadvantages: non-obviousness with auto-tests, imperfect writing of test cases.
Total: Okay, let's leave for consideration.
Testia Tarantula
Free Finnish toolkit, which is easy to implement in the QA team. Pros - quite well allows you to manage the test process: test cases, test wounds, reports. True structure: test case -> test suite. There are quite funny moments, such as “tags” in test cases, by which you can find the set you need (for example, you need negative test cases only for a specific functionality that affects this and this -> initially set in all scenarios the correct set of tags, from here it is easy to make a sample in a separate test run). Convenience is not bad, for auto tests you can customize ... Cons: reporting is lame, customization is problematic, it is not connected with JIRA.
Total: Leave as one of the options for consideration, why not?
Test rail
In my opinion, it is one of the easiest, but at the same time very large-scale existing bodies. The main advantage is that customization is possible in almost everything. Ranging from design to logic. Direct integration with JIRA. Effective recording of test cases and splitting them into the necessary suites. As one of the drawbacks - until recently, Test Rail had a poorly implemented reporting system: poor results on test injuries, lack of customization of reports, etc. However, Gurock & Co took up this business and by the 3rd version of the tool the guys did the work on the bugs. Reporting started playing with new colors: if you want, a report on mylston, if you want, only for defects, if you want, we will set up a regular e-mail with sending a report on testing a strictly defined functionality, etc. Well done! The result - this tool met all (!) Of my requirements, listed a little higher.
Total: our choice is the introduction of Test Rail. Actually, what is Test Rail?
• This is an effective management of test cases, plans and test cycles (creating / storing / editing test scenarios, managing test plans, launching test cycles, recording test results)
• This is a significant improvement in the quality of testing (a clear description of test scenarios, their review, correlation with the requirements, division into areas - all this allows us to evaluate both the completeness of the coverage of the functional tests and the necessary material for the entire project team)
• This is the receipt of test results in real time (creating reports on completely different criteria: Defect Summary, Comparison for Cases, test results for projects / components / milestown, etc.)
• This is an easy organization and convenient tracking of the testing department’s load (opportunities for full customization of the “working dashboard”, as well as convenient receipt of the work status of the QA department)
Implementation
1. Put the TestRail on the Windows Server 2008 R2 virtual machine with the MSSQL database
2. Set up AD authentication
3. Set up a full connection with JIRA
To begin with, each logged in TestRail user needs to configure a direct access to JIRA. To do this, we need a couple of custom variables jira_user + jira_password (welcome to: docs.gurock.com/testrail-integration/defects-plugins-variables ). After creating them in the JIRA_Rest plugin (Administration -> Integration -> JIRA_Rest), set the following:
[connection]
address = http: // jira / ***
user =% jira_user%
password =% jira_password%
Hooray. Now each user has the opportunity in My Settings to set login / password from JIRA.
Further, in JIRA_Rest we register both the connection directly from the established JIRA, and subsequently the connections of all the existing JIRA fields: whether it is the main field or custom (welcome to: docs.gurock.com/testrail-integration/tools-jira-rest ).
Direct connection with JIRA (example):
Reference View Url: your-server / jira / browse /% id%
Reference Add Url: your-server / jira / secure / CreateIssue ! Default.jspa
Links to main fields (example):
[push.fields]
summary = on
project = on
issuetype = on
component = on
assignee = on
priority = on
affects_version = on
fix_version = off
estimate = off
labels = off
environment = off
description = on
Links to custom fields (example):
[push.fields]
customfield_XXXXX = on
[push.field.customfield_XXXXX]
label = Customer
size = compact
type = dropdown
required = true
As a result, when fixing a bug, you can safely use the Test Rail toolkit, and at the same time do not waste time going into the next window with JIRA. Plus? You know, when running large test cycles and setting test results, this is a definite plus, since those same tens of seconds finally add up to precious minutes and simply convenience of work, which is important.
And again: When generating a test report, we usually output after the results a list of known defects that block the passage of the test case. So, imagine such a list, and when you hover the cursor over a defect in this report, a popup pops up with JIRA about the ticket - Summary, Description, etc. Just brilliant! Everything is done for you.
4. After the main tech. settings, you can safely create projects linked to milestones and test suites containing various sections with test cases, as well as customize the dashboard exclusively according to your desires and criteria
We all know that when creating test cases for a functional, for the beginning, it is desirable to create a test plan that clearly describes the equivalence classes, sections and splitting at least into positive-negative scenarios. TestRail allows you to create such a plan as effectively as possible and subsequently expand it to full-fledged scenarios.
A little more detail: create a test suite. Then inside it we can add both sections and test cases inside these sections. We have the opportunity, without going to the inner essence of the case, to create templates (= check-list / test plan), enter only Summary and press Enter after entering. It looks like this:
Type the name (test / group / equivalence class), press Enter -> and you are already on the next case with a cursor in the input field. Thus, as quickly as possible, you can make the necessary plan, which was often written in Word, Excel, or even not even created. These developments should go through the teams of analysts / developers -> and by the results of the review it will be absolutely clear whether you need to add something or not. Do not forget about the fact that this is just a plan, and in order not to get confused with test cases that have already been filled out completely, we introduced an additional. Only Title field, which defaults to Yes.
This field can be corrected to No, only by writing the test case itself in the edit form (you just need to tick “Only Title” = No).
Any custom fields that you enter can also be put on the dashboards like all the others by selecting them in the displayed columns (pressing the “tags” on the main page of writing test cases in the line listing the field names: ID, Title so so on).
5. All internal fields of the test case are also customizable.
In the initial configuration of Test Rail, a classic and absolutely necessary set of internal fields of the test scenario is presented: ID, Priority, Initial Conditions, View, Steps, Expected Result, Reference to requirements or Implementation task in JIRA, etc. You have the right to add any field and customize it as you want: whether it is a string, checkbox, dropdown - it does not matter. Everything is in your hands and heads! What did we do:
- Added the Data for test-case field, in which for the test from one equivalence class simply enumerate any data with which the test should be performed. It can be anything: from the roles of performing, ending with listing the input lines.
- Added ID Auto and Auto fields . ID Auto - a unique ID for the auto-test. Auto - checkbox Yes / No, which shows whether the test case is automated or not.
This was done so that later when setting up Test Rail for automation, you could immediately use these parameters to run the necessary auto tests and generate the necessary report on them (http://www.gurock.com/testrail/tour/5/ - Automated tests and API).
6. For any test run, you can choose absolutely any set of test cases.
Test Rail allows you to create test cycles for any criteria. This is a very important feature of the tool, as it is often necessary to get rid of strictly defined scenarios (selected by priorities, by milestones, by name, by part of the name, but in fact, everything depends on the goals set). So, when assigning Test Run, we have the opportunity to choose “our” suite by filter. The general view is as follows:
But, let's say, we choose by name, which is “equal or contains something”:
With all this, the test suite can be selected to run completely.
7. Customizable reporting system
Any test run can be presented in the form of a clear report: “what-where-when-why”. It is possible to select any criteria, in the same way as when creating a test cycle. Among other things, it makes sense to set up the distribution of various reports for the necessary time and interested recipients.
Conclusion
Check-lists, ad-hoc testing is good, however, without clear procedures, the product often reveals weak spots, which eventually increase and drag the company down. Having both a set of test cases and a coverage matrix (at least componentwise) - this can be avoided.
I have listed far from all the possibilities of TestRail and cited as an example only a few options for using it. Nevertheless, it is obvious that such flexible customization will allow many people to customize this tool for their development process. Among other things, the tool allows you to completely transparently monitor the QA department, which, firstly, saves everyone from the understatement of "what they are testing," and, secondly, it establishes a clear interaction between the teams and gives an understanding of "why and how to test" .
Choose your tool, test, control the process! Good luck in all your endeavors!
The author of the article: Vasily Petukhov, head of the department of analysis and quality management Diary.ru
Source: https://habr.com/ru/post/192572/
All Articles