Kitchenware testing: or what to choose from online resources for product testing?
In software testing, the organization of the process is very important: who does what? Who, what, when tested? On which assembly did one or another error appear? Who tested it? What is our claim status?
Trying to answer all these questions, we organized the storage of requirements and checklists in the form of google-tables, but over time we realized that it was time to look for a specialized tool. The results of the search and comparison of options I want to share with you in this article.
In connection with the remote working team and generally because of habit, we considered only web options. Rejecting the completely inappropriate decisions on the primary analysis, there were three candidates to choose from:
')
Testlink
Testray
Sitechco
The result is: Triple Threat Match. Someone who - see in a fair fight.
To begin with, we in the team defined the selection criteria:
According to these criteria and analyze each of the candidates.
Let's go as the decrease:
As for compatibility, all browsers are declared to support TestRails and TestLink, but IE is not supported in Sitechco:
Considering our set of environments, this is not critical, but for many users (especially those who are testing some kind of banking software in reduced environments) it may be.
The most important criterion in choosing a system for us, of course, is its functionality. We jointly worked out the list of necessary features:
Of the desirable, but not mandatory, the list includes:
In TestRail, test creation is implemented using the Test Suite, into which the Tests are placed.
Adding is done quickly, according to the principle of creating a Test Case and a separate description, with the possibility of sectioning and attaching files, if necessary. Inside the test, steps can be described in detail, but in most cases we have limited ourselves to clear headings.
Sitechco has implemented test suites in the form of Check and Cheat lists. The checklists themselves are simple, nothing superfluous or abstruse: test, verification, priority, and various additions: attachment of requirements, labor costs, hierarchy in the form of groups and master groups.
An important, as for me, role have cheat lists, with an already existing set of tests. Save time on creating your own and will be useful for the future. These elements play the role of a constructor: write standard cheat lists and insert them into check lists as needed. It is convenient when there are a lot of standard checks, for example, we need to check different fields for usability guidelines and XSS / SQL injections. The data is repeated, copying is evil, but the status of passing in different contexts and for different fields is needed.
Creating tests in TestLink resembles something like creating a test in TestRail. By analogy, test suites are created in Test Suite, in which tests are embedded.
It looks quite confusing and at the same time primitive:
The same tests can be added to different scenarios, which in our case performs the same function as the cheat sheets in the Strainer. But for the tests to be atomic and small (with a clear result for the report) in TestLink there seemed to be too many gestures. Well, a lot.
TestRail developers have offered their customers several reporting options:
That allows you to collect information about testing for bugs, results, launches and users. But! Each report must be processed, which is very inconvenient and makes it necessary to make unnecessary movements to obtain information, plus it takes time to generate a report:
Although this has its plus - there is the possibility of sending reports to several e-mails. As for the report type itself, a graphical representation of the results and a list of all tests with the corresponding value are displayed.
Sitechco allows you to generate reports automatically, without making unnecessary actions to receive it. 4 main types of the report give full information about the testing project.
The launch report displays the entire list of Check-lists for which the testing was performed, there is an opportunity to view general and detailed information. According to the results of the passage, it is possible to see a report on the 5 latest launches of the Checklist, plus, allows you to customize the report for yourself. The group report form shows in graphical form the result:
TestLink forced to sweat and torment with the launch of tests for passing. For a long time the system did not give in, which was no longer pleasant. After joint proceedings, we guessed how to do it. The version of the report was proposed by the developers in a form that seemed to us insufficiently informative:
Although there are a lot of options for reports, but all of the same type and in an unacceptable format, at least for us.
In TestRail can be exported in 3 formats: xml, CSV and Excel. But oddly enough, when I tried to get a report in Exel, I was issued in CSV format.
In the import process, everything looks simpler and is presented in the form of selecting a file from a PC in xml format. Also there are the possibility of adding to an existing list of tests and creating a new one.
Sitechco does not lose its ambitions and presented to its users data export in xls format. But here, too, its own snag, the opening of the file is possible only in Microsoft Office.
But the import was very interesting. I didn’t have to save anything in a separate file - insertion from a file on a PC or Google Docs was successful and without unnecessary gestures, simply copying the correct lines of the table.
Export to TestLink is presented only in one format - xml. It is possible to download a complete list of test cases, and individual folders.
Import is supported in a similar format - xml. It is presented in the form of downloading a separate file from a PC for a common list and for a folder separately.
So let's start in order. TestRail offers us a choice of 4 roles:
Roles are easily edited using checkboxes to allow actions:
Also pleased with the function of creating separate user groups, which allows you to split users into separate lists:
In the “kitchen” friend, I found 5 user roles:
For a long time he tried to understand who and what can, but the answer was “on the surface” - there is a detailed description in the certificate that simplifies the task. The only thing I would like to do is create my own lists, but apparently this is not provided for in the web version.
TestLink offers its users to use the already built-in options:
Also there is the ability to create your own lists of users, with the rights:
I formed the opinion that TestLink and TestRail have in some moments identical modification in the settings. But here in TestRail they worked on usability, and TestLink does not please us with convenience.
Our company uses Redmine and it is logical that we were looking for a system that supports integration with this bug tracker. All three applicants fall under the selection list. I will try to describe in more detail for each.
Testrail. Integration, as other users have said before: “simple and ingenious” at the same time:
This setting is suitable for most online bug trackers. When making a bug, the form, as it seemed to me, was not thought out to the end and therefore contained a minimum number of fields.
Perhaps the developers did not want to complicate their lives, and perhaps it was done in haste. But personally, I didn’t like it much, we fill in more fields when making mistakes, and it’s inconvenient to enter and edit it in Redmine after setting up an error from TMS. The link to the bug itself is displayed next to the test data, which does not make perception difficult, and when you click it, you can see the information in more detail — this is very convenient:
But the main disadvantage of integration via TestRail is the authorization of only one user, which is therefore very inconvenient when a programmer parses bugs. There are unnecessary questions and fuss.
The integration settings in Sitechco are quite simple and are carried out step by step. It is difficult to get confused. Initially, a project is selected, a link to the bug tracker itself and admin access to it.
At first, everything is usually, but further on there is an obvious plus: the ability to customize the fields that will be displayed in the form of the institution of the defect. Undoubtedly the right thing!
All my settings had an impact on the forms to fill out. You can not say anything, soundly thought out and for people.
The reports are also nothing supernatural, a list of bugs, separated by commas, the reference data and display the status of the bug at the moment.
TestLink. Integration by analogy with TestRail, because I will not be strongly dispersed, since I did not like the integration itself initially.
It is very inconvenient to start up bugs in this format, living in the world of developing technologies.
The next important point for us is links with project requirements. In our project, they are not conducted in detail, but the status of them should be transparent and timely, and earlier we used a google-tablet, in which we forgot to update something.
Having touched all three of our candidates, I did not notice the obvious in TestRail, so on this point he leaves the game.
In Sitechco, the “Requirements” item is in the main menu, but it made me take over before I could make it active (through the project settings, as it turned out).
Then we get to the list of requirements that we can create or import from tables.
The creation window contains nothing superfluous: the name and description. All that is needed is the addition of attached files - you may have to use links to third-party file hosting sites.
I was also pleased with the import of requirements, apparently created by analogy with the import of tests, also nothing complicated or abstruse. Presented as a window, with a choice of catalog for import and the field itself. There is also a help:
By creating / importing requirements, we can make a link between each test (checking in the checklist, not the entire checklist) and the requirement, this link works on the principle of many to many:
Let's see how things are in TestLink. The manufacturer offers a set of functions for working with requirements: create, import, export, create test suites and copy requirements.
The requirements creation operations are quite simple, they have the ability to attach files, which more than covered all my desires for the requirements type. But it was not very inconvenient to use - just like everything in TestLink.
The import itself as well as the export is supported in CSV \ XML \ DocBook format. Apparently the entire Testlink project is focused on the CSV format.
Requirements are attached to the Test Case and assist in the testing process in order to verify the completeness of coverage of requirements with checks and no gaps during testing.
I said a little about the simplicity of work at the beginning, I would like to continue the conversation on this topic and reveal the following features. The ability to use hotkeys to create tests and groups in checklists is implemented only in Sitechco.
The same points as the description of user groups and ease of use are noted in all three products. But against the background of all standard requirements, I really liked the fact that TestRails allows you to create a project address yourself, and testlink stepped even further and offered its own data storage server. Sitechco, however, was distinguished by conservatism in this regard and did not indulge in additional bells and whistles, apparently relying on the broadest functionality and ease of use.
Something supernatural I could not find. The only thing that caught my eye was the Task Scheduler, which is somewhat similar to Todoist , due to the display of the number of tasks on me and the ability to plan for the future.
Most importantly, as our test manager said: “I can now control you,” and smiled slyly. It turns out that this tab also allows you to view which of the checklists are currently running, which one has been completed. “Palivo” - I thought and tried to look for this function in TestRail and TestLink. Alas! This functionality is supported only in Sitechco, and to consider this as a plus or a minus - we are taking different points of view with the authorities.
Due to the development of all kinds of mobile chips, I also expected to dig up such things (mobile versions, mobile applications), but, unfortunately, none of the opponents had this functionality in their armament. Testing with a bunch of tabs is inconvenient, agree. Therefore, it would be “to the cashier” to make marks on the results on the tablet and at the same time not clutter up the space on the table.
As one of the Ukrainian anchormen, Olga Freimut, says in her checks: “Well, what? Let's talk? "(Well? Let's talk?)
Based on your preferences, I tried to objectively compare these three fighters, but I have not yet decided on 2 winners: kitchen utensils or a business class thing. Both candidates have their pros and cons, which in their own way attract. In any case, the issue will be decided by the fin. Director based on the allocated budget, but I can only wait.
Trying to answer all these questions, we organized the storage of requirements and checklists in the form of google-tables, but over time we realized that it was time to look for a specialized tool. The results of the search and comparison of options I want to share with you in this article.
In connection with the remote working team and generally because of habit, we considered only web options. Rejecting the completely inappropriate decisions on the primary analysis, there were three candidates to choose from:
')
Testlink
Testray
Sitechco
The result is: Triple Threat Match. Someone who - see in a fair fight.
To begin with, we in the team defined the selection criteria:
- License cost : preferably free of charge, or at least, inexpensive;
- Supported environments : the team uses Google Chrome, Mozilla Firefox, Safari - no one is ready to refuse or change browsers;
- Functionality of test management : here the most important thing for us was the convenient creation of test suites, the statistics of passing and visual reporting;
- The presence of elements of test management : I really want the tool to plan my tasks, control plans and deadlines;
- Usability : if someone in the team does not like something, then we will hardly waste our time and “delve into”. Everything should be clear right away.
According to these criteria and analyze each of the candidates.
Cost of
Let's go as the decrease:
- TestRails offers us exclusively paid licenses, depending on their conditions, it will cost from $ 69 to $ 8999, a serious payment for a serious product. In order not to buy a cat in a bag, we are offered to use the test version for 30 days, which will be free - according to it and we carried out the analysis. In our case (up to 15 users) TestRails will cost € 1399 or $ 2149.
- Sitechco for local installation costs 12,000 rubles, SaaS use for free. Given the work on the https protocol and not too strict, in terms of security, customers, we are completely satisfied with the web version.
- TestLink is free if you install it on your server. I had to tinker, but started. For the price of a clear leader.
Supported environments
As for compatibility, all browsers are declared to support TestRails and TestLink, but IE is not supported in Sitechco:
Considering our set of environments, this is not critical, but for many users (especially those who are testing some kind of banking software in reduced environments) it may be.
Test management functionality
The most important criterion in choosing a system for us, of course, is its functionality. We jointly worked out the list of necessary features:
- Creation of tests and checklists;
- Control of testing;
- Import and export of tests (the desired format is tables);
- Testing reports and software quality assessment;
- The division of user rights;
Of the desirable, but not mandatory, the list includes:
- Redmine integration;
- Maintain requirements with test coverage analysis;
- External API for auto tests;
Creating tests and checklists
In TestRail, test creation is implemented using the Test Suite, into which the Tests are placed.
Adding is done quickly, according to the principle of creating a Test Case and a separate description, with the possibility of sectioning and attaching files, if necessary. Inside the test, steps can be described in detail, but in most cases we have limited ourselves to clear headings.
Sitechco has implemented test suites in the form of Check and Cheat lists. The checklists themselves are simple, nothing superfluous or abstruse: test, verification, priority, and various additions: attachment of requirements, labor costs, hierarchy in the form of groups and master groups.
An important, as for me, role have cheat lists, with an already existing set of tests. Save time on creating your own and will be useful for the future. These elements play the role of a constructor: write standard cheat lists and insert them into check lists as needed. It is convenient when there are a lot of standard checks, for example, we need to check different fields for usability guidelines and XSS / SQL injections. The data is repeated, copying is evil, but the status of passing in different contexts and for different fields is needed.
Creating tests in TestLink resembles something like creating a test in TestRail. By analogy, test suites are created in Test Suite, in which tests are embedded.
It looks quite confusing and at the same time primitive:
The same tests can be added to different scenarios, which in our case performs the same function as the cheat sheets in the Strainer. But for the tests to be atomic and small (with a clear result for the report) in TestLink there seemed to be too many gestures. Well, a lot.
Control results
TestRail developers have offered their customers several reporting options:
That allows you to collect information about testing for bugs, results, launches and users. But! Each report must be processed, which is very inconvenient and makes it necessary to make unnecessary movements to obtain information, plus it takes time to generate a report:
Although this has its plus - there is the possibility of sending reports to several e-mails. As for the report type itself, a graphical representation of the results and a list of all tests with the corresponding value are displayed.
Sitechco allows you to generate reports automatically, without making unnecessary actions to receive it. 4 main types of the report give full information about the testing project.
The launch report displays the entire list of Check-lists for which the testing was performed, there is an opportunity to view general and detailed information. According to the results of the passage, it is possible to see a report on the 5 latest launches of the Checklist, plus, allows you to customize the report for yourself. The group report form shows in graphical form the result:
TestLink forced to sweat and torment with the launch of tests for passing. For a long time the system did not give in, which was no longer pleasant. After joint proceedings, we guessed how to do it. The version of the report was proposed by the developers in a form that seemed to us insufficiently informative:
Although there are a lot of options for reports, but all of the same type and in an unacceptable format, at least for us.
Import and export tests (desired format - tables)
In TestRail can be exported in 3 formats: xml, CSV and Excel. But oddly enough, when I tried to get a report in Exel, I was issued in CSV format.
In the import process, everything looks simpler and is presented in the form of selecting a file from a PC in xml format. Also there are the possibility of adding to an existing list of tests and creating a new one.
Sitechco does not lose its ambitions and presented to its users data export in xls format. But here, too, its own snag, the opening of the file is possible only in Microsoft Office.
But the import was very interesting. I didn’t have to save anything in a separate file - insertion from a file on a PC or Google Docs was successful and without unnecessary gestures, simply copying the correct lines of the table.
Export to TestLink is presented only in one format - xml. It is possible to download a complete list of test cases, and individual folders.
Import is supported in a similar format - xml. It is presented in the form of downloading a separate file from a PC for a common list and for a folder separately.
Division of user rights
So let's start in order. TestRail offers us a choice of 4 roles:
Roles are easily edited using checkboxes to allow actions:
Also pleased with the function of creating separate user groups, which allows you to split users into separate lists:
In the “kitchen” friend, I found 5 user roles:
For a long time he tried to understand who and what can, but the answer was “on the surface” - there is a detailed description in the certificate that simplifies the task. The only thing I would like to do is create my own lists, but apparently this is not provided for in the web version.
TestLink offers its users to use the already built-in options:
Also there is the ability to create your own lists of users, with the rights:
I formed the opinion that TestLink and TestRail have in some moments identical modification in the settings. But here in TestRail they worked on usability, and TestLink does not please us with convenience.
Redmine integration
Our company uses Redmine and it is logical that we were looking for a system that supports integration with this bug tracker. All three applicants fall under the selection list. I will try to describe in more detail for each.
Testrail. Integration, as other users have said before: “simple and ingenious” at the same time:
This setting is suitable for most online bug trackers. When making a bug, the form, as it seemed to me, was not thought out to the end and therefore contained a minimum number of fields.
Perhaps the developers did not want to complicate their lives, and perhaps it was done in haste. But personally, I didn’t like it much, we fill in more fields when making mistakes, and it’s inconvenient to enter and edit it in Redmine after setting up an error from TMS. The link to the bug itself is displayed next to the test data, which does not make perception difficult, and when you click it, you can see the information in more detail — this is very convenient:
But the main disadvantage of integration via TestRail is the authorization of only one user, which is therefore very inconvenient when a programmer parses bugs. There are unnecessary questions and fuss.
The integration settings in Sitechco are quite simple and are carried out step by step. It is difficult to get confused. Initially, a project is selected, a link to the bug tracker itself and admin access to it.
At first, everything is usually, but further on there is an obvious plus: the ability to customize the fields that will be displayed in the form of the institution of the defect. Undoubtedly the right thing!
All my settings had an impact on the forms to fill out. You can not say anything, soundly thought out and for people.
The reports are also nothing supernatural, a list of bugs, separated by commas, the reference data and display the status of the bug at the moment.
TestLink. Integration by analogy with TestRail, because I will not be strongly dispersed, since I did not like the integration itself initially.
It is very inconvenient to start up bugs in this format, living in the world of developing technologies.
Maintain requirements with test coverage analysis
The next important point for us is links with project requirements. In our project, they are not conducted in detail, but the status of them should be transparent and timely, and earlier we used a google-tablet, in which we forgot to update something.
Having touched all three of our candidates, I did not notice the obvious in TestRail, so on this point he leaves the game.
In Sitechco, the “Requirements” item is in the main menu, but it made me take over before I could make it active (through the project settings, as it turned out).
Then we get to the list of requirements that we can create or import from tables.
The creation window contains nothing superfluous: the name and description. All that is needed is the addition of attached files - you may have to use links to third-party file hosting sites.
I was also pleased with the import of requirements, apparently created by analogy with the import of tests, also nothing complicated or abstruse. Presented as a window, with a choice of catalog for import and the field itself. There is also a help:
By creating / importing requirements, we can make a link between each test (checking in the checklist, not the entire checklist) and the requirement, this link works on the principle of many to many:
Let's see how things are in TestLink. The manufacturer offers a set of functions for working with requirements: create, import, export, create test suites and copy requirements.
The requirements creation operations are quite simple, they have the ability to attach files, which more than covered all my desires for the requirements type. But it was not very inconvenient to use - just like everything in TestLink.
The import itself as well as the export is supported in CSV \ XML \ DocBook format. Apparently the entire Testlink project is focused on the CSV format.
Requirements are attached to the Test Case and assist in the testing process in order to verify the completeness of coverage of requirements with checks and no gaps during testing.
Usability:
I said a little about the simplicity of work at the beginning, I would like to continue the conversation on this topic and reveal the following features. The ability to use hotkeys to create tests and groups in checklists is implemented only in Sitechco.
The same points as the description of user groups and ease of use are noted in all three products. But against the background of all standard requirements, I really liked the fact that TestRails allows you to create a project address yourself, and testlink stepped even further and offered its own data storage server. Sitechco, however, was distinguished by conservatism in this regard and did not indulge in additional bells and whistles, apparently relying on the broadest functionality and ease of use.
Special offers:
Something supernatural I could not find. The only thing that caught my eye was the Task Scheduler, which is somewhat similar to Todoist , due to the display of the number of tasks on me and the ability to plan for the future.
Most importantly, as our test manager said: “I can now control you,” and smiled slyly. It turns out that this tab also allows you to view which of the checklists are currently running, which one has been completed. “Palivo” - I thought and tried to look for this function in TestRail and TestLink. Alas! This functionality is supported only in Sitechco, and to consider this as a plus or a minus - we are taking different points of view with the authorities.
Due to the development of all kinds of mobile chips, I also expected to dig up such things (mobile versions, mobile applications), but, unfortunately, none of the opponents had this functionality in their armament. Testing with a bunch of tabs is inconvenient, agree. Therefore, it would be “to the cashier” to make marks on the results on the tablet and at the same time not clutter up the space on the table.
As one of the Ukrainian anchormen, Olga Freimut, says in her checks: “Well, what? Let's talk? "(Well? Let's talk?)
- TestLink. Actually, what did I expect from the free service? All guesses were confirmed. Put a tick that is free, there is a basic set of functions - only that. Sorry, buddy, you're knocked out, it's somehow complicated and you haven't seen the highlight.
- TestRails . Yes, you will not say anything, a powerful thing, in its pluses and minuses. But due to the lack of multilingualism and not distinguished by such cheapness, I could not afford it. I'm not the only one who will work, because I had to “remove from the settings” my egoism.
- Sitechco . Simple and tasteful, nothing abstruse or difficult to understand. It may have seemed to me, but the company only recently began to appear in the IT arena and underestimates its capabilities, or the special price of the product due to the economic crisis of the countries.
| Comparison criteria | Sitechco | Testlink | Testrails |
|---|---|---|---|
| Main | |||
| license cost | $ 0- $ 500 | - | $ 69- $ 8,999 |
| browsers | except iE | everything | everything |
| functional | |||
| autosave / version control | Yes | Yes | Yes |
| templates | Yes | not | not |
| print / report link | Yes | Yes | Yes |
| import Export | Yes | well no | Yes Yes |
| mobile version | not | not | not |
| environments / builds / requirements | Yes Yes Yes | no / yes / yes | no / yes / no |
| tips | Yes | not | Yes |
| Elements of test management: | |||
| test plan | Yes | Yes | not |
| test cases | Yes | Yes | Yes |
| autotest support | not | not | not |
| checklists | Yes | Yes | not |
| Differentiation of user roles | there is | there is | there is |
| file attachment support (file storage) | not | not | not |
| E-mail Notification | there is | not | there is |
| Work with Check-lists | |||
| setting fields in CL (different types of CL, setting up a test environment, for example) | there is | not | not |
| ease of editing (subjective assessment) | conveniently | not comfortable | not very comfortable |
| communication with requirements | Yes | Yes | not |
| Reporting | |||
| on assemblies | Yes | Yes | not |
| around | Yes | not | not |
| by testers | Yes | not | not |
| by execution date | Yes | not | Yes |
| by result | Yes | Yes | Yes |
| any display of the status of the tested product | there is | there is | there is |
| reporting on different points at once | Yes | Yes | Yes |
| claims coverage reporting | Yes | Yes | not |
| reporting test results | Yes | Yes | not |
| Usability | |||
| simplicity | Yes | Yes | Yes |
| hotkeys | Yes | not | not |
| description of user groups | Yes | Yes | Yes |
Based on your preferences, I tried to objectively compare these three fighters, but I have not yet decided on 2 winners: kitchen utensils or a business class thing. Both candidates have their pros and cons, which in their own way attract. In any case, the issue will be decided by the fin. Director based on the allocated budget, but I can only wait.
Source: https://habr.com/ru/post/242025/
All Articles