Automated QA System: a headache pill for testers using the Star Crusade game as an example
Interview with QA-architect Vladimir Savelyev continues a series of our publications on current IT topics. Vladimir talks about the main headaches of testers and shares an effective way to solve them.
Who is "who":
Automated QA System is a testing constructor platform that has a single structure, a unified model and allows automated tests for this unified model. This platform is also suitable for a simple run of autotests on waterfall projects, and for integrated quality control on agile projects, as well as for projects that use continuous integration development and other techniques.
Star Crusade is a collectible card game with a large number of cards (more than 320 pcs.) The game has a complex server part and a client application with a large number of graphic effects, animated graphics and various functions (chats, friends system and duels).
')
The product has a number of web interfaces:
- server web interfaces that are used by the product development and support team
- the site of the game, which is visible to end users and where you can download the game assembly. Currently available alpha assembly.
What are the problems in testing? Can you list them?
V.S.: Based on my experience, I can highlight a few major problems in testing.
Problem 1: Bugs due to unstructured testing.
If a tester performs a series of tests that are logically inconsistent and unstructured, then such testing may not cover the entire product, so every time he finds more and more new bugs, for which developers and testers have to communicate a lot to fix. The excess of such communications leads to loss of time and deterioration of the product quality.
Problem 2: lack of tools for effective bug analysis or the inability to use these tools 100%. The fact is that the analysis of the bug does not boil down only to its verbal description (for example, that some button does not work or the action in the game is done incorrectly). In order for the developer to understand what the bug is and correct it, the tester must collect a certain amount of information about the found bug.
The classic example is the logs (reports) of the application under test, including screenshots of the screen, some extended tables (samples from databases). On complex projects it is difficult to put all the information together, and it takes too much time. Therefore, it is necessary to have tools that allow you to quickly and efficiently collect information about defects found. The lack of such tools is another tester’s headache and, as a result, a developer’s headache.
Problem 3: constant routine work. As everyone knows, during the implementation of IT projects, developers almost daily add and optimize functionality, correct found bugs. Therefore, the application must be re-tested each time to check the functionality of the new functionality and to search for regression bugs. Otherwise, it will not be possible to guarantee that the code that worked before will continue to work after the changes have been made. The tester has to run a full set of tests every day, which only increases each time.
And separately, I would like to add regression bugs to a subclass . When a developed product is not tested regularly or unstructuredly tested, a huge number of regression bugs appear. Over time, these bugs accumulate like a snowball and make further product development simply impossible.
How can all of these problems be solved to simplify the development and testing process?
VS: First of all, it is necessary to solve the problems of communications. All team members (both testing and development) must not only fully understand the component for which they are responsible, but also see the product under development in the complex, and have minimal knowledge of all other components of the product. In this we are helped by our practices for quality control, which include tools such as:
These tools were created in the course of work on various projects, and they allow you to build a high-quality graphical tree-like model of project testing in the same way as in Agile development. The project is divided into Epic'i, which in turn are divided into User Story and specific tasks of development and testing. The model fits well into the structure of development on Agile projects and helps to structure and systematize testing on Waterfall projects.
At first glance, the tools are nothing special. This is just a diagram showing the product model and a tree-like model for displaying tests. I have repeatedly heard reviews that these are “obvious things”. However, it is easy to speak, looking at the finished result. The reality of working on projects shows that these things are not so obvious.
On the graphic model of quality control, the product is displayed in the form in which testers need to see it. For example, at the top level, it may not matter to me that the product code runs on Tomcat and which libraries are used to communicate with MySQL and LDAP. It is important for me to check that the product is able to perform a user operation: authorization (which includes database access and data exchange with clients), message exchange (interaction between 2 XMPP clients) and so on. However, nothing prevents me from going down one level and refining the model or breaking it up into several slides / screens, showing additionally the internal structure of the components.
This quality control model can be made an interactive tool, collecting in it not just information about the device of the product, but also project documentation, information on unit tests, various knowledge base articles and so on. After all, solving a specific test / development task, the specialist does not need all the project documentation, he needs information on the part of the product on which he is currently working.
Thus, we guarantee that the team has the same idea about the product and any team member can quickly get the information he needs. The team should not waste time explaining everything to each new employee from scratch, and the employee should not waste time reading a bunch of documentation, from which no more than 30% is needed to solve his current task.
Graphic quality control model
This is also the case for the test coverage map. It logically continues the quality control model and allows you to cover the product with tests in a format in which the functionality is being developed. If it was decided to modify some functionality or even rework, then the test coverage can be updated quickly, since it is presented in the form of an understandable tree model.
There was a lot of discussion about how much time it takes. For example, I spent about 15 minutes creating a test coverage map, which is shown here.
Test coverage represented by the test list
Test coverage using a tree model (test coverage maps)
Tell us more about the results of the implementation of the developed tools.
VS: An example from life: I took a test coverage of one of the components, made by an experienced colleague-tester, where there were about 70 tests, and broke it as shown on the test coverage map. About 15 missed tests were found, and the whole work took about 30 minutes.
Over time, each product you begin to see a similar model, and any activity associated with testing the product, become structured and consistent. This is especially true for beginners. I remember how I started writing tests and could not understand which way I should approach testing, how to cover the product with tests so that I would not miss anything. If I had such tools then, I would start working as a full-fledged employee much faster.
We also conducted an experiment. The newcomer was provided with an unstructured and partially obsolete test coverage and an example with this card. Having an example at hand, a test coverage chart, and asking some questions for a hike, a person who had no experience in testing could create a good test coverage within 2 days. He reached the level of a full-time testing engineer within a month, i.e. a month later he was already fully working with the team, while it was about testing complex telecom systems.
Another example of using the model + card bundle is our experience in communication with a technical writer. Starting to write user documentation for a practically finished product, he first decided to figure out how this product is structured in order to document it in a structured way. I also gave him a model + card and made a few comments. He instantly figured out the device he needed part of the product. He liked the model so much that he included it in the official product documentation.
That is, the use of our developments allows us to quickly delve into the projects of customers and quickly adapt to the system the customer’s specialists who work with us.
And what about the lack of tools for testing and analyzing bugs?
VS: As all testers know, there are a large number of bug analysis tools. Some of them are better, some are worse, but in general, only the ability to use several tools at once can give a good result.
If the available tools do not allow us to solve the tasks before us, we create them ourselves.
As an example, I can give a technique for analyzing the logs of any application under test during the test. Logs are the main source of information about what happened in the application when the tester found a bug.
The tools we have created allow us to efficiently collect logs for all the necessary components and provide a clear cut to the developer, which will allow him to find out what the defect is. The engineer can see the complete picture at once, and not collect it in parts.
The lowest level of our developments is testing automation, i.e. We invest a comprehensive model of quality control, a test coverage map and the tools we create in our auto tests. Thus, the problems of routine work and regression bugs are solved simultaneously.
We have identified three basic stages of the AutoTest:
At the first stage, we launch the application, make certain settings in it and prepare the implementation of the functional part. For example, if we test the game menu, we need to start the application, open this menu and continue to perform testing operations in it.
The second stage is directly the testing operations themselves - those actions that will help us understand whether the functionality that we are testing is working correctly or is wrong.
The third stage is especially important for autotests. This is a cancellation of the changes made. Each test, incl. and automatic, must perform some testing operations in a test environment, leaving it unchanged after completion of the work. Because the key requirement for quality testing is to perform test operations in an unchanged environment and in steps like the user will do.
Thus, our testing experience solves all the listed problems of testers.
Let's talk more about the system. What other benefits of the Automated QA System can be highlighted?
VS: For IT professionals, the advantages are our integrated quality control model , which allows you to systematize and structure communications on a project, which will optimize the testing and development process.
From my point of view, our constructor platform will also be of interest. I am well understood by people who are familiar with Linux. As we all know, Linux is a large complex of small tools, each of which performs a separate function, but if the tools are used together, they can solve much larger tasks. According to this principle, I built our platform constructor, which we call the Automated QA System .
Our autotests are used for large-scale testing tasks . As an example, I’ll cite our functional autotests from the Star Crusade project, which, starting with simple scripts for testing maps, have been supplemented with new tools and are now a tool for testing the entire product: the client and the server. Star Crusade has many cards, the parameters of which are constantly changing as they are balanced, which leads to the need for constant testing of the product.
When developing tests for the project, we chose between two options:
1) stable autotests that can independently handle certain map changes
2) tests that need to be changed manually, but which are also easy to create and update.
As a result, we stopped at the second version, since working with autotests and observing game scenarios through the client, the tester remains constantly immersed in this environment. He starts to try, experiment, find new areas, non-standard bugs and can quickly fix all these scenarios with an autotest.
This further underlines the fact that testers use autotests as a diagnostic tool to control the quality of the entire product, and not just to solve a local task.
For professionals, a huge advantage will be the task management mechanism , which is implemented in the Automated QA System. To test large projects, each time you first need to download a fresh build of the product, deploy it in a test environment, perform a number of specific settings, and only then you can run the autotests directly. The task management mechanism allows you to efficiently create such tasks, automate them and launch them in the sequence you need for the tester.
We also use automation by macromodules — test modules that perform a more global task than a regular module.
For comparison, I will give the usual object in programming, which performs a certain number of tasks. The macromodule in this case will be the part of the test that performs the complex task. For example, making certain product settings using the web interface or using the server-side functionality.
The advantage of Automated QA System is the possibility of hybrid automation . Using macromodules, we can create hybrid tests that perform operations in completely different environments (managing the graphical part of the product, performing operations on Windows workstations, Unix servers, mobile devices, or in the web interface). All these operations can be used within one test, organize the exchange of information between macromodules and provide a centralized result on the performance of such a test.
What benefits does the system offer to customers?
VS: First, we provide software package quality control services. This means that we provide comprehensive quality control of our client’s project. The model has the experience of the testing team for more than 10 years. This is quite a long period during which we have developed practices, approaches, revealed a large number of “bottlenecks”, systematized and consolidated the experience gained, and came to the quality control model that I described. Only through integrated quality control can the project achieve the best quality.
Secondly, our automated testing system and our manual testing model (which includes a test coverage map and a graphical quality control model), along with structured tests, allow us to quickly provide the results to the customer . This means that the tests that we write and conduct, initially represent a structured tree model, from which it is easy to create a report in manual or automatic mode and present it to the customer. Our automated exit tests give a result in a relatively understandable format for people, i.e. The results of the execution of most tests can be assessed by the manager independently. They do not contain a large amount of technical data, which are understandable only to specialists.
Thirdly, in the case of automatic tests, it is possible to monitor the runs of automatic test packages of the product online . To do this, we have implemented a web interface to which we can provide the customer with access. The customer has the ability to track also various metrics: graphs with the dynamics of the so-called. pass rate - the percentage of successful test execution, the dynamics of adding automatic tests and a number of other trends, incl. we can add the necessary graphs and organize the collection of information on these graphs.
The dynamics of passing tests (pass-rate trends)
This means that our system is expandable and upgradeable.
Thus, the customer receives a comprehensive test of the entire project and the opportunity to receive detailed reports on the tests carried out both in offline and online modes.
We also provide the opportunity to work together with our customer's team, which allows you to share experiences.
Could you please give some practical examples of using the Automated QA System?
VS: Our automation experience began with testing complex telecom systems, where key functionality worked on a linux server, and user interaction was carried out via web interfaces. We collected most of our experience on such projects.
At the moment we are actively using the model described above on our flagship project Star Crusade. We use automated functional tests to test the server side of the product. We test the server part with the help of controlled bots that perform certain operations on the server using the AutoTest command.
We test actions on the playing field in the client part by watching the AutoTest run through a special observer mode, a functional that will also be a game. When the product interface comes to the "finish line" we fix it with autotests using graphical vision technologies, when the test performs operations in the client as well, as the end user would do. From the point of view of testing, this is an ideal option, because, receiving commands from mouse or keyboard drivers, the application is controlled by the same interfaces that the user will control.
And we are testing web services (website, server-side web interfaces) with the help of Selenium, while we also have solutions that simplify Selenium automation and make it more efficient. Thus, all functional tests are combined into our testing package.
The second level of testing is system tests. System tests are definitely hybrid testing, i.e. management of several tools within one test. Here we use the same macromodule constructor. For example, within the framework of one test, we perform certain settings on the server using the Linux console and the web interface, check the necessary mechanics using controlled bots, and complete the testing using graphical vision. The key here is the ability to quickly change autotests.
For example, we are testing the ability of a user to register on the site, confirm his e-mail account, and then enter the game under his account and play some kind of game game. This can be done in different ways. We can register on the site using the web interface explicitly, or we can send requests to the server at a low level. The output will be the same. But in the first case, we also test the site, its web interface, and in the second, we simply quickly perform the necessary operation, without concentrating on the web interface of the site.
With the use of controlled bots we play the game in the game. In this case, we quickly and reliably test the server part of the product to a greater extent. It is also possible to test the product through the client using a graphical view, then we also check the client part. In the created autotest, we can switch various functional elements. For example, the first time we test using the product settings via the web interface and run the tests with the help of guided bots, getting a centralized output of the result. And another time, we are already concentrating on the graphic part of the product, and setting up the product by direct request to the server and playing the game through the client application.
Switching between modules occurs by changing one or two options in automatic tests. Auto tests perfectly complement other tools used by our testers.
Automated QA System :
1. ,
2.
3.
4.
5.
Automated QA System . ?
..: Automated QA System:
match-3. Lil Quest , Muffin Quest - – , . . , , .
– . , -. , .
. , .
, - , . . , -, , , , . , StarCrusade .
, : , . , - .
. , .. . , , .
, , . , .
.
!
Who is "who":
Automated QA System is a testing constructor platform that has a single structure, a unified model and allows automated tests for this unified model. This platform is also suitable for a simple run of autotests on waterfall projects, and for integrated quality control on agile projects, as well as for projects that use continuous integration development and other techniques.
Star Crusade is a collectible card game with a large number of cards (more than 320 pcs.) The game has a complex server part and a client application with a large number of graphic effects, animated graphics and various functions (chats, friends system and duels).
')
The product has a number of web interfaces:
- server web interfaces that are used by the product development and support team
- the site of the game, which is visible to end users and where you can download the game assembly. Currently available alpha assembly.
What are the problems in testing? Can you list them?
V.S.: Based on my experience, I can highlight a few major problems in testing.
Problem 1: Bugs due to unstructured testing.
If a tester performs a series of tests that are logically inconsistent and unstructured, then such testing may not cover the entire product, so every time he finds more and more new bugs, for which developers and testers have to communicate a lot to fix. The excess of such communications leads to loss of time and deterioration of the product quality.
Problem 2: lack of tools for effective bug analysis or the inability to use these tools 100%. The fact is that the analysis of the bug does not boil down only to its verbal description (for example, that some button does not work or the action in the game is done incorrectly). In order for the developer to understand what the bug is and correct it, the tester must collect a certain amount of information about the found bug.
The classic example is the logs (reports) of the application under test, including screenshots of the screen, some extended tables (samples from databases). On complex projects it is difficult to put all the information together, and it takes too much time. Therefore, it is necessary to have tools that allow you to quickly and efficiently collect information about defects found. The lack of such tools is another tester’s headache and, as a result, a developer’s headache.
Problem 3: constant routine work. As everyone knows, during the implementation of IT projects, developers almost daily add and optimize functionality, correct found bugs. Therefore, the application must be re-tested each time to check the functionality of the new functionality and to search for regression bugs. Otherwise, it will not be possible to guarantee that the code that worked before will continue to work after the changes have been made. The tester has to run a full set of tests every day, which only increases each time.
And separately, I would like to add regression bugs to a subclass . When a developed product is not tested regularly or unstructuredly tested, a huge number of regression bugs appear. Over time, these bugs accumulate like a snowball and make further product development simply impossible.
How can all of these problems be solved to simplify the development and testing process?
VS: First of all, it is necessary to solve the problems of communications. All team members (both testing and development) must not only fully understand the component for which they are responsible, but also see the product under development in the complex, and have minimal knowledge of all other components of the product. In this we are helped by our practices for quality control, which include tools such as:
- graphic quality control model
- test coverage map
- clearly structured tests
These tools were created in the course of work on various projects, and they allow you to build a high-quality graphical tree-like model of project testing in the same way as in Agile development. The project is divided into Epic'i, which in turn are divided into User Story and specific tasks of development and testing. The model fits well into the structure of development on Agile projects and helps to structure and systematize testing on Waterfall projects.
At first glance, the tools are nothing special. This is just a diagram showing the product model and a tree-like model for displaying tests. I have repeatedly heard reviews that these are “obvious things”. However, it is easy to speak, looking at the finished result. The reality of working on projects shows that these things are not so obvious.
On the graphic model of quality control, the product is displayed in the form in which testers need to see it. For example, at the top level, it may not matter to me that the product code runs on Tomcat and which libraries are used to communicate with MySQL and LDAP. It is important for me to check that the product is able to perform a user operation: authorization (which includes database access and data exchange with clients), message exchange (interaction between 2 XMPP clients) and so on. However, nothing prevents me from going down one level and refining the model or breaking it up into several slides / screens, showing additionally the internal structure of the components.
This quality control model can be made an interactive tool, collecting in it not just information about the device of the product, but also project documentation, information on unit tests, various knowledge base articles and so on. After all, solving a specific test / development task, the specialist does not need all the project documentation, he needs information on the part of the product on which he is currently working.
Thus, we guarantee that the team has the same idea about the product and any team member can quickly get the information he needs. The team should not waste time explaining everything to each new employee from scratch, and the employee should not waste time reading a bunch of documentation, from which no more than 30% is needed to solve his current task.
Graphic quality control model
This is also the case for the test coverage map. It logically continues the quality control model and allows you to cover the product with tests in a format in which the functionality is being developed. If it was decided to modify some functionality or even rework, then the test coverage can be updated quickly, since it is presented in the form of an understandable tree model.
There was a lot of discussion about how much time it takes. For example, I spent about 15 minutes creating a test coverage map, which is shown here.
Test coverage represented by the test list
Test coverage using a tree model (test coverage maps)
Tell us more about the results of the implementation of the developed tools.
VS: An example from life: I took a test coverage of one of the components, made by an experienced colleague-tester, where there were about 70 tests, and broke it as shown on the test coverage map. About 15 missed tests were found, and the whole work took about 30 minutes.
Over time, each product you begin to see a similar model, and any activity associated with testing the product, become structured and consistent. This is especially true for beginners. I remember how I started writing tests and could not understand which way I should approach testing, how to cover the product with tests so that I would not miss anything. If I had such tools then, I would start working as a full-fledged employee much faster.
We also conducted an experiment. The newcomer was provided with an unstructured and partially obsolete test coverage and an example with this card. Having an example at hand, a test coverage chart, and asking some questions for a hike, a person who had no experience in testing could create a good test coverage within 2 days. He reached the level of a full-time testing engineer within a month, i.e. a month later he was already fully working with the team, while it was about testing complex telecom systems.
Another example of using the model + card bundle is our experience in communication with a technical writer. Starting to write user documentation for a practically finished product, he first decided to figure out how this product is structured in order to document it in a structured way. I also gave him a model + card and made a few comments. He instantly figured out the device he needed part of the product. He liked the model so much that he included it in the official product documentation.
That is, the use of our developments allows us to quickly delve into the projects of customers and quickly adapt to the system the customer’s specialists who work with us.
And what about the lack of tools for testing and analyzing bugs?
VS: As all testers know, there are a large number of bug analysis tools. Some of them are better, some are worse, but in general, only the ability to use several tools at once can give a good result.
If the available tools do not allow us to solve the tasks before us, we create them ourselves.
As an example, I can give a technique for analyzing the logs of any application under test during the test. Logs are the main source of information about what happened in the application when the tester found a bug.
The tools we have created allow us to efficiently collect logs for all the necessary components and provide a clear cut to the developer, which will allow him to find out what the defect is. The engineer can see the complete picture at once, and not collect it in parts.
The lowest level of our developments is testing automation, i.e. We invest a comprehensive model of quality control, a test coverage map and the tools we create in our auto tests. Thus, the problems of routine work and regression bugs are solved simultaneously.
We have identified three basic stages of the AutoTest:
- environment preparation for testing
- testing
- undo changes
At the first stage, we launch the application, make certain settings in it and prepare the implementation of the functional part. For example, if we test the game menu, we need to start the application, open this menu and continue to perform testing operations in it.
The second stage is directly the testing operations themselves - those actions that will help us understand whether the functionality that we are testing is working correctly or is wrong.
The third stage is especially important for autotests. This is a cancellation of the changes made. Each test, incl. and automatic, must perform some testing operations in a test environment, leaving it unchanged after completion of the work. Because the key requirement for quality testing is to perform test operations in an unchanged environment and in steps like the user will do.
Thus, our testing experience solves all the listed problems of testers.
Let's talk more about the system. What other benefits of the Automated QA System can be highlighted?
VS: For IT professionals, the advantages are our integrated quality control model , which allows you to systematize and structure communications on a project, which will optimize the testing and development process.
From my point of view, our constructor platform will also be of interest. I am well understood by people who are familiar with Linux. As we all know, Linux is a large complex of small tools, each of which performs a separate function, but if the tools are used together, they can solve much larger tasks. According to this principle, I built our platform constructor, which we call the Automated QA System .
Our autotests are used for large-scale testing tasks . As an example, I’ll cite our functional autotests from the Star Crusade project, which, starting with simple scripts for testing maps, have been supplemented with new tools and are now a tool for testing the entire product: the client and the server. Star Crusade has many cards, the parameters of which are constantly changing as they are balanced, which leads to the need for constant testing of the product.
When developing tests for the project, we chose between two options:
1) stable autotests that can independently handle certain map changes
2) tests that need to be changed manually, but which are also easy to create and update.
As a result, we stopped at the second version, since working with autotests and observing game scenarios through the client, the tester remains constantly immersed in this environment. He starts to try, experiment, find new areas, non-standard bugs and can quickly fix all these scenarios with an autotest.
This further underlines the fact that testers use autotests as a diagnostic tool to control the quality of the entire product, and not just to solve a local task.
For professionals, a huge advantage will be the task management mechanism , which is implemented in the Automated QA System. To test large projects, each time you first need to download a fresh build of the product, deploy it in a test environment, perform a number of specific settings, and only then you can run the autotests directly. The task management mechanism allows you to efficiently create such tasks, automate them and launch them in the sequence you need for the tester.
We also use automation by macromodules — test modules that perform a more global task than a regular module.
For comparison, I will give the usual object in programming, which performs a certain number of tasks. The macromodule in this case will be the part of the test that performs the complex task. For example, making certain product settings using the web interface or using the server-side functionality.
The advantage of Automated QA System is the possibility of hybrid automation . Using macromodules, we can create hybrid tests that perform operations in completely different environments (managing the graphical part of the product, performing operations on Windows workstations, Unix servers, mobile devices, or in the web interface). All these operations can be used within one test, organize the exchange of information between macromodules and provide a centralized result on the performance of such a test.
What benefits does the system offer to customers?
VS: First, we provide software package quality control services. This means that we provide comprehensive quality control of our client’s project. The model has the experience of the testing team for more than 10 years. This is quite a long period during which we have developed practices, approaches, revealed a large number of “bottlenecks”, systematized and consolidated the experience gained, and came to the quality control model that I described. Only through integrated quality control can the project achieve the best quality.
Secondly, our automated testing system and our manual testing model (which includes a test coverage map and a graphical quality control model), along with structured tests, allow us to quickly provide the results to the customer . This means that the tests that we write and conduct, initially represent a structured tree model, from which it is easy to create a report in manual or automatic mode and present it to the customer. Our automated exit tests give a result in a relatively understandable format for people, i.e. The results of the execution of most tests can be assessed by the manager independently. They do not contain a large amount of technical data, which are understandable only to specialists.
Thirdly, in the case of automatic tests, it is possible to monitor the runs of automatic test packages of the product online . To do this, we have implemented a web interface to which we can provide the customer with access. The customer has the ability to track also various metrics: graphs with the dynamics of the so-called. pass rate - the percentage of successful test execution, the dynamics of adding automatic tests and a number of other trends, incl. we can add the necessary graphs and organize the collection of information on these graphs.
The dynamics of passing tests (pass-rate trends)
This means that our system is expandable and upgradeable.
Thus, the customer receives a comprehensive test of the entire project and the opportunity to receive detailed reports on the tests carried out both in offline and online modes.
We also provide the opportunity to work together with our customer's team, which allows you to share experiences.
Could you please give some practical examples of using the Automated QA System?
VS: Our automation experience began with testing complex telecom systems, where key functionality worked on a linux server, and user interaction was carried out via web interfaces. We collected most of our experience on such projects.
At the moment we are actively using the model described above on our flagship project Star Crusade. We use automated functional tests to test the server side of the product. We test the server part with the help of controlled bots that perform certain operations on the server using the AutoTest command.
We test actions on the playing field in the client part by watching the AutoTest run through a special observer mode, a functional that will also be a game. When the product interface comes to the "finish line" we fix it with autotests using graphical vision technologies, when the test performs operations in the client as well, as the end user would do. From the point of view of testing, this is an ideal option, because, receiving commands from mouse or keyboard drivers, the application is controlled by the same interfaces that the user will control.
And we are testing web services (website, server-side web interfaces) with the help of Selenium, while we also have solutions that simplify Selenium automation and make it more efficient. Thus, all functional tests are combined into our testing package.
The second level of testing is system tests. System tests are definitely hybrid testing, i.e. management of several tools within one test. Here we use the same macromodule constructor. For example, within the framework of one test, we perform certain settings on the server using the Linux console and the web interface, check the necessary mechanics using controlled bots, and complete the testing using graphical vision. The key here is the ability to quickly change autotests.
For example, we are testing the ability of a user to register on the site, confirm his e-mail account, and then enter the game under his account and play some kind of game game. This can be done in different ways. We can register on the site using the web interface explicitly, or we can send requests to the server at a low level. The output will be the same. But in the first case, we also test the site, its web interface, and in the second, we simply quickly perform the necessary operation, without concentrating on the web interface of the site.
With the use of controlled bots we play the game in the game. In this case, we quickly and reliably test the server part of the product to a greater extent. It is also possible to test the product through the client using a graphical view, then we also check the client part. In the created autotest, we can switch various functional elements. For example, the first time we test using the product settings via the web interface and run the tests with the help of guided bots, getting a centralized output of the result. And another time, we are already concentrating on the graphic part of the product, and setting up the product by direct request to the server and playing the game through the client application.
Switching between modules occurs by changing one or two options in automatic tests. Auto tests perfectly complement other tools used by our testers.
Automated QA System :
1. ,
2.
3.
4.
5.
Automated QA System . ?
..: Automated QA System:
match-3. Lil Quest , Muffin Quest - – , . . , , .
– . , -. , .
. , .
, - , . . , -, , , , . , StarCrusade .
, : , . , - .
. , .. . , , .
, , . , .
.
!
Source: https://habr.com/ru/post/259533/
All Articles