Report from SQA Days - Automation testing: discarding too much and checking the essence
We give the report of khroliz Igor Hrol, Wargaming company, Minsk, from the conference SQA Days 15.
Video report:
vimeo.com/93944414
Presentation:
www.slideshare.net/slideshow/embed_code/33725306#
')
I have been working in this industry for 8 years and I believe that we have problems)
First of all, testing is considered an uninteresting profession. What follows from this? Inexperienced people come to the industry, or not enough qualified. Difficult work, because inexperienced people, no one to do. And it turns out that we are doing a simple job. And since we do simple work, then we pay little, because We don’t bring as much business as we would like. And the circle closes.
As a result, if talented people get into testing, then by chance. And then they still go into development. That is, testing is considered a simple and uninteresting IT profession, and the quality of testing leaves much to be desired.
First of all, I urge you to remember: Who are we? Software testing engineer. Engineers. That is, we are not called testers, testers, etc., we are engineers. And I think that the situation can be changed by engineers with their technical skills, their approaches.
I would like to note that our work is more complicated than it seems at first glance. I'm talking about testing now, but for me testing and automation have merged into one, because I believe that automation should be the main tool of the tester.
The name plays a role.
• Test automation (we do something with our hands, and we automate it)
• Automatic testing (what does he want us to do business? So that testing goes on its own, automatically, and we looked at the logs and so on. The goal is not to automate testing, but to make testing automatic)
• Effective automated testing.
Consider an abstract project in a vacuum.
This is a model; it describes the life situation in some measure of approximation.
We have 5 modules. In the first, you need to test 5 options, in the second - 8, and the third - 2 options, etc. In total, 800 tests should be conducted to make sure that everything works.
Change the numbers to V: V1 * V2 * V3 * ... * Vn As a result, Vn is the average complexity of the module. What is Vn? Mathematical complexity and exponent. Everything is built on the fact that a problem with such complexity is solved very badly.
Exponential complexity. See graph.
The green graph, which for a long time holds below the others, but at some point goes up rapidly. And when we test with the black box method, when we try to sort through all the options, we get that the black box (the complexity of such testing) is exponential complexity. We have a lot of autotests that work for hours, what do we do next? Run them in parallel. We divide this complexity by a constant. We have the same exponent, but a slightly reduced constant. That is, it does not solve the problem.
Divide and rule
Let's look at our task from the other side.
We had 26 tests (if we test each module separately). But you will say that we need to make sure that all this works together. Well, let's test each connection with a separate test case and run some cross-cutting scenario. It turned out 31 test. As a result, we change the exponential complexity to linear. And the problem becomes effectively solved. That is, if we poorly tested, and hired 10 more testers, then we simply parallelized by 10, and if we changed the very complexity of the task, then we should not increase the resources required for testing so much.
If we have a certain abstract system, which has one option, and at the end - 10, if we multiply, then we have 10 test cases, and totally - 14. And plus more connections, check - 19. But the tests have become easier. Instead of one long test, we write 10 small ones. That is, in fact, the complexity of the tests is still simplified.
“Pyramid” automation.
If we do not build our tests on this “pyramid”, we will get the exponential complexity of our task, our testing. The smallest task decomposition module is separate classes, separate methods. The main thing is the decomposition of tasks. If it is monosyllabic, then we get multiplication, if not, then we get addition, and in the long run it will be better.
Testing is not so simple. Because in order to carry out such a decomposition of one black box into many smaller ones, it is necessary to sort it out to the end, right down to the lowest levels, in the technological stand in which you develop. Those. one must have qualifications commensurate with the qualifications of the developer. Without this, we run into an exhibitor, we do not do our work qualitatively better. Accordingly, it is necessary to learn, and due to this, it seems to me that our work is not as simple as it seems.
The project of life.
Siebel, Oracle is a tough enterprise solution, it was a project on customization, on the introduction of mobile operator solutions based on Siebel technologies. Before I joined the project, the tests were written on QTP. They were such a structure: there was an internal API, and the tests go, click on the buttons, in the middle there is a black box, and there is a database at the bottom. A pretty typical picture.
As a result: tests are long, tests are unstable. The task came: to optimize the preconditions that are made for hours.
I wrote a script in java, which sends a GET request, parses XML, looks at XML, sends another GET request, etc. Tests began to work for 20 minutes and began to parallel well.
As a result of abandoning the browser, tests began to work faster, there were no problems with synchronization, tests became more reliable and easy to run in parallel. Of the minuses: you can not see how the tests work; no confidence, and a higher entry threshold.
The question “not visible” was decided as in the old joke: “Do you want to go or go?” Do you want to look at the tests, or do you need to test it? Those. here is the question: do we want automatic testing or automate testing?
Solution structure: Java + TestNG + Maven + HttpClient. Put several levels of abstractions in order not to work directly, not to write queries, but to work with entities that are specific to Siebel.
But it turned out that the browser did not leave, and part of the logic was implemented in the browser itself. Those. part of the logic was on the server, and part of the logic was in the browser. Therefore, Selenium appeared. But knowing that the browser is long, unstable and fragile, we did it carefully. It was launched only when a browser is really needed, when logic appears in the test case, which is implemented in the browser and which we want to check (about 3% of scripts). A headless session is also used with browserless interaction.
Then decompose this problem into some pieces. It turned out that the layout of the browser is generated by Siebel himself. And it doesn’t matter to us that pressing the button is real, because it’s already developed in Siebel, it’s tested in Siebel, we just have to drop the extra and test only what we did.
It turned out that to develop browser logic there is an API (of. Oracle documentation): if you want to do something in the UI, call this method.
Database.
This is the fastest way to work with the system, but it’s better not to write directly there (but this was the specificity of our project). We made some changes in the database through special verified stored procedures, and this is well suited for inspections.
Web services
JAX-WS is a protocol in Java for working with a web service. Instead of testing our Siebel in the context of all other systems, we silenced all external interactions, and when we had to check that something was leaving Siebel, we looked at it not on any external systems, but on stubs. We also sent java requests for web services to check outgoing information.
Our server consists of two parts: a web server and an application server. The Web server is responsible for the layout and sending of responses, and the Application server for the application logic. And there is access to the logic through Java Data Bean. And through it, we got a complete programming interface for creating application logic in Siebel from Java.
Performance tests
Since Our tests are written in java, because they are well parallelized, we can simply “pitch” our tests on the JMeter, and in essence load the system with functional tests. Also tests were supported along with support for functional tests. Those. if new versions of scripts were needed for the new build, we still need to update these scripts for functional tests.
We figured out how our technology stack works, understand the interaction options, and we understood that the lower we go, the more quickly and reliably.
As a result, we received:
• quick tests
• predictable results that scale easily
• ability to interact with Siebel at any level
I would not like my report to be taken as an instruction: “How to work with Siebel“. I would like you to look at your technology stack and see what is “under the hood” because This makes our work better and more interesting.
A look at testing from the implementation of the system allows you to:
• reduce the complexity of the testing task itself
• reduce the complexity and length of scripts
• increase speed and stability
• find new areas for auto tests
Video report:
vimeo.com/93944414
Presentation:
www.slideshare.net/slideshow/embed_code/33725306#
')
I have been working in this industry for 8 years and I believe that we have problems)
First of all, testing is considered an uninteresting profession. What follows from this? Inexperienced people come to the industry, or not enough qualified. Difficult work, because inexperienced people, no one to do. And it turns out that we are doing a simple job. And since we do simple work, then we pay little, because We don’t bring as much business as we would like. And the circle closes.
As a result, if talented people get into testing, then by chance. And then they still go into development. That is, testing is considered a simple and uninteresting IT profession, and the quality of testing leaves much to be desired.
First of all, I urge you to remember: Who are we? Software testing engineer. Engineers. That is, we are not called testers, testers, etc., we are engineers. And I think that the situation can be changed by engineers with their technical skills, their approaches.
I would like to note that our work is more complicated than it seems at first glance. I'm talking about testing now, but for me testing and automation have merged into one, because I believe that automation should be the main tool of the tester.
The name plays a role.
• Test automation (we do something with our hands, and we automate it)
• Automatic testing (what does he want us to do business? So that testing goes on its own, automatically, and we looked at the logs and so on. The goal is not to automate testing, but to make testing automatic)
• Effective automated testing.
Consider an abstract project in a vacuum.
This is a model; it describes the life situation in some measure of approximation.
We have 5 modules. In the first, you need to test 5 options, in the second - 8, and the third - 2 options, etc. In total, 800 tests should be conducted to make sure that everything works.
Change the numbers to V: V1 * V2 * V3 * ... * Vn As a result, Vn is the average complexity of the module. What is Vn? Mathematical complexity and exponent. Everything is built on the fact that a problem with such complexity is solved very badly.
Exponential complexity. See graph.
The green graph, which for a long time holds below the others, but at some point goes up rapidly. And when we test with the black box method, when we try to sort through all the options, we get that the black box (the complexity of such testing) is exponential complexity. We have a lot of autotests that work for hours, what do we do next? Run them in parallel. We divide this complexity by a constant. We have the same exponent, but a slightly reduced constant. That is, it does not solve the problem.
Divide and rule
Let's look at our task from the other side.
We had 26 tests (if we test each module separately). But you will say that we need to make sure that all this works together. Well, let's test each connection with a separate test case and run some cross-cutting scenario. It turned out 31 test. As a result, we change the exponential complexity to linear. And the problem becomes effectively solved. That is, if we poorly tested, and hired 10 more testers, then we simply parallelized by 10, and if we changed the very complexity of the task, then we should not increase the resources required for testing so much.
If we have a certain abstract system, which has one option, and at the end - 10, if we multiply, then we have 10 test cases, and totally - 14. And plus more connections, check - 19. But the tests have become easier. Instead of one long test, we write 10 small ones. That is, in fact, the complexity of the tests is still simplified.
“Pyramid” automation.
If we do not build our tests on this “pyramid”, we will get the exponential complexity of our task, our testing. The smallest task decomposition module is separate classes, separate methods. The main thing is the decomposition of tasks. If it is monosyllabic, then we get multiplication, if not, then we get addition, and in the long run it will be better.
Testing is not so simple. Because in order to carry out such a decomposition of one black box into many smaller ones, it is necessary to sort it out to the end, right down to the lowest levels, in the technological stand in which you develop. Those. one must have qualifications commensurate with the qualifications of the developer. Without this, we run into an exhibitor, we do not do our work qualitatively better. Accordingly, it is necessary to learn, and due to this, it seems to me that our work is not as simple as it seems.
The project of life.
Siebel, Oracle is a tough enterprise solution, it was a project on customization, on the introduction of mobile operator solutions based on Siebel technologies. Before I joined the project, the tests were written on QTP. They were such a structure: there was an internal API, and the tests go, click on the buttons, in the middle there is a black box, and there is a database at the bottom. A pretty typical picture.
As a result: tests are long, tests are unstable. The task came: to optimize the preconditions that are made for hours.
I wrote a script in java, which sends a GET request, parses XML, looks at XML, sends another GET request, etc. Tests began to work for 20 minutes and began to parallel well.
As a result of abandoning the browser, tests began to work faster, there were no problems with synchronization, tests became more reliable and easy to run in parallel. Of the minuses: you can not see how the tests work; no confidence, and a higher entry threshold.
The question “not visible” was decided as in the old joke: “Do you want to go or go?” Do you want to look at the tests, or do you need to test it? Those. here is the question: do we want automatic testing or automate testing?
Solution structure: Java + TestNG + Maven + HttpClient. Put several levels of abstractions in order not to work directly, not to write queries, but to work with entities that are specific to Siebel.
But it turned out that the browser did not leave, and part of the logic was implemented in the browser itself. Those. part of the logic was on the server, and part of the logic was in the browser. Therefore, Selenium appeared. But knowing that the browser is long, unstable and fragile, we did it carefully. It was launched only when a browser is really needed, when logic appears in the test case, which is implemented in the browser and which we want to check (about 3% of scripts). A headless session is also used with browserless interaction.
Then decompose this problem into some pieces. It turned out that the layout of the browser is generated by Siebel himself. And it doesn’t matter to us that pressing the button is real, because it’s already developed in Siebel, it’s tested in Siebel, we just have to drop the extra and test only what we did.
It turned out that to develop browser logic there is an API (of. Oracle documentation): if you want to do something in the UI, call this method.
Database.
This is the fastest way to work with the system, but it’s better not to write directly there (but this was the specificity of our project). We made some changes in the database through special verified stored procedures, and this is well suited for inspections.
Web services
JAX-WS is a protocol in Java for working with a web service. Instead of testing our Siebel in the context of all other systems, we silenced all external interactions, and when we had to check that something was leaving Siebel, we looked at it not on any external systems, but on stubs. We also sent java requests for web services to check outgoing information.
Our server consists of two parts: a web server and an application server. The Web server is responsible for the layout and sending of responses, and the Application server for the application logic. And there is access to the logic through Java Data Bean. And through it, we got a complete programming interface for creating application logic in Siebel from Java.
Performance tests
Since Our tests are written in java, because they are well parallelized, we can simply “pitch” our tests on the JMeter, and in essence load the system with functional tests. Also tests were supported along with support for functional tests. Those. if new versions of scripts were needed for the new build, we still need to update these scripts for functional tests.
We figured out how our technology stack works, understand the interaction options, and we understood that the lower we go, the more quickly and reliably.
As a result, we received:
• quick tests
• predictable results that scale easily
• ability to interact with Siebel at any level
I would not like my report to be taken as an instruction: “How to work with Siebel“. I would like you to look at your technology stack and see what is “under the hood” because This makes our work better and more interesting.
A look at testing from the implementation of the system allows you to:
• reduce the complexity of the testing task itself
• reduce the complexity and length of scripts
• increase speed and stability
• find new areas for auto tests
Source: https://habr.com/ru/post/241772/
All Articles