End-to-end testing: what, why, why
Testing in large companies in an enterprise is often difficult and ungrateful. The gap between business units and IT is huge: when a developer has a vision at the level of code, and verification is at the level of unit tests, and the customer thinks working or non-working not even with services, but with entire processes beyond one development team, or even the whole divisions of the company. And asks to organize business testing, or end-to-end testing, or testing based on scenarios from beginning to end (end 2 end).
Let's start from the very beginning - from two pillars, where did this notorious “end-to-end business testing” come from, namely, the testing pyramid and the ISO9000 standard.
Testing pyramid is surely familiar to any tester who has become skilled in his profession and has crammed when communicating with adjacent departments. Especially often it is necessary to appeal to her when justifying test automation. What tests are cheaper and more important to develop? And run?
')
The essence of the pyramid of testing is not tricky: at the heart of testing, you should use the simplest and fastest tests in writing and execution - unit tests. Of course, checking the interfaces of classes and functions is hardly the thing that can be shown to the customer, but without this solid, monolithic, trouble-free foundation, it is unlikely that it will be possible to build something higher. As a rule, several dozen functions, methods, classes implement some kind of functionality for the customer, and in fact a dozen of unit tests can be reduced to some high-level tests. The customer already needs a beautiful apartment with decoration, but it is unlikely that he will be satisfied when the skewed windows in his apartment stop opening and the floor and ceiling go cracked from the first breeze that blows. However, the customer himself can go into the apartment and check its quality may not be the best idea. Agree, it is difficult for the user to check the quality of concrete in the foundation, and to reproduce all weather conditions. So in testing, of course, high-level testing is needed, then only when we have completed the unit tests, and the tests are already of a higher level.
It is more difficult to develop and to perform longer, higher-level tests - integration tests that verify the correctness of the work of simultaneously working modules, on which the whole team worked, releasing their product (system). That is, the integration of the code is checked, the system is tested without interaction with external systems. Such tests already imply a high-level check, most likely through a system call via the system API or even a GUI (front). Working with this type of test is more difficult - to cover all the branches and nuances of the code, you most likely need to use a large number of highly intersecting checks on various test data, and during automation often develop a whole bunch of conditions and branching in the scripts. That is, on the one hand, we have already approached the user, complicating our lives, but on the other hand, it is still difficult for us to find a common language, it costs us more, and the quality of checks is still not enough. That is, we can launch the customer to a new apartment, he can check everything, but without taking into account interaction with other tenants, weather conditions and public utilities. Agree, the sense of an ideal world, a model, in real life, as a rule, is not much.
If we add these conditions - we will see how our system interacts with external systems - suppliers and consumers, with our environment, that is, we will carry out system testing, it is easy to see that the complexity of testing will also increase. We will need to strive for the simultaneous operability of all interacting systems, albeit without the involvement of specialists in them. For now, it is enough for us to simply accept some data from our suppliers and transfer our data to our customers. In the correct sequence and format. The further fate of the data does not bother us. The main thing - our system works correctly in the right environment. And everything would be fine here - for our customer we can already have a full-scale demonstration, but in real life this is not all the success criteria for our development. Of course it's good that the customer got his apartment in a solid house, but if you have to get away from it, climbing over the barbed wire, then on a canoe on a lake with crocodiles, into the tents teeming with snakes, then maybe we have something wrong. did you do there?
Therefore, here the first idea for end-to-end testing is to check not only our environment, but also all the interconnected systems through which the data received or sent by our system passes. And this, in turn, means that we will have to combine several such "testing pyramids" among themselves. The construction of a fragile bridge, on which we will hold for the handle data that is valuable to the user.
That's just the question of how to do this? Who to do it? How to put together?
A series of standards describing quality management systems, including that any process in an organization should be described, documented, even if it is the process of issuing a rake to the janitor until the fall. And if so, that no process that runs inside the software used and developed in the organization, can not be described. The question is how to do it? Of course, the best description, from the point of view of BDD, is the description of the behavior of tests, under which the testing pyramid will lie. But we will immediately return to our dilemma with combining several pyramids with thin ropes from top to top, which our ropewalker customer and his users will walk without insurance.
Therefore, the easiest way to use abstractions is to create at least a process diagram, specify its inputs and outputs, make the process controllable and measurable, ensure interconnection with parent and child processes, as required by ISO9000
Business diagrams are best suited for this purpose, and standards such as UML, BPMN, ARIS, etc. are most often used. And the processes themselves become flowcharts with "cubes" strung on them. Between the "cubes" there is an interaction, in the standard BPMN is a flow of actions and a message flow. And this is exactly what we need!
Any company that wants to have a certificate and follows the ISO9000 standard has most likely acquired such schemes, and they are an integral part of high-level requirements. If good analysts work in the company, then, most likely, references to specific actions from the schemes will go down to low-level requirements. We need them.
In fact, in the diagrams, we can see the whole process, and understand which scenario we need to build, and which system / team to run with what data at what point.
I do not underestimate the work of developers who write literate code that forwards messages between different parts of software and hardware systems, but it’s impossible to keep everything in mind. And when the process is used in many other processes, it is better to have such a “map” with you for conducting competent testing, and even more so for building a test model.
So, we have two introductory - every team / system should have a pyramid of tests prepared - from the smallest unit tests to complex system tests, as well as the fact that we must describe the requirements in the form of business within the organization -processes. This fact will allow us to quickly respond to the customer, which business process works and at what point because of what it breaks, and when receiving defects from commercial operation, it is possible to quickly perform root cause analysis (analysis of root causes of defects). In theory.
But in practice, everything again falls on the tester - how to choose the right ones from a pile of tests, especially strangers, build them into a chain, and send the necessary data to the input of each system and verify it with the correctly defined expected result?
The easiest option is to initially develop tests based on business models, and to divide teams into projects that implement a particular business process. To do this, some testing management tools already have the ability to download BPMN schemes (for example, for HPE ALM - download is supported in XPDL format). HP ALM itself will break the scheme into a set of requirements (actions), and if desired, create a hierarchy of requirements (module Requirements-> Business Models). Further, our business is to cover the requirements with tests, and then build the requirements, and hence the tests in the chains covering our business process. These chains in HPE ALM are called “paths” (path), and allow you to see all combinations of sequences. If desired, the requirements, the chains can be immediately converted into tests.
But even if you do not use the testing tools, you still have to form chains from the business process. Especially considering the imperfection of the tools (not everything is so rosy), as well as the fact that, most likely, the test model will need to be assembled post-factum, and executed as a recourse at all by the “common team” not clinging to new projects.
How many ways can a squirrel walk to a bump?
In this case, we will need to open the tests of each team, find the requirements tied to the requirements in the business model, and build chains of them, retaining them in the “common space”. Creating a common space is some kind of substitute, but in any case it should be, even in the form of a barn book, excel, or a project area in the test management tool. If we talk again about HPE ALM, then the BPT (Business Process Testing) module is responsible for this functionality, which at the same time allows you to transfer the results of one test to the parameters of another. However, if you wish, and hard work on HPE ALM, this is possible and implemented through rebuilding test sets (Test set) into the execution flow (Execution flow). Then when you run the full set, the testers responsible for passing each of the components of the end-to-end scenario will be called in turn.
And, alas, testing controls alone are indispensable. From my practice, almost all tools have some fatal flaws, and therefore, if you get to the stage of automating testing for a business process, you will come to create a script that will pull tests in the right sequence.
As a result, two conclusions can be drawn:
1) for end-to-end scenarios, previously developed tests for each of the systems included in the chain (scenario) of a business process are used with high probability.
All complete test suites of a company can be represented in the form of a sparse matrix, where the columns are distributed for each system (for simplicity, system tests), and the rows are business processes. That is, for those or other business processes, it is necessary to select / create tests covering the business process, establish relationships. If there is no coverage, this is a reason to fill in the gaps in the test model, or to make sure that the quality is ensured by other levels of testing (integration testing, unit testing, code review and running it through analyzers).
2) A tool is needed to monitor, trace and update the business process for synchronization with the test model.
And if testing tools are more or less tolerably done with the creation of a test model, then everything is really very bad with updating, it is often easier to recreate the model, rather than trying to see the changes in the process and the test model. And the experience of real teams suggests that it is better to create a live visualization of architecture. The easiest way to do this is in the common area, using a simple whiteboard and stickers. Then, the teams that are involved in the business process can clearly see how the process is modified (links are removed and added, actions are removed and added). The main thing is that everyone has access to the board. Plus, note that if the process involves messages between systems, then, as a rule, at least there should be two tests from each system - for sending and receiving data. However, instead of stickers, you can use a whole Lego-city (from large blocks), or something even more creative. The main thing here is one language and one information space, which is very lacking in the enterprise.
Organizing visual and proper testing for business processes is a difficult and very expensive thing. Please note that E2E testing is not just an acceptance, user testing that the customer will perform, this is building a bridge, taking into account all the possible situations that the customer will follow and lead the users to follow.
Once again - E2E - this is not a walk on the Lada-Kalina across the bridge, and not even travel by two KamAZ. This is a complex engineering work, hanging bridges with sensors and conducting all possible checks and situations - at least a description of these scenarios.
Whether or not your company needs such an ideal finishing run is solely a matter of your goals and needs. Always, as with any testing, the potential risks from missed defects at this stage should be assessed, as well as the cost of preparation and carrying out end-to-end testing. Rate that from this will cost you more, and only then act. But in the case of end-to-end testing by business processes, it should be remembered that it does not make sense without a solid foundation in the form of 100% passrate unit tests (~ 90-100% coverage), without integration tests (~ 60-80% coverage, 90 100% passrate), without system tests (20-40% coverage, 80-100% passrate). Establishing success criteria (quality gates) is more of a requirement for the quality of the product being produced, the main thing to remember here is that the volume of E2E tests is only the tip of the pyramid (1-2% coverage, ~ 99% passrate), which should not be greater than its base, not at the same time be a plug of holes from the previous stages. This is an addition that is considered a priori closed in the previous stages.
The organization of such testing is mainly work on the preparation and synchronization of test cases and data (test analytics), as well as a set of organizational measures, the synchronization of teams in one place at one time on a working test site. Keeping this in mind, one should not try to show the customer “end-to-end testing” ahead of time, in order not to waste time at once a large number of people without all the working components assembled together.
PS The described tools, as well as practices - purely for example, the author did not set himself the goal of advertising products and reciting the only true approach to end-to-end testing.
Let's start from the very beginning - from two pillars, where did this notorious “end-to-end business testing” come from, namely, the testing pyramid and the ISO9000 standard.
Pyramid testing
Testing pyramid is surely familiar to any tester who has become skilled in his profession and has crammed when communicating with adjacent departments. Especially often it is necessary to appeal to her when justifying test automation. What tests are cheaper and more important to develop? And run?
')
The essence of the pyramid of testing is not tricky: at the heart of testing, you should use the simplest and fastest tests in writing and execution - unit tests. Of course, checking the interfaces of classes and functions is hardly the thing that can be shown to the customer, but without this solid, monolithic, trouble-free foundation, it is unlikely that it will be possible to build something higher. As a rule, several dozen functions, methods, classes implement some kind of functionality for the customer, and in fact a dozen of unit tests can be reduced to some high-level tests. The customer already needs a beautiful apartment with decoration, but it is unlikely that he will be satisfied when the skewed windows in his apartment stop opening and the floor and ceiling go cracked from the first breeze that blows. However, the customer himself can go into the apartment and check its quality may not be the best idea. Agree, it is difficult for the user to check the quality of concrete in the foundation, and to reproduce all weather conditions. So in testing, of course, high-level testing is needed, then only when we have completed the unit tests, and the tests are already of a higher level.
It is more difficult to develop and to perform longer, higher-level tests - integration tests that verify the correctness of the work of simultaneously working modules, on which the whole team worked, releasing their product (system). That is, the integration of the code is checked, the system is tested without interaction with external systems. Such tests already imply a high-level check, most likely through a system call via the system API or even a GUI (front). Working with this type of test is more difficult - to cover all the branches and nuances of the code, you most likely need to use a large number of highly intersecting checks on various test data, and during automation often develop a whole bunch of conditions and branching in the scripts. That is, on the one hand, we have already approached the user, complicating our lives, but on the other hand, it is still difficult for us to find a common language, it costs us more, and the quality of checks is still not enough. That is, we can launch the customer to a new apartment, he can check everything, but without taking into account interaction with other tenants, weather conditions and public utilities. Agree, the sense of an ideal world, a model, in real life, as a rule, is not much.
If we add these conditions - we will see how our system interacts with external systems - suppliers and consumers, with our environment, that is, we will carry out system testing, it is easy to see that the complexity of testing will also increase. We will need to strive for the simultaneous operability of all interacting systems, albeit without the involvement of specialists in them. For now, it is enough for us to simply accept some data from our suppliers and transfer our data to our customers. In the correct sequence and format. The further fate of the data does not bother us. The main thing - our system works correctly in the right environment. And everything would be fine here - for our customer we can already have a full-scale demonstration, but in real life this is not all the success criteria for our development. Of course it's good that the customer got his apartment in a solid house, but if you have to get away from it, climbing over the barbed wire, then on a canoe on a lake with crocodiles, into the tents teeming with snakes, then maybe we have something wrong. did you do there?
Therefore, here the first idea for end-to-end testing is to check not only our environment, but also all the interconnected systems through which the data received or sent by our system passes. And this, in turn, means that we will have to combine several such "testing pyramids" among themselves. The construction of a fragile bridge, on which we will hold for the handle data that is valuable to the user.
That's just the question of how to do this? Who to do it? How to put together?
ISO9000
A series of standards describing quality management systems, including that any process in an organization should be described, documented, even if it is the process of issuing a rake to the janitor until the fall. And if so, that no process that runs inside the software used and developed in the organization, can not be described. The question is how to do it? Of course, the best description, from the point of view of BDD, is the description of the behavior of tests, under which the testing pyramid will lie. But we will immediately return to our dilemma with combining several pyramids with thin ropes from top to top, which our ropewalker customer and his users will walk without insurance.
The process approach is a management strategy to manage the processes between them. Thus, they need to consider these processes and their supporting processes.
Therefore, the easiest way to use abstractions is to create at least a process diagram, specify its inputs and outputs, make the process controllable and measurable, ensure interconnection with parent and child processes, as required by ISO9000
All processes have:
• inputs;
• outputs;
• operational control;
• appropriate measurement & monitoring.
Each process will have support processes
Business diagrams are best suited for this purpose, and standards such as UML, BPMN, ARIS, etc. are most often used. And the processes themselves become flowcharts with "cubes" strung on them. Between the "cubes" there is an interaction, in the standard BPMN is a flow of actions and a message flow. And this is exactly what we need!
Any company that wants to have a certificate and follows the ISO9000 standard has most likely acquired such schemes, and they are an integral part of high-level requirements. If good analysts work in the company, then, most likely, references to specific actions from the schemes will go down to low-level requirements. We need them.
In fact, in the diagrams, we can see the whole process, and understand which scenario we need to build, and which system / team to run with what data at what point.
I do not underestimate the work of developers who write literate code that forwards messages between different parts of software and hardware systems, but it’s impossible to keep everything in mind. And when the process is used in many other processes, it is better to have such a “map” with you for conducting competent testing, and even more so for building a test model.
What to practice
So, we have two introductory - every team / system should have a pyramid of tests prepared - from the smallest unit tests to complex system tests, as well as the fact that we must describe the requirements in the form of business within the organization -processes. This fact will allow us to quickly respond to the customer, which business process works and at what point because of what it breaks, and when receiving defects from commercial operation, it is possible to quickly perform root cause analysis (analysis of root causes of defects). In theory.
But in practice, everything again falls on the tester - how to choose the right ones from a pile of tests, especially strangers, build them into a chain, and send the necessary data to the input of each system and verify it with the correctly defined expected result?
The easiest option is to initially develop tests based on business models, and to divide teams into projects that implement a particular business process. To do this, some testing management tools already have the ability to download BPMN schemes (for example, for HPE ALM - download is supported in XPDL format). HP ALM itself will break the scheme into a set of requirements (actions), and if desired, create a hierarchy of requirements (module Requirements-> Business Models). Further, our business is to cover the requirements with tests, and then build the requirements, and hence the tests in the chains covering our business process. These chains in HPE ALM are called “paths” (path), and allow you to see all combinations of sequences. If desired, the requirements, the chains can be immediately converted into tests.
But even if you do not use the testing tools, you still have to form chains from the business process. Especially considering the imperfection of the tools (not everything is so rosy), as well as the fact that, most likely, the test model will need to be assembled post-factum, and executed as a recourse at all by the “common team” not clinging to new projects.
How many ways can a squirrel walk to a bump?
In this case, we will need to open the tests of each team, find the requirements tied to the requirements in the business model, and build chains of them, retaining them in the “common space”. Creating a common space is some kind of substitute, but in any case it should be, even in the form of a barn book, excel, or a project area in the test management tool. If we talk again about HPE ALM, then the BPT (Business Process Testing) module is responsible for this functionality, which at the same time allows you to transfer the results of one test to the parameters of another. However, if you wish, and hard work on HPE ALM, this is possible and implemented through rebuilding test sets (Test set) into the execution flow (Execution flow). Then when you run the full set, the testers responsible for passing each of the components of the end-to-end scenario will be called in turn.
And, alas, testing controls alone are indispensable. From my practice, almost all tools have some fatal flaws, and therefore, if you get to the stage of automating testing for a business process, you will come to create a script that will pull tests in the right sequence.
As a result, two conclusions can be drawn:
1) for end-to-end scenarios, previously developed tests for each of the systems included in the chain (scenario) of a business process are used with high probability.
All complete test suites of a company can be represented in the form of a sparse matrix, where the columns are distributed for each system (for simplicity, system tests), and the rows are business processes. That is, for those or other business processes, it is necessary to select / create tests covering the business process, establish relationships. If there is no coverage, this is a reason to fill in the gaps in the test model, or to make sure that the quality is ensured by other levels of testing (integration testing, unit testing, code review and running it through analyzers).
2) A tool is needed to monitor, trace and update the business process for synchronization with the test model.
And if testing tools are more or less tolerably done with the creation of a test model, then everything is really very bad with updating, it is often easier to recreate the model, rather than trying to see the changes in the process and the test model. And the experience of real teams suggests that it is better to create a live visualization of architecture. The easiest way to do this is in the common area, using a simple whiteboard and stickers. Then, the teams that are involved in the business process can clearly see how the process is modified (links are removed and added, actions are removed and added). The main thing is that everyone has access to the board. Plus, note that if the process involves messages between systems, then, as a rule, at least there should be two tests from each system - for sending and receiving data. However, instead of stickers, you can use a whole Lego-city (from large blocks), or something even more creative. The main thing here is one language and one information space, which is very lacking in the enterprise.
Finally
Organizing visual and proper testing for business processes is a difficult and very expensive thing. Please note that E2E testing is not just an acceptance, user testing that the customer will perform, this is building a bridge, taking into account all the possible situations that the customer will follow and lead the users to follow.
Once again - E2E - this is not a walk on the Lada-Kalina across the bridge, and not even travel by two KamAZ. This is a complex engineering work, hanging bridges with sensors and conducting all possible checks and situations - at least a description of these scenarios.
Whether or not your company needs such an ideal finishing run is solely a matter of your goals and needs. Always, as with any testing, the potential risks from missed defects at this stage should be assessed, as well as the cost of preparation and carrying out end-to-end testing. Rate that from this will cost you more, and only then act. But in the case of end-to-end testing by business processes, it should be remembered that it does not make sense without a solid foundation in the form of 100% passrate unit tests (~ 90-100% coverage), without integration tests (~ 60-80% coverage, 90 100% passrate), without system tests (20-40% coverage, 80-100% passrate). Establishing success criteria (quality gates) is more of a requirement for the quality of the product being produced, the main thing to remember here is that the volume of E2E tests is only the tip of the pyramid (1-2% coverage, ~ 99% passrate), which should not be greater than its base, not at the same time be a plug of holes from the previous stages. This is an addition that is considered a priori closed in the previous stages.
The organization of such testing is mainly work on the preparation and synchronization of test cases and data (test analytics), as well as a set of organizational measures, the synchronization of teams in one place at one time on a working test site. Keeping this in mind, one should not try to show the customer “end-to-end testing” ahead of time, in order not to waste time at once a large number of people without all the working components assembled together.
PS The described tools, as well as practices - purely for example, the author did not set himself the goal of advertising products and reciting the only true approach to end-to-end testing.
Source: https://habr.com/ru/post/417395/
All Articles