Patterns and anti-patterns CI / CD. Part 3
Hello again. Today we want to share with you the translation of the third part of the article “Patterns and anti-patterns of CI / CD” , the previous parts of which can be found here and here . Recall, this series of publications is timed to the launch of a new stream on the course “DevOps practices and tools,” which set will close in the coming days.
1.3.5.1 Disadvantages of End-to-End Testing
“Any advantage of communicating with the present system is overshadowed by the need to eradicate non-determinism.” Martin Fowler.
')
End-to-End Testing is a practice in which a large number of automated end-to-end and manual regression tests are used during assembly along with a small number of automated modular and acceptance tests. Test Ratio in End-to-End Testing can be visualized as Ice Cream Cones.
End-to-End Testing often seems attractive due to the subjective benefits of end-to-end tests:
Given the above, one can understand why many organizations are implementing End-to-End Testing. As Don Reinertsen remarked: “ This combination of low cost and high reliability creates the illusion of the efficiency of such system tests .” However, the value proposition of End-to-End Testing is absolutely false due to the incorrectness of both statements:
Martin Fowler had previously warned that “ non-deterministic tests can completely destroy the value of an automated regression set ”, and Stephen Covey 's Control, Influence and Concern Cycles underscore how many factors of end-to-end test complicate the search for and determination of non-determinism. If different teams in the same Companies R Us organization owned Company Accounts and Ptyments service, the Company Accounts team would control their service in the pass-through test, but this would only affect the Payments service.
The time taken to improve the end-to-end test depends on the location of the change in the SUT. The Company Accounts team needs to analyze and implement this change in the service in a short time. However, the time required to make changes to the Payments service cannot be less than the time required for the Company Accounts team to convince the Payments service team to get down to business.
In another case, if a separate organization Payments R Us owned Payments service, it would be just a third-party service and not a problem for the Company Accounts team.
In this case, changing Payments service will require much more time, because the Company Accounts team does not affect the organization Payments R Us. Moreover, the Payments service may be unexpectedly updated, which will increase the non-determinism in the end-to-end tests of the Company Accounts and make it impossible to establish the predictable outcome of the test run.
Relating to End-to-End Testing is often a symptom of long-term underfunding, which makes the system fragile, unable to change, increases its delivery time and optimizes for Average Failure Time , and not Average Recovery Time . In such a system, it is impossible to accurately predict user experience and operational performance due to changes caused by external factors, and focusing on the probability of failure instead of cost of failure creates exposure to highly unlikely but very expensive events known as Black Swans, for example, Knights Capital lost $ 440 million in 45 minutes. So if Payments data center suffers from a catastrophic failure, all customer payments made by Company Accounts will not be executed.
The unavailability of Payments service on the fly will block the payments of Company Accounts clients, and the slow recovery of the service will make dissatisfied clients turn to other firms. If any payments are lost and this becomes known publicly, this will lead to a significant loss of customer confidence.
End-to-end testing is an expensive and restrictive strategy. A pass-through test does not check behavior, takes a long time to complete and will intermittently fail, so a test suite consisting mainly of cross-cutting will have poor coverage, slow execution time, and non-deterministic result. Disadvantages will be overlooked, feedback is slow and unreliable, maintenance costs will increase, and as a result, testers will have to rely on their own manual, end-to-end, regression testing. End-to-End Testing cannot provide short lead times and is completely incompatible with Continuous Delivery.
1.3.5.2 Benefits of Continuous Testing
“End the addiction to mass control. Eliminate the need for mass inspection, initially making quality a part of the product. ” Dr. V. Edwards Deming .
Continuous Delivery requires Continuous Testing - a strategy in which a large number of automated modular and acceptance tests are complemented by a small number of automated end-to-end and targeted research tests. The ratio of tests in Continuous Testing can be visualized in the form of a Pyramid - a kind of antithesis of Ice Cream Horn.
Continuous Testing is coordinated with Development through Testing and Development through Acceptance Tests. Speaking for cross-functional testing as part of a total commitment to quality, it embodies the principle of Continuous Delivery: Built-in Quality in the Process. However, Continuous Testing may seem complicated due to the imaginary flaws of unit and acceptance tests:
Many use End-to-End Testing from the incorrect assumption of high test coverage and low support costs, and at the same time think that Continuous Testing provides low test coverage and is expensive to maintain:
A non-deterministic acceptance test can be corrected much faster than a pass-through test, thanks to the sole owner of the SUT. If Companies R Us owned Company Accounts and Payments R Us were Payments services, then the Company Accounts acceptance test would use only services controlled by their team.
If the Company Accounts team tries to identify and resolve non-determinism in the acceptance test, they will be able to implement the necessary changes in a short time. Also, you will not have to fear unexpected changes in the Payments service, which may interfere with the acceptance test of the latest Company Accounts code. This will create a predictable level of testing.
End-to-end tests are part of Continuous Testing, not least because the idea of “testing individual parts of the system is testing the entire system” is Addition Error. A small number of automated end-to-end tests should be used to validate core user scenarios, but not during build, when unattended, dependent services are unreliable and unrepresentative. End-to-end tests should be used for testing snapshots during release and in-service monitoring with artificial transactions to simulate user activity. This approach increases confidence in releases for battle, so it needs to be combined with real-time monitoring of business and operational metrics to speed up feedback cycles and improve understanding of user behavior.
Continuous Delivery acknowledges that optimization under Average Recovery Time is more important than optimization under Average Time between Failures, as it allows an organization to minimize the impact of manufacturing defects and is easier to achieve. The cost of defects can be controlled, as under the Law of Little, short releases reduce the defect elimination period, and Continuous Testing provides the necessary infrastructure to reduce feedback cycles for frequent releases. The combination of Continuous Testing and Continuous Delivery, such as Blue Green Releases and Canary Releases, allows organizations to create a robust system that can neutralize unforeseen events, and advanced practices like Dark Launching and Chaos Engineering can lead to anti-frail systems that benefit from Black Swans. For example, if Chaos Engineering detects problems with Payments service, the Company Accounts team can make Dark Launch their Payments Stub into production and use it in unforeseen cases of the Payments failure of the data center.
While Payments data center will be offline, Company Accounts will gracefully switch to collecting payments from buyers in Payments Store until the Payments service starts working. This incident will not affect customers, and if competitors depend on the same third-party Payments service, this may be a strategic advantage in the market. Excessive operational capabilities may seem redundant, but Continuous Testing promotes operational excellence and, as Nassim Nicholas Taleb noted, “everything unusual happens.”
Continuous Testing can be an integrated and inexpensive testing strategy. According to Dave Farley and Jez Humble, “embedding quality means writing automated tests at several levels,” and the test suite, consisting mostly of modular and acceptance tests, contains carefully tested scenarios with a high degree of test coverage. , low runtime and predictable test results. So end-to-end tests can be left for smoke testing and production monitoring, and testers instead of manual regression testing can do more valuable things, such as research testing. This will lead to a reduction in the number of manufacturing defects, fast and reliable feedback, a reduction in preparation time for entering the market, and new revenue growth opportunities.
1.3.5.3 From End-to-End Testing to Continuous
“Test as deeply as possible for maximum return on investment and the fastest feedback.” Janet Gregory and Lisa Crispin .
The transition from End-to-End Testing to Continuous is a long-term investment, based on the notion that end-to-end tests can be found at the bottom of the Testing Pyramid, if we designate their areas of responsibility as follows:
Suppose the Company Accounts service depends on the Payments endpoint of the Payments service, which accepts the company id and payment amount, and then issues a confirmation code and the number of days until the payment is made. The Company Accounts service sends the request fields id and the size of the payment and depends on the response fields.
The connection between services can be tested modularly with the help of test twins, which will allow the Company Accounts service to test its reaction to the different behavior of the Payments service. Company Accounts unit tests will replace the connector with a Mock or Stab connector for correct processing of scenarios such as an unexpected Pay timeout.
The dialogue between services can be tested modularly using Consumer Driven Contracts, which will enable Company Accounts to continuously check its interactions with Payments. Payments will issue a Provider Contract describing the Pay API at build time, and Company Accounts will return a Consumer Contract describing usage. The Payments service will create a Consumer Driven Contract, which will be checked during each build.
Since Company Accounts does not use the days field, it will be excluded from the Consumer Contract and the Consumer Driven Contract. Therefore, the assembly of Payments service, with the removal of days or the addition of a new field for response comments will be successful. And if the response code field is deleted, the Consumer Driven Contract will fail, and the Payments team will have to agree on a different approach with the Company Accounts team.
The behavior of the services can be tested modularly using API Examples, which will allow the Company Accounts service to check the behavior changes in new Payments releases. Each of its releases will be accompanied by an artifact containing examples of the request and response APIs for the Pay endpoint, which is connected to the Company Accounts unit tests as representative test data and to warn you about changes in behavior.
If the new version of the Payments service changes the format of the response code field from alphanumeric to numeric, the Company Accounts service will fail during assembly, indicating a change in behavior within the Payments service and the need to discuss this issue for both teams.
1.3.5.4 Conclusion
“System Testing will not only not detect all the bugs, but it will also require more time and money - more than you could save by skipping effective acceptance testing” - Jerry Weinberg .
End-to-end testing attracts organizations with promises of high test coverage and low support costs. But the continued use of automated end-to-end and manual regression tests will only lead to a fragile system with slow and unreliable feedback, which will increase delivery time and will be incompatible with Continuous Delivery. Continuous Testing requires preliminary and ongoing investment in test automation, but a comprehensive set of automated unit and acceptance tests will provide fast, deterministic feedback that reduces manufacturing defects and delivery time, and also stimulates Continuous Delivery of reliable and anti-fragile systems.
1.3.5.1 Disadvantages of End-to-End Testing
“Any advantage of communicating with the present system is overshadowed by the need to eradicate non-determinism.” Martin Fowler.
')
End-to-End Testing is a practice in which a large number of automated end-to-end and manual regression tests are used during assembly along with a small number of automated modular and acceptance tests. Test Ratio in End-to-End Testing can be visualized as Ice Cream Cones.
End-to-End Testing often seems attractive due to the subjective benefits of end-to-end tests:
- A pass-through test maximizes System Under Tests, which suggests a high degree of test coverage;
- End-to-end test uses the system itself as a test client, which implies low costs for the test infrastructure.
Given the above, one can understand why many organizations are implementing End-to-End Testing. As Don Reinertsen remarked: “ This combination of low cost and high reliability creates the illusion of the efficiency of such system tests .” However, the value proposition of End-to-End Testing is absolutely false due to the incorrectness of both statements:
- The very idea that testing a whole system involves checking components is Decomposition Error . Verifying compliance with requirements is not the same as verifying implementation for purposes. This means that the pass-through test will check the interactions between the code paths, but not the behavior of these paths.
- The very idea that testing the whole system will be cheaper than testing its components - the Error of Cheap Investments . Test execution time and non-determinism are directly proportional to the SUT volume. This means that the pass-through test will be slow and prone to non-determinism.
Martin Fowler had previously warned that “ non-deterministic tests can completely destroy the value of an automated regression set ”, and Stephen Covey 's Control, Influence and Concern Cycles underscore how many factors of end-to-end test complicate the search for and determination of non-determinism. If different teams in the same Companies R Us organization owned Company Accounts and Ptyments service, the Company Accounts team would control their service in the pass-through test, but this would only affect the Payments service.
The time taken to improve the end-to-end test depends on the location of the change in the SUT. The Company Accounts team needs to analyze and implement this change in the service in a short time. However, the time required to make changes to the Payments service cannot be less than the time required for the Company Accounts team to convince the Payments service team to get down to business.
In another case, if a separate organization Payments R Us owned Payments service, it would be just a third-party service and not a problem for the Company Accounts team.
In this case, changing Payments service will require much more time, because the Company Accounts team does not affect the organization Payments R Us. Moreover, the Payments service may be unexpectedly updated, which will increase the non-determinism in the end-to-end tests of the Company Accounts and make it impossible to establish the predictable outcome of the test run.
Relating to End-to-End Testing is often a symptom of long-term underfunding, which makes the system fragile, unable to change, increases its delivery time and optimizes for Average Failure Time , and not Average Recovery Time . In such a system, it is impossible to accurately predict user experience and operational performance due to changes caused by external factors, and focusing on the probability of failure instead of cost of failure creates exposure to highly unlikely but very expensive events known as Black Swans, for example, Knights Capital lost $ 440 million in 45 minutes. So if Payments data center suffers from a catastrophic failure, all customer payments made by Company Accounts will not be executed.
The unavailability of Payments service on the fly will block the payments of Company Accounts clients, and the slow recovery of the service will make dissatisfied clients turn to other firms. If any payments are lost and this becomes known publicly, this will lead to a significant loss of customer confidence.
End-to-end testing is an expensive and restrictive strategy. A pass-through test does not check behavior, takes a long time to complete and will intermittently fail, so a test suite consisting mainly of cross-cutting will have poor coverage, slow execution time, and non-deterministic result. Disadvantages will be overlooked, feedback is slow and unreliable, maintenance costs will increase, and as a result, testers will have to rely on their own manual, end-to-end, regression testing. End-to-End Testing cannot provide short lead times and is completely incompatible with Continuous Delivery.
1.3.5.2 Benefits of Continuous Testing
“End the addiction to mass control. Eliminate the need for mass inspection, initially making quality a part of the product. ” Dr. V. Edwards Deming .
Continuous Delivery requires Continuous Testing - a strategy in which a large number of automated modular and acceptance tests are complemented by a small number of automated end-to-end and targeted research tests. The ratio of tests in Continuous Testing can be visualized in the form of a Pyramid - a kind of antithesis of Ice Cream Horn.
Continuous Testing is coordinated with Development through Testing and Development through Acceptance Tests. Speaking for cross-functional testing as part of a total commitment to quality, it embodies the principle of Continuous Delivery: Built-in Quality in the Process. However, Continuous Testing may seem complicated due to the imaginary flaws of unit and acceptance tests:
- A modular or acceptance test minimizes SUT, which implies low test coverage;
- The modular or acceptance test uses its own test client, which implies high costs for the test infrastructure.
Many use End-to-End Testing from the incorrect assumption of high test coverage and low support costs, and at the same time think that Continuous Testing provides low test coverage and is expensive to maintain:
- The unit test verifies that the goal is met, and the acceptance test verifies that the implementation meets the requirements. This means that all the ways of execution and their interaction with each other can be checked.
- The unit test limits the amount of SUT to a single execution path, and the acceptance test is limited to a single service. So both can have a minimum execution time and deterministic results.
A non-deterministic acceptance test can be corrected much faster than a pass-through test, thanks to the sole owner of the SUT. If Companies R Us owned Company Accounts and Payments R Us were Payments services, then the Company Accounts acceptance test would use only services controlled by their team.
If the Company Accounts team tries to identify and resolve non-determinism in the acceptance test, they will be able to implement the necessary changes in a short time. Also, you will not have to fear unexpected changes in the Payments service, which may interfere with the acceptance test of the latest Company Accounts code. This will create a predictable level of testing.
End-to-end tests are part of Continuous Testing, not least because the idea of “testing individual parts of the system is testing the entire system” is Addition Error. A small number of automated end-to-end tests should be used to validate core user scenarios, but not during build, when unattended, dependent services are unreliable and unrepresentative. End-to-end tests should be used for testing snapshots during release and in-service monitoring with artificial transactions to simulate user activity. This approach increases confidence in releases for battle, so it needs to be combined with real-time monitoring of business and operational metrics to speed up feedback cycles and improve understanding of user behavior.
Continuous Delivery acknowledges that optimization under Average Recovery Time is more important than optimization under Average Time between Failures, as it allows an organization to minimize the impact of manufacturing defects and is easier to achieve. The cost of defects can be controlled, as under the Law of Little, short releases reduce the defect elimination period, and Continuous Testing provides the necessary infrastructure to reduce feedback cycles for frequent releases. The combination of Continuous Testing and Continuous Delivery, such as Blue Green Releases and Canary Releases, allows organizations to create a robust system that can neutralize unforeseen events, and advanced practices like Dark Launching and Chaos Engineering can lead to anti-frail systems that benefit from Black Swans. For example, if Chaos Engineering detects problems with Payments service, the Company Accounts team can make Dark Launch their Payments Stub into production and use it in unforeseen cases of the Payments failure of the data center.
While Payments data center will be offline, Company Accounts will gracefully switch to collecting payments from buyers in Payments Store until the Payments service starts working. This incident will not affect customers, and if competitors depend on the same third-party Payments service, this may be a strategic advantage in the market. Excessive operational capabilities may seem redundant, but Continuous Testing promotes operational excellence and, as Nassim Nicholas Taleb noted, “everything unusual happens.”
Continuous Testing can be an integrated and inexpensive testing strategy. According to Dave Farley and Jez Humble, “embedding quality means writing automated tests at several levels,” and the test suite, consisting mostly of modular and acceptance tests, contains carefully tested scenarios with a high degree of test coverage. , low runtime and predictable test results. So end-to-end tests can be left for smoke testing and production monitoring, and testers instead of manual regression testing can do more valuable things, such as research testing. This will lead to a reduction in the number of manufacturing defects, fast and reliable feedback, a reduction in preparation time for entering the market, and new revenue growth opportunities.
1.3.5.3 From End-to-End Testing to Continuous
“Test as deeply as possible for maximum return on investment and the fastest feedback.” Janet Gregory and Lisa Crispin .
The transition from End-to-End Testing to Continuous is a long-term investment, based on the notion that end-to-end tests can be found at the bottom of the Testing Pyramid, if we designate their areas of responsibility as follows:
- Communication — can services connect to each other;
- Dialogue - can services communicate with each other;
- Behavior - can services respond to each other?
Suppose the Company Accounts service depends on the Payments endpoint of the Payments service, which accepts the company id and payment amount, and then issues a confirmation code and the number of days until the payment is made. The Company Accounts service sends the request fields id and the size of the payment and depends on the response fields.
The connection between services can be tested modularly with the help of test twins, which will allow the Company Accounts service to test its reaction to the different behavior of the Payments service. Company Accounts unit tests will replace the connector with a Mock or Stab connector for correct processing of scenarios such as an unexpected Pay timeout.
The dialogue between services can be tested modularly using Consumer Driven Contracts, which will enable Company Accounts to continuously check its interactions with Payments. Payments will issue a Provider Contract describing the Pay API at build time, and Company Accounts will return a Consumer Contract describing usage. The Payments service will create a Consumer Driven Contract, which will be checked during each build.
Since Company Accounts does not use the days field, it will be excluded from the Consumer Contract and the Consumer Driven Contract. Therefore, the assembly of Payments service, with the removal of days or the addition of a new field for response comments will be successful. And if the response code field is deleted, the Consumer Driven Contract will fail, and the Payments team will have to agree on a different approach with the Company Accounts team.
The behavior of the services can be tested modularly using API Examples, which will allow the Company Accounts service to check the behavior changes in new Payments releases. Each of its releases will be accompanied by an artifact containing examples of the request and response APIs for the Pay endpoint, which is connected to the Company Accounts unit tests as representative test data and to warn you about changes in behavior.
If the new version of the Payments service changes the format of the response code field from alphanumeric to numeric, the Company Accounts service will fail during assembly, indicating a change in behavior within the Payments service and the need to discuss this issue for both teams.
1.3.5.4 Conclusion
“System Testing will not only not detect all the bugs, but it will also require more time and money - more than you could save by skipping effective acceptance testing” - Jerry Weinberg .
End-to-end testing attracts organizations with promises of high test coverage and low support costs. But the continued use of automated end-to-end and manual regression tests will only lead to a fragile system with slow and unreliable feedback, which will increase delivery time and will be incompatible with Continuous Delivery. Continuous Testing requires preliminary and ongoing investment in test automation, but a comprehensive set of automated unit and acceptance tests will provide fast, deterministic feedback that reduces manufacturing defects and delivery time, and also stimulates Continuous Delivery of reliable and anti-fragile systems.
Source: https://habr.com/ru/post/443558/
All Articles