For all the answer! Consumer Driven Contracts through the eyes of a developer
In this article, we’ll tell you about the problems that Consumer Driven Contracts solve, show how to use this using the example of Pact with Node.js and Spring Boot. And tell you about the limitations of this approach.
When testing products, scenario tests are often used in which the integration of various system components on a dedicated environment is checked. Such tests on live services give the most reliable results (not counting tests in battle). But at the same time, they are among the most expensive.
Stable tests in the integration environment require serious input from QA, dev, and even ops. No wonder they are at the very top of the test pyramid . Such tests are useful, but the economy of resources does not allow them to check everything. The main source of their value is the environment.
Below, for the same pyramid, there are other tests in which we exchange credibility for smaller support headaches - by means of isolation checks. The granular, the smaller the scale of verification, the less dependent on the external environment. At the very bottom of the pyramid are unit tests. We check individual functions, classes, we operate not so much with business semantics, as with constructions of a concrete implementation. These tests give quick feedback.
')
But as soon as we descend down the pyramid, we have to replace the environment with something. There are stubs - as whole services, and individual entities of the programming language. It is through stubs that we can test components in isolation. But they also reduce the reliability of checks. How to make sure that the stub gives the correct data? How to ensure its quality?
The solution can be comprehensive documentation that describes various scenarios and possible states of the system components. But any formulations still leave freedom of interpretation. Therefore, good documentation is a living artifact that is constantly being improved as the team understands the problem area. How then to ensure compliance of stubs of documentation?
On many projects, you can observe the situation when the same guys who were developing the artifact under test write stubs. For example, mobile application developers make stubs for their tests themselves. As a result, programmers can understand the documentation in their own way (which is completely normal), do the stub with the wrong expected behavior, write the code (with green tests) in accordance with it, and in real integration errors occur.
Moreover, the documentation usually moves downstream - clients use service specs (while the service client may be another service). It does not express how consumers use data, what data is generally needed, what assumptions they make for this data. The consequence of such ignorance is the law of Hyrum .
Hiram Wright has long been developing public sharing tools inside Google and observed how the slightest changes can cause damage to clients who used the implicit (undocumented) features of his libraries. This hidden connectivity complicates the evolution of the API.
These problems can be solved to some extent by using Consumer Driven Contracts. Like any approach and tool, it has a field of applicability and cost, which we also consider. Implementations of this approach have reached a sufficient level of maturity to try on their projects.
Three key elements:
Thus, the contract is executable. And the main feature of the approach is that the requirements for the behavior of the API go upstream , from client to server.
The contract focuses on the behavior that is really important to the consumer. Makes explicit its assumptions regarding the API.
The main task of the CDC is to bring together an understanding of the behavior of the API by its developers and the developers of its clients. This approach is well combined with BDD, at meetings of three amigos, you can throw blanks for a contract. Ultimately, this contract also serves to improve communications; sharing a common understanding of the problem area and implementing solutions within and between teams.
Consider the use of CDC on the example of Pact, one of its implementation. Suppose we make a web application for conference participants. In the next iteration, the team develops a display of the schedule of speeches - so far without any stories like voting or notes, only the output of a grid of reports. Sample sources are here .
At the meeting ofthree and four amigo there are product, tester, backend and mobile application developers. They pronounce that
After that, the frontend developer goes to write client code (backend for frontend). He sets up a library for working with a pact contract in the project:
And he begins to write a test. It configures the local server stubs, which will simulate the work of the service with the schedule of reports:
The contract is a JSON file that describes scenarios of client interaction with the service. But it is not necessary to describe it manually, since it is formed from the settings of the stub in the code. The developer before the test describes the following behavior.
Here in the example we specified a specific expected request to the service, but pact-js also supports several ways to determine matches .
Finally, the programmer writes a test of the part of the code that uses this stub. In the following example, we will call it directly for simplicity.
In a real project, this can be either a quick unit test of a separate response interpretation function, or a slow UI test for displaying the data received from the service.
During the test run, pact checks that the stub received the query specified in the tests. The discrepancies can be viewed as diff in the pact.log file.
If the test is successful, then the contract is formed in JSON format. It describes the expected behavior of the API.
He gives this contract to the backend developer. Let's say the API will be on Spring Boot. Pact has a pact-jvm-provider-spring library that can work with MockMVC. But we will look at Spring Cloud Contract, which implements the CDC in the Spring ecosystem. It uses its own contract format, but also has an extension point for connecting converters from other formats. Its native contract format is supported only by the Spring Cloud Contract itself - unlike Pact, which has libraries for JVM, Ruby, JS, Go, Python, etc.
Suppose in our example the backend developer uses Gradle to build the service. It connects the following dependencies:
And puts the pact contract received from the floender into the
From it, the default plugin spring-cloud-contract reads contracts. When building, the gradle-task of generateContractTests is executed, which generates the following test in the build / generated-test-sources directory.
When you run this test, we will see an error:
Since we can use different tools for testing, we need to tell the plugin which one we have configured. This is done through a base class that tests generated from contracts will inherit.
In order for this base class to be used when generating, you need to tune the spring-cloud-contract gradle-plugin.
Now we have generated such a test:
The test runs successfully, but ends with a validation error — the developer has not yet written the implementation of the service. But he can now do it, relying on the contract. He can make sure that he is able to process the client’s request and return the expected response.
The developer of the service knows through the contract what he needs to do, what behavior to implement.
Pact can be integrated deeper into the development process. You can deploy Pact-broker, which aggregates such contracts, supports their versioning and can show a dependency graph.
Downloading the newly generated contract to the broker can be done with a CI step when building the client. And in the server code specify the dynamic loading of the contract by URL. Spring Cloud Contract it also supports.
What are the limitations of Consumer Driven Contracts?
You have to pay for using this approach with additional tools like pact. Contracts themselves are an additional artifact, another abstraction that needs to be carefully maintained, consciously applied engineering practices to it.
They do not replace e2e tests , since the stubs still remain stubs - models of the real components of the system, which may slightly, but do not correspond to reality. Through them do not check the complex scenarios.
Also, CDCs do not replace API functional tests . They are more expensive to maintain than Plain Old Unit Tests. Pact developers recommend using the following heuristics - if you remove the contract and it does not cause errors or misinterpretation by the client, it means that it is not needed. For example, it is not necessary to describe absolutely all API error codes through a contract if the client processes them in the same way. In other words, the contract describes for the service only what is important to his client . Not more, but not less.
Too many contracts also complicate the evolution of the API. Each additional contract is a reason for red tests . It is necessary to design the CDC so that each fail test carries a useful semantic load that outweighs the cost of supporting it. For example, if the contract fix the minimum length of a text field, which is indifferent to the consumer (he uses the Toleran Reader technique), then each change in this minimum value will break the contract and the nerves of others. This check needs to be transferred to the level of the API itself and implemented depending on the source of the restrictions.
CDC improves product quality by explicitly describing integration behavior. It helps customer and service developers to achieve a common understanding, allows you to talk through code. But it does this at the cost of adding tools, introducing new abstractions and additional actions of team members.
At the same time, CDC tools and frameworks are being actively developed and have already reached maturity for testing on your projects. Test :)
Problematics
When testing products, scenario tests are often used in which the integration of various system components on a dedicated environment is checked. Such tests on live services give the most reliable results (not counting tests in battle). But at the same time, they are among the most expensive.
- It is often erroneously considered that the integration environment should not be fault tolerant. SLA, guarantees for such environments are rarely spoken, but if it is not available, teams either have to delay releases, or hope for the best and go into battle without tests. Although everyone knows that hope is not a strategy . And modern infrastructure technologies only complicate the work with integration environments.
- Separate pain - work with test data . Many scenarios require a certain state of the system, fixture. How close should they be to combat data? How to bring them to the current state before the test and cleaned after completion?
- Tests are too unstable . And not only because of the infrastructure that we mentioned in the first paragraph. The test may fall because the neighboring team has launched its checks, which broke the expected state of the system! Many false negative checks, flaky tests end their lives in
@Ignored. Also, different parts of the integration can be supported by different teams. They rolled out a new release candidate with errors - they broke all consumers. Someone solves this problem with selected test circuits. But at the cost of multiplying the cost of support. - Such tests take a lot of time . Even taking into account automation, the results can be expected for hours.
- And on top of that, if the test has really fairly dropped, it is far from always possible to immediately find the cause of the problem. It can be hidden deep behind the layers of integrations. Or it may be the result of an unexpected combination of states of multiple components of the system.
Stable tests in the integration environment require serious input from QA, dev, and even ops. No wonder they are at the very top of the test pyramid . Such tests are useful, but the economy of resources does not allow them to check everything. The main source of their value is the environment.
Below, for the same pyramid, there are other tests in which we exchange credibility for smaller support headaches - by means of isolation checks. The granular, the smaller the scale of verification, the less dependent on the external environment. At the very bottom of the pyramid are unit tests. We check individual functions, classes, we operate not so much with business semantics, as with constructions of a concrete implementation. These tests give quick feedback.
')
But as soon as we descend down the pyramid, we have to replace the environment with something. There are stubs - as whole services, and individual entities of the programming language. It is through stubs that we can test components in isolation. But they also reduce the reliability of checks. How to make sure that the stub gives the correct data? How to ensure its quality?
The solution can be comprehensive documentation that describes various scenarios and possible states of the system components. But any formulations still leave freedom of interpretation. Therefore, good documentation is a living artifact that is constantly being improved as the team understands the problem area. How then to ensure compliance of stubs of documentation?
On many projects, you can observe the situation when the same guys who were developing the artifact under test write stubs. For example, mobile application developers make stubs for their tests themselves. As a result, programmers can understand the documentation in their own way (which is completely normal), do the stub with the wrong expected behavior, write the code (with green tests) in accordance with it, and in real integration errors occur.
Moreover, the documentation usually moves downstream - clients use service specs (while the service client may be another service). It does not express how consumers use data, what data is generally needed, what assumptions they make for this data. The consequence of such ignorance is the law of Hyrum .
Hiram Wright has long been developing public sharing tools inside Google and observed how the slightest changes can cause damage to clients who used the implicit (undocumented) features of his libraries. This hidden connectivity complicates the evolution of the API.
These problems can be solved to some extent by using Consumer Driven Contracts. Like any approach and tool, it has a field of applicability and cost, which we also consider. Implementations of this approach have reached a sufficient level of maturity to try on their projects.
What is a CDC?
Three key elements:
- Contract Described using some DSL, implementation dependent. It contains the description of the API in the form of interaction scenarios: if a certain request arrives, the client should receive a definite answer.
- Customer tests . And they use a stub, which is automatically formed from the contract.
- Tests for API . They are also generated from the contract.
Thus, the contract is executable. And the main feature of the approach is that the requirements for the behavior of the API go upstream , from client to server.
The contract focuses on the behavior that is really important to the consumer. Makes explicit its assumptions regarding the API.
The main task of the CDC is to bring together an understanding of the behavior of the API by its developers and the developers of its clients. This approach is well combined with BDD, at meetings of three amigos, you can throw blanks for a contract. Ultimately, this contract also serves to improve communications; sharing a common understanding of the problem area and implementing solutions within and between teams.
Pact
Consider the use of CDC on the example of Pact, one of its implementation. Suppose we make a web application for conference participants. In the next iteration, the team develops a display of the schedule of speeches - so far without any stories like voting or notes, only the output of a grid of reports. Sample sources are here .
At the meeting of
- The UI will display a list with the text: Report title + Speakers + Date and time.
- For this, the backend must return data as in the example below.
{ "talks":[ { "title":" ", "speakers":[ { "name":" " } ], "time":"2019-05-27T12:00:00+03:00" } ] } After that, the frontend developer goes to write client code (backend for frontend). He sets up a library for working with a pact contract in the project:
yarn add --dev @pact-foundation/pact And he begins to write a test. It configures the local server stubs, which will simulate the work of the service with the schedule of reports:
const provider = new Pact({ // consumer: "schedule-consumer", provider: "schedule-producer", // , port: pactServerPort, // pact log: path.resolve(process.cwd(), "logs", "pact.log"), // , dir: path.resolve(process.cwd(), "pacts"), logLevel: "WARN", // DSL spec: 2 }); The contract is a JSON file that describes scenarios of client interaction with the service. But it is not necessary to describe it manually, since it is formed from the settings of the stub in the code. The developer before the test describes the following behavior.
provider.setup().then(() => provider .addInteraction({ uponReceiving: "a request for schedule", withRequest: { method: "GET", path: "/schedule" }, willRespondWith: { status: 200, headers: { "Content-Type": "application/json;charset=UTF-8" }, body: { talks: [ { title: " ", speakers: [ { name: " " } ], time: "2019-05-27T12:00:00+03:00" } ] } } }) .then(() => done()) ); Here in the example we specified a specific expected request to the service, but pact-js also supports several ways to determine matches .
Finally, the programmer writes a test of the part of the code that uses this stub. In the following example, we will call it directly for simplicity.
it("fetches schedule", done => { fetch(`http://localhost:${pactServerPort}/schedule`) .then(response => response.json()) .then(json => expect(json).toStrictEqual({ talks: [ { title: " ", speakers: [ { name: " " } ], time: "2019-05-27T12:00:00+03:00" } ] })) .then(() => done()); }); In a real project, this can be either a quick unit test of a separate response interpretation function, or a slow UI test for displaying the data received from the service.
During the test run, pact checks that the stub received the query specified in the tests. The discrepancies can be viewed as diff in the pact.log file.
E, [2019-05-21T01:01:55.810194 #78394] ERROR -- : Diff with interaction: "a request for schedule" Diff -------------------------------------- Key: - is expected + is actual Matching keys and values are not shown { "headers": { - "Accept": "application/json" + "Accept": "*/*" } } Description of differences -------------------------------------- * Expected "application/json" but got "*/*" at $.headers.Accept If the test is successful, then the contract is formed in JSON format. It describes the expected behavior of the API.
{ "consumer": { "name": "schedule-consumer" }, "provider": { "name": "schedule-producer" }, "interactions": [ { "description": "a request for schedule", "request": { "method": "GET", "path": "/schedule", "headers": { "Accept": "application/json" } }, "response": { "status": 200, "headers": { "Content-Type": "application/json;charset=UTF-8" }, "body": { "talks":[ { "title":" ", "speakers":[ { "name":" " } ], "time":"2019-05-27T12:00:00+03:00" } ] }}} ], "metadata": { "pactSpecification": { "version": "2.0.0" } } } He gives this contract to the backend developer. Let's say the API will be on Spring Boot. Pact has a pact-jvm-provider-spring library that can work with MockMVC. But we will look at Spring Cloud Contract, which implements the CDC in the Spring ecosystem. It uses its own contract format, but also has an extension point for connecting converters from other formats. Its native contract format is supported only by the Spring Cloud Contract itself - unlike Pact, which has libraries for JVM, Ruby, JS, Go, Python, etc.
Suppose in our example the backend developer uses Gradle to build the service. It connects the following dependencies:
buildscript { // ... dependencies { classpath "org.springframework.cloud:spring-cloud-contract-pact:2.1.1.RELEASE" } } plugins { id "org.springframework.cloud.contract" version "2.1.1.RELEASE" // ... } // ... dependencies { // ... testImplementation 'org.springframework.cloud:spring-cloud-starter-contract-verifier' } And puts the pact contract received from the floender into the
src/test/resources/contracts directory.From it, the default plugin spring-cloud-contract reads contracts. When building, the gradle-task of generateContractTests is executed, which generates the following test in the build / generated-test-sources directory.
public class ContractVerifierTest extends ContractsBaseTest { @Test public void validate_aggregator_client_aggregator_service() throws Exception { // given: MockMvcRequestSpecification request = given() .header("Accept", "application/json"); // when: ResponseOptions response = given().spec(request) .get("/scheduler"); // then: assertThat(response.statusCode()).isEqualTo(200); assertThat(response.header("Content-Type")).isEqualTo("application/json;charset=UTF-8"); // and: DocumentContext parsedJson = JsonPath.parse(response.getBody().asString()); assertThatJson(parsedJson).array("['talks']").array("['speakers']").contains("['name']").isEqualTo( /*...*/ ); assertThatJson(parsedJson).array("['talks']").contains("['time']").isEqualTo( /*...*/ ); assertThatJson(parsedJson).array("['talks']").contains("['title']").isEqualTo( /*...*/ ); } } When you run this test, we will see an error:
java.lang.IllegalStateException: You haven't configured a MockMVC instance. You can do this statically Since we can use different tools for testing, we need to tell the plugin which one we have configured. This is done through a base class that tests generated from contracts will inherit.
public abstract class ContractsBaseTest { private ScheduleController scheduleController = new ScheduleController(); @Before public void setup() { RestAssuredMockMvc.standaloneSetup(scheduleController); } } In order for this base class to be used when generating, you need to tune the spring-cloud-contract gradle-plugin.
contracts { baseClassForTests = 'ru.example.schedule.ContractsBaseTest' } Now we have generated such a test:
public class ContractVerifierTest extends ContractsBaseTest { @Test public void validate_aggregator_client_aggregator_service() throws Exception { // ... } } The test runs successfully, but ends with a validation error — the developer has not yet written the implementation of the service. But he can now do it, relying on the contract. He can make sure that he is able to process the client’s request and return the expected response.
The developer of the service knows through the contract what he needs to do, what behavior to implement.
Pact can be integrated deeper into the development process. You can deploy Pact-broker, which aggregates such contracts, supports their versioning and can show a dependency graph.
Downloading the newly generated contract to the broker can be done with a CI step when building the client. And in the server code specify the dynamic loading of the contract by URL. Spring Cloud Contract it also supports.
Applicability CDC
What are the limitations of Consumer Driven Contracts?
You have to pay for using this approach with additional tools like pact. Contracts themselves are an additional artifact, another abstraction that needs to be carefully maintained, consciously applied engineering practices to it.
They do not replace e2e tests , since the stubs still remain stubs - models of the real components of the system, which may slightly, but do not correspond to reality. Through them do not check the complex scenarios.
Also, CDCs do not replace API functional tests . They are more expensive to maintain than Plain Old Unit Tests. Pact developers recommend using the following heuristics - if you remove the contract and it does not cause errors or misinterpretation by the client, it means that it is not needed. For example, it is not necessary to describe absolutely all API error codes through a contract if the client processes them in the same way. In other words, the contract describes for the service only what is important to his client . Not more, but not less.
Too many contracts also complicate the evolution of the API. Each additional contract is a reason for red tests . It is necessary to design the CDC so that each fail test carries a useful semantic load that outweighs the cost of supporting it. For example, if the contract fix the minimum length of a text field, which is indifferent to the consumer (he uses the Toleran Reader technique), then each change in this minimum value will break the contract and the nerves of others. This check needs to be transferred to the level of the API itself and implemented depending on the source of the restrictions.
Conclusion
CDC improves product quality by explicitly describing integration behavior. It helps customer and service developers to achieve a common understanding, allows you to talk through code. But it does this at the cost of adding tools, introducing new abstractions and additional actions of team members.
At the same time, CDC tools and frameworks are being actively developed and have already reached maturity for testing on your projects. Test :)
At the QualityConf conference on May 27-28, Andrei Markelov will talk about testing equipment for the sale, and Arthur Khineltsev will tell you about monitoring a heavily loaded front end when the price of even a small mistake is tens of thousands of sad users.
Come chat for quality!
Source: https://habr.com/ru/post/452960/
All Articles