Stripe API version control system as a separate tool
The author of the material talks about the device version control system, which is implemented in the company Stripe.
When it comes to the API, change is unpopular. While many software developers are accustomed to working in the mode of frequent and fast iterations, API developers lose this flexibility as soon as they get at least the first user of their interface. Many of us are familiar with the evolutionary history of the Unix operating system.
')
In 1994, the book Unix-Hater Desktop Guide (Unix-Haters Handbook) was published, which touched upon a whole list of a variety of sensitive topics, from the names of tele-optimized commands with a completely incomprehensible history of origin to irreversible deletion of data, incomprehensible intuitive programs from excess options. More than 20 years later, the overwhelming majority of these complaints are still relevant, despite the diversity of currently available systems-heirs and branches. Unix has become so widely popular that changing its behavior can lead to far-reaching consequences. Good or bad, but between him and his users have already developed certain agreements that determine the behavior of Unix-interfaces.
Similarly, the API is a communication contract, which cannot be changed without a significant amount of cooperation and effort on both sides. Many businesses rely on Stripe as an infrastructure provider, and therefore we have been thinking about this kind of interaction from the very beginning of our company. Currently, we have managed to maintain support for each version of our API since the company appeared in 2011. In this article, we would like to share with you how we in Stripe manage to organize work with API versions.
Written to integrate with the code, the code is initially associated with some expectations. If the endpoint returns a boolean field called to indicate the status of a bank account, the user can write something like this:
If after this we replace the Boolean field verified by the status field, which may include the verified field (as we did in 2014), the code will stop working because it depends on the field that no longer exists. This type of change leads to a reverse incompatibility, and therefore we avoid such changes. Fields that were previously present must continue to be present and always retain the same type and name. However, not all changes lead to reverse incompatibility. For example, it is safe to add a new API endpoint or a completely new field to an existing point.
With sufficient coordination of efforts, we could keep users abreast of the upcoming changes and ask them to update their integrations in advance, but even if it were possible, such an approach cannot be called customer-oriented. Like connecting electricity or water to a network, our API should work as long as possible without any need to make changes.
Stripe provides an economic infrastructure for the Internet. Just like electricity producers should not change the voltage every two years, we also believe that our users should be sure that the API will maintain its stability as long as possible.
There is a common approach that allows you to develop an Internet API - support for different versions. When performing queries, users indicate the version of the API, and its suppliers can make changes in its next iteration, while maintaining compatibility in the current one. As new versions are released, users can upgrade to newer at a convenient time for them.
The essence of the most common version control scheme today is to use names like v1, v2 and v3, transmitted as a prefix to the URL (for example, / v1 / widgets) or via an HTTP header like Accept. This approach may work, but its main drawback is that when the size of the update between versions is large, and it itself includes major changes, the complexity of the transition is equivalent to the need to re-integrate from scratch.
The positive aspects of such a transition are not so obvious, since there is always a class of users who are unable or unwilling to upgrade, as a result of which they find themselves trapped in old versions of the API. In this case, suppliers face a difficult choice between abandoning old versions and, as a result, losing such customers, and continuing support for old versions forever, which entails substantial costs. Despite the fact that the second option may, at first glance, seem to be right from the point of view of customer-oriented solutions, support for outdated versions indirectly affects the quality of the project as a whole, because in fact it reduces the pace of work on innovations. Instead of developing new features, the working time of engineers is partially eaten by the support of the old code.
We at Stripe apply version control, named for the release date (for example, 2017-05-24). Despite their backward incompatibility, each such update contains a small set of changes that make updating and updating their integration into a smooth and relatively simple process.
When you first access the API from the user, the freshest version available is automatically assigned to his account, after which the system automatically implies that every subsequent API call will access this version from their side. This approach eliminates the situation when users accidentally get a change that violates their integration, and also makes the initial integration less painful by reducing the amount of work required to configure it. Users can enforce the version of each individual request by manually setting the Stripe-Version header, or by updating the version assigned in their account from the Stripe control panel.
Some readers may have already noticed that the Stripe API also identifies major versions using the path prefix (for example, / v1 / charges). And although we reserve the right to take advantage of this scheme sooner or later, it is unlikely that this will happen in the foreseeable future. As noted above, major changes usually make upgrades difficult and unpleasant, and it is difficult for us to imagine such an important API redesign that could justify such inconvenience to users. Our current approach has shown its effectiveness over nearly hundreds of backward-incompatible updates released over the past six years.
Versioning is always a trade-off between improved developer tools and the added burden of supporting older versions. In every way we strive to achieve the first, while at the same time minimizing the cost of work on the second item. To achieve these goals, we have introduced a version control system. Let's take a quick example of how it works. Each possible answer from the Stripe API is written in the form of a class called the API resource. Such resources define their possible fields using a domain-specific language:
API resources are written in such a way that the structure they describe is what we expect to get from the current version of the API. When we need to make a backward-incompatible change, we encapsulate it in a version change module that defines the change documentation, the transformation itself, and the set of API resource types that are subject to change:
In other cases, changes are assigned in accordance with the data from the master list:
Version changes are written in such a way that, if necessary, they are automatically applied in the reverse order from the current version of the API. Each version change assumes, in spite of the presence of more recent changes ahead, that the data they receive will look the same as they were originally written.
When generating the response, the API primarily formats the data by describing the API resource of the current version. After that, the target API version is determined based on:
After this, the API does a reverse version crawl and applies each version change module in its path until it reaches the required version.
Before the API returns a response, all requests are processed by versioning modules.
Modules version changes allow you to abstract from the old version of the API when working with the main code. As a result, most of the time, developers can avoid thinking about older versions while developing new products.
Most of our back-incompatible API changes change its response, but this happens forever. Sometimes a more complex change is required that results from the module defining it. We assign the note has_side_effects to such modules, (there are side effects) and the transformation they describe turns into a blank code:
The fact of their deactivation will also be checked in other parts of the code:
This reduction in encapsulation makes it difficult to support changes with side effects, so we try to avoid this approach.
One of the advantages of standalone version change modules is that they can declare documentation that describes which fields and resources they impact. We can use this to quickly provide our users with more useful information. For example, the change log of our API is generated programmatically and updated as soon as we deploy new versions of services.
We also customize API reference documentation to fit individual user needs. It checks that the user is authorized in the system, and leaves notes to the fields, based on the current API version of his account. In the image below, for example, we warn the developer that backward-incompatible changes have been made to newer versions of the API compared to its fixed version. The event request field was previously a string, but now it is a sub-object that also contains the idempotency key (created within the version change code shown above):
Our documentation defines the user version of the API and shows it the corresponding warnings.
Providing enhanced backward compatibility is not a gift. Each new version adds more code that needs to be understood and maintained. We try to write as clean as possible, but over time dozens of revisions of versions that cannot be clearly encapsulated can cause the project to become overgrown with unnecessary things and, as a result, become slower, brittle and lose its readability. In order to avoid the accumulation of this kind of expensive technical debt, we have taken several measures.
Despite the fact that we have a well thought-out version control system, we do everything possible to avoid using it and, above all, we try to build the architecture of our API from the very beginning. Any planned changes go through a simple review process, in which they are described in a brief information document and sent by mailing. This allows you to look at the proposed changes more broadly, in terms of different departments of the company. Increases the likelihood of detecting errors and inconsistencies before they get into the release.
We try to always keep in mind the balance between the need to maintain previous versions and the development of new features. Compatibility support is important, but even so, we expect to eventually abandon the older versions. Helping users to upgrade to new versions gives them access to new features and simplifies the foundation we use to create new features.
The combination of rolling out the version and the internal framework supporting them allowed us to significantly expand our user base and make a huge number of changes to the API, which, however, had practically no effect on the quality of existing integrations. This approach is based on several principles that we have chosen as a result of many years of practice. We believe that it is important for API updates to meet the following criteria:
Despite the fact that we are very interested to observe disputes and developments in topics such as REST vs. GraphQL vs. gRPC, and also - in a broader sense - discussions of how the API will look in the future, we believe to continue supporting version control schemes for quite some time.
When it comes to the API, change is unpopular. While many software developers are accustomed to working in the mode of frequent and fast iterations, API developers lose this flexibility as soon as they get at least the first user of their interface. Many of us are familiar with the evolutionary history of the Unix operating system.
')
In 1994, the book Unix-Hater Desktop Guide (Unix-Haters Handbook) was published, which touched upon a whole list of a variety of sensitive topics, from the names of tele-optimized commands with a completely incomprehensible history of origin to irreversible deletion of data, incomprehensible intuitive programs from excess options. More than 20 years later, the overwhelming majority of these complaints are still relevant, despite the diversity of currently available systems-heirs and branches. Unix has become so widely popular that changing its behavior can lead to far-reaching consequences. Good or bad, but between him and his users have already developed certain agreements that determine the behavior of Unix-interfaces.
Similarly, the API is a communication contract, which cannot be changed without a significant amount of cooperation and effort on both sides. Many businesses rely on Stripe as an infrastructure provider, and therefore we have been thinking about this kind of interaction from the very beginning of our company. Currently, we have managed to maintain support for each version of our API since the company appeared in 2011. In this article, we would like to share with you how we in Stripe manage to organize work with API versions.
Written to integrate with the code, the code is initially associated with some expectations. If the endpoint returns a boolean field called to indicate the status of a bank account, the user can write something like this:
if bank_account[:verified] ... else ... End If after this we replace the Boolean field verified by the status field, which may include the verified field (as we did in 2014), the code will stop working because it depends on the field that no longer exists. This type of change leads to a reverse incompatibility, and therefore we avoid such changes. Fields that were previously present must continue to be present and always retain the same type and name. However, not all changes lead to reverse incompatibility. For example, it is safe to add a new API endpoint or a completely new field to an existing point.
With sufficient coordination of efforts, we could keep users abreast of the upcoming changes and ask them to update their integrations in advance, but even if it were possible, such an approach cannot be called customer-oriented. Like connecting electricity or water to a network, our API should work as long as possible without any need to make changes.
Stripe provides an economic infrastructure for the Internet. Just like electricity producers should not change the voltage every two years, we also believe that our users should be sure that the API will maintain its stability as long as possible.
API Versioning Schemes
There is a common approach that allows you to develop an Internet API - support for different versions. When performing queries, users indicate the version of the API, and its suppliers can make changes in its next iteration, while maintaining compatibility in the current one. As new versions are released, users can upgrade to newer at a convenient time for them.
The essence of the most common version control scheme today is to use names like v1, v2 and v3, transmitted as a prefix to the URL (for example, / v1 / widgets) or via an HTTP header like Accept. This approach may work, but its main drawback is that when the size of the update between versions is large, and it itself includes major changes, the complexity of the transition is equivalent to the need to re-integrate from scratch.
The positive aspects of such a transition are not so obvious, since there is always a class of users who are unable or unwilling to upgrade, as a result of which they find themselves trapped in old versions of the API. In this case, suppliers face a difficult choice between abandoning old versions and, as a result, losing such customers, and continuing support for old versions forever, which entails substantial costs. Despite the fact that the second option may, at first glance, seem to be right from the point of view of customer-oriented solutions, support for outdated versions indirectly affects the quality of the project as a whole, because in fact it reduces the pace of work on innovations. Instead of developing new features, the working time of engineers is partially eaten by the support of the old code.
We at Stripe apply version control, named for the release date (for example, 2017-05-24). Despite their backward incompatibility, each such update contains a small set of changes that make updating and updating their integration into a smooth and relatively simple process.
When you first access the API from the user, the freshest version available is automatically assigned to his account, after which the system automatically implies that every subsequent API call will access this version from their side. This approach eliminates the situation when users accidentally get a change that violates their integration, and also makes the initial integration less painful by reducing the amount of work required to configure it. Users can enforce the version of each individual request by manually setting the Stripe-Version header, or by updating the version assigned in their account from the Stripe control panel.
Some readers may have already noticed that the Stripe API also identifies major versions using the path prefix (for example, / v1 / charges). And although we reserve the right to take advantage of this scheme sooner or later, it is unlikely that this will happen in the foreseeable future. As noted above, major changes usually make upgrades difficult and unpleasant, and it is difficult for us to imagine such an important API redesign that could justify such inconvenience to users. Our current approach has shown its effectiveness over nearly hundreds of backward-incompatible updates released over the past six years.
Under the hood version control system
Versioning is always a trade-off between improved developer tools and the added burden of supporting older versions. In every way we strive to achieve the first, while at the same time minimizing the cost of work on the second item. To achieve these goals, we have introduced a version control system. Let's take a quick example of how it works. Each possible answer from the Stripe API is written in the form of a class called the API resource. Such resources define their possible fields using a domain-specific language:
class ChargeAPIResource required :id, String required :amount, Integer End API resources are written in such a way that the structure they describe is what we expect to get from the current version of the API. When we need to make a backward-incompatible change, we encapsulate it in a version change module that defines the change documentation, the transformation itself, and the set of API resource types that are subject to change:
class CollapseEventRequest < AbstractVersionChange description \ “C ( -) “ \ “ request, id “ \ “ “ \ “ id .” response EventAPIResource do change :request, type_old: String, type_new: Hash run do |data| data.merge(:request => data[:request][:id]) end end end In other cases, changes are assigned in accordance with the data from the master list:
class VersionChanges VERSIONS = { '2017-05-25' => [ Change::AccountTypes, Change::CollapseEventRequest, Change::EventAccountToUserID ], '2017-04-06' => [Change::LegacyTransfers], '2017-02-14' => [ Change::AutoexpandChargeDispute, Change::AutoexpandChargeRule ], '2017-01-27' => [Change::SourcedTransfersOnBts], ... } end Version changes are written in such a way that, if necessary, they are automatically applied in the reverse order from the current version of the API. Each version change assumes, in spite of the presence of more recent changes ahead, that the data they receive will look the same as they were originally written.
When generating the response, the API primarily formats the data by describing the API resource of the current version. After that, the target API version is determined based on:
- Heading Stripe-Version, if one has been transmitted.
- Versions of an authorized OAuth application, if requested on behalf of the user.
- The version assigned to the user, assigned when the very first request is sent to Stripe.
After this, the API does a reverse version crawl and applies each version change module in its path until it reaches the required version.
Before the API returns a response, all requests are processed by versioning modules.
Modules version changes allow you to abstract from the old version of the API when working with the main code. As a result, most of the time, developers can avoid thinking about older versions while developing new products.
Changes with side effects
Most of our back-incompatible API changes change its response, but this happens forever. Sometimes a more complex change is required that results from the module defining it. We assign the note has_side_effects to such modules, (there are side effects) and the transformation they describe turns into a blank code:
class LegacyTransfers < AbstractVersionChange description "..." has_side_effects End The fact of their deactivation will also be checked in other parts of the code:
VersionChanges.active?(LegacyTransfers) This reduction in encapsulation makes it difficult to support changes with side effects, so we try to avoid this approach.
Declarative changes
One of the advantages of standalone version change modules is that they can declare documentation that describes which fields and resources they impact. We can use this to quickly provide our users with more useful information. For example, the change log of our API is generated programmatically and updated as soon as we deploy new versions of services.
We also customize API reference documentation to fit individual user needs. It checks that the user is authorized in the system, and leaves notes to the fields, based on the current API version of his account. In the image below, for example, we warn the developer that backward-incompatible changes have been made to newer versions of the API compared to its fixed version. The event request field was previously a string, but now it is a sub-object that also contains the idempotency key (created within the version change code shown above):
Our documentation defines the user version of the API and shows it the corresponding warnings.
Minimizing change
Providing enhanced backward compatibility is not a gift. Each new version adds more code that needs to be understood and maintained. We try to write as clean as possible, but over time dozens of revisions of versions that cannot be clearly encapsulated can cause the project to become overgrown with unnecessary things and, as a result, become slower, brittle and lose its readability. In order to avoid the accumulation of this kind of expensive technical debt, we have taken several measures.
Despite the fact that we have a well thought-out version control system, we do everything possible to avoid using it and, above all, we try to build the architecture of our API from the very beginning. Any planned changes go through a simple review process, in which they are described in a brief information document and sent by mailing. This allows you to look at the proposed changes more broadly, in terms of different departments of the company. Increases the likelihood of detecting errors and inconsistencies before they get into the release.
We try to always keep in mind the balance between the need to maintain previous versions and the development of new features. Compatibility support is important, but even so, we expect to eventually abandon the older versions. Helping users to upgrade to new versions gives them access to new features and simplifies the foundation we use to create new features.
Principles underlying change
The combination of rolling out the version and the internal framework supporting them allowed us to significantly expand our user base and make a huge number of changes to the API, which, however, had practically no effect on the quality of existing integrations. This approach is based on several principles that we have chosen as a result of many years of practice. We believe that it is important for API updates to meet the following criteria:
- Lightness Upgrades should be made as low-cost (both for users and for us) as possible.
- First-class approach . Turn version control into a first-class concept for your API. Make it so that you can use it for accurate documentation and improvement of tools, as well as for automatic generation of changelogs.
- Fixed cost Make sure that the old versions add only a minimum of maintenance work, by firmly encapsulating them into versioning modules. In other words, the less you need to think about the old behavior patterns while writing new code, the better.
Despite the fact that we are very interested to observe disputes and developments in topics such as REST vs. GraphQL vs. gRPC, and also - in a broader sense - discussions of how the API will look in the future, we believe to continue supporting version control schemes for quite some time.
Source: https://habr.com/ru/post/319758/
All Articles