We recently restarted the Yandex.Cash API, a payment service with a 15-year history. I want to tell you how to solve such an ambitious task. The material has accumulated on a series of articles, so here I will talk in detail about the design, processing of our API, as well as about our tools and processes.

Keywords for utility evaluation: API, REST, OpenAPI, Swagger, system interaction refactoring .

Posing a problem, or what is this conversation about

When a small team is transformed from a startup into a large company, the whole amount of knowledge about software components and their interaction cannot be kept in mind. Hence, two difficulties:

• Different departments have their own plans, and it is not always possible to do the work at the same time. Agree on the order of interaction and document it falls on the shore.
• It is important to preserve the reference knowledge of the rules of the service and its integration. It is much more difficult if you do not fix the order of interaction between the parties.

Development methodologies like Scrum make it possible to briefly remove these problems:

• The complexity of some tasks is beyond the scope of one team and they need to be solved together.
• There are tasks that can not be uniquely attributed to only one of the products, they require consistent changes in several components of the system.
• Maintenance tasks require refinement of old services. So old that their knowledge has lost its former brightness.

For ten years, Yandex.Money developers have been convinced of this by going the way of describing interactions from Wiki pages, via XML schema, JSON schema to OpenAPI / Swagger.

API specification design tools - the word was first

It all started with a description of the order of interaction of services in the Wiki and Microsoft Word, with examples of requests and responses. For data transfer, as a rule, XML was used. It was already better than nothing, but this way of documenting is suitable only for the transfer of knowledge from person to person.

Arbitrary textual description can be ambiguously interpreted by people from different departments, therefore, a unified system of concepts and terms was required. As the number of developers grows, the formalization of interaction descriptions has become necessary.

A formal description of operations, data types and their boundary conditions is needed not only for development, but also for automating unit testing. We described the first formal contracts of API services in the XML schema format, and later tried the JSON schema . But both are not perfect, which makes it difficult to completely switch to them from text descriptions on Wiki or Microsoft Word:

• It is not possible to add full-fledged documentation describing the work scenarios in the XML schema;
• JSON schema - too raw specification without sufficient instrumental support, also does not allow to include full documentation.

OpenAPI , previously known as Swagger, ideally suited us. The OpenAPI Initiative is an open source project managed by the Linux Foundation. I am convinced that he has a great future.

OpenAPI 3 allows you to combine documentation and a description of the interaction format in a single specification file. This is a very important quality - you will never have out of sync text documentation and API specification file.

Our projects use OpenAPI 3 specification files in YAML format, and here's why:

• YAML is easier to read than JSON.
• It includes CommonMark documentation (formerly Markdown). These are formatting, lists, tables, quotes, code samples.
• Documentation is added to the desired specification object, and not written in separate sections.

Service Decomposition and Change Management

A typical problem that many organizations face is document change management. It happens that in parallel there are many documents with separate changes for different projects. Maintaining documentation for many projects is very difficult, so the probability of human errors is high.

The OpenAPI specification is a text file in YAML format, which means you can work with it as with code:

• Use version control systems.
• Flexibly manage changes based on branches, tags, releases.
• Parallel to make changes for many projects by many people.
• Build procedures for reviewing and accepting changes to the main document.
• Refer to specific document versions in the context of a specific project.

Our projects use the Atlassian Bitbucket version control system with the Web Pages plugin. This allows you to simultaneously work with the specification, as with the code, and see the collected documentation in HTML-format.

When decomposing any major task into a set of services, I recommend relying on the principle of separation of functionality by domain of application areas, and not by technological commonality.

Behind each API service we have a specific business process, and each API service is described by a separate OpenAPI specification file. And its own product team is responsible for it. With this in mind, in the OpenAPI specification file set, each file corresponds to its application product.

Therefore, the rest was a matter of technology: it remains to set up the corresponding repositories in Bitbucket and set up the reviewer groups responsible for each API service. As a result, in Bitbucket, we have a project for API specifications, which is a common catalog of our APIs.

Specification files are grouped by target products:

• API specifications of a single product are placed in a single repository.
• APIs for different products are hosted in different repositories.

In other words, one repository corresponds to one product - the team responsible for the development and maintenance of business processes of the product and the specifications of its API.

According to the results of many completed projects, the structure of the Git-repository was as follows:

• The master branch reflects the state of the current specification, which is de facto on the combat system and is available for use.
• The prototype branch is intended for conceptual designs, to work out a set of product usage scenarios. In the future, it may be useful for retrospectives of initial constructions, which formed the basis of subsequent technical solutions.
• Project tasks are developed in feature-branches. One feature branch is a single project or API modification task.
• The project contains an index.html file with Swagger-UI initialization settings. Thanks to the Web Pages plugin, this allows you to display documentation in the form of HTML. Thus, each repository branch displays HTML documentation online, and can be referenced from external systems and documents.

In addition to the structure, I had to develop the rules for working with the repository:

• Nobody has the right to write to the master and prototype branches, direct recording is prohibited. All specification changes are drawn as Pull Requests from feature-branches.
• In order for changes from the Pull-Request to fall into the main specification, this Pull-Request must receive Approve from all mandatory viewers and from any number of optional viewers.
• Pull-Requests are excellent for discussing and agreeing on changes. Their history is a history of tasks that make changes to the specification.
• It is possible to make any changes to the feature-branch at any stage of the Pull-Request, however, the new changes will require reconciliation with reviewers.
• The merger of the Pull-Request in the master branch is carried out with the release of the working release into combat operation.

Thanks to these principles, we have a convenient, transparent and predictable environment for working with specifications.

Design-first approach as the basis for quality problem solving

When designing new services, we rely on REST principles, but do not fully comply with them - for example, if this complicates the architecture of the service or contradicts common sense.

The value of REST is that this approach requires the decomposition of the service into a set of entities and actions with them.

REST is a convenient reflection of the Domain Driven Design design approach. In my opinion, thanks to the REST architecture, we have more simple and high-quality object models of applied problems, compared to what we previously did using RPC approaches.

To create a high-quality product, the developer needs to carefully study the problem domain outside the context of the experience of previous projects. Therefore, Design-first has proven itself as an approach to solving applied problems. It can be divided into two stages:

1. Studying, the description, and also formalization of entities and processes of a subject domain.

   
   
2. Formalization of use cases for your API service taking into account possible links with other services. Use the terms of the domain, rather than internal technical jargon - so you will avoid a misunderstanding of the work of your service by its users.

The result of this work will be a service API specification that:

• Defines a set of domain entities and their attributes.
• Defines the life cycle and state set of each entity.
• Defines a set of actions with each entity and possible error situations.

I would not recommend the Code-first approach when the API specification is generated from an existing implementation: it always inherits solutions from previous projects, and does not solve the problem. There are no known methods to solve the chicken and egg paradox - to create a high-quality implementation, you first need to study the subject area and design the service, it is impossible to create the required implementation before the task study and design have been carried out.

REST antipatterns

Inspired by the article “ REST is a new SOAP ”, I would like to share practical considerations on the topic of examples of incorrect use of REST - this is not a silver bullet and, I hope, not a golden hammer in your hands.

REST is not only CRUD

It’s just amazing how the colleagues in the workshop look to pack all the processes into the Create-Read-Update-Delete model. Life is more complicated and richer: business processes can consist of many operations, entities can have many states and transitions between them.

Many articles about REST refer to the work of Roy Thomas Fielding “ Architectural Styles and the Design of Network-based Software Architectures ” as the primary source of the REST definition (see the fifth and sixth chapters of this work). Recommendations to use the http-verbs GET-POST-PUT-DELETE as the only way to define operations you will not find there.

REST is the principle of decomposition of services into a set of resources and operations on them. If you implement all the required functionality based on POST requests only, then this will also be REST.

Reflecting business logic errors based on HTTP status

Remember that the HTTP protocol performs the transport function of delivering data in requests and responses. Do not mix levels of business logic and data transfer. Clearly separate the concepts of "http-request" and "action of the business process." HTTP-status codes are designed to reflect the status of the HTTP request, they should not be used for tasks at the level of business logic.

However, we often use related HTTP protocols that impose their obligations on the response format, for example:

• The OAuth 2.0 Authorization Framework
• The OAuth 2.0 Authorization Framework: Bearer Token Usage

Here are some typical situations in which to use HTTP status:

1. 400: Invalid request format, request argument format.
2. 401: No client authentication.
3. 403: Insufficient client authorization.
4. 5xx: Technical request execution error, unavailability of service or intermediate gateway.

Business logic level failures should be reflected as attributes of the entity on which the operation was performed by this HTTP request.

For example, refusal to conduct a payment may be due to a lack of funds in the client’s account. At the same time, all http requests for payment will be completed successfully. This situation should be reflected in the form of state attributes of the “Payment” entity.

Using PUT / PATCH Requests for Business Logic Operations

PUT requests are not always suitable for business logic operations, since they are intended to replace a document on the server with a new document of the same type and structure. A GET request to the same resource must return a PUT request argument.

The standard defines a PUT request as:

This is a case in point where the request for payload has been received.

RFC 7231 sec 4.3.4

An example of the correct application of a PUT request is downloading a file to Yandex.Disk .

RFC 5789 A PATCH request is also limited in applicability: its semantics is similar to PUT, the request body is an RFC 6902 JSON patch document, and not any document at all.

Sample JSON patch document:

[ { "op": "remove", "path": "/a/b/c" }, { "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] }, { "op": "replace", "path": "/a/b/c", "value": 42 }, { "op": "move", "from": "/a/b/c", "path": "/a/b/d" } ] 

Agree, to describe the operations of business logic in a simple and accessible way with this syntax will not be easy.

So, if your task meets the above requirements, use PUT / PATCH, otherwise it is better to use POST.

A few words in conclusion

IT systems are constantly evolving and becoming more complex. Perhaps the main challenge today is to limit the increase in the complexity of systems. In my opinion, the methods of proper system decomposition and change management are a good help in our hard work.

I hope the material presented was useful to you. See also my report on designing a REST-like API on JavaJam, here is the link to the post .