From API first on Swagger to Single contract on RAML

Hi% username%!

You probably know what API interfaces are and how much depends on them in your project. Moreover, I also believe that you are already familiar with the fact that API is the first approach and you know that Swagger and its Open API are among the most popular tools to help it follow.
')
But in this article I want to tell you about the approach to the implementation of the API first, which is conceptually different from what Swagger and Apiary offers. At the head of the idea is the concept of Single contract and the possibility of its implementation based on RAML 1.0.

Under the cut:

• A brief description of the principles of the API first;
• Single contract - input of the concept, prerequisites for appearance, consideration of the possibility of its implementation on the basis of OAS (Swagger);
• RAML + annotations + overlays as a base for Single contract , examples;
• RAML problems, conceptual differences of developers;
• The idea of ​​a SaaS service based on the above idea (picture of the prototype above).

From API first on Swagger to Single contract on RAML

When designing modern software systems, it often becomes necessary to coordinate and develop interfaces for the interaction of their components with each other. In the last decade, SPA and thick mobile applications interacting with the server through API interfaces have gained immense popularity and development. If earlier the development of an interactive web site was done by phasing the server side code to generate HTML markup and then passing it to the client’s browser, now the development of dynamic web applications has shifted towards the creation of a single service API and parallel development of many applications (including SPA) running with this API as with the main data source. This approach allows you to more conveniently separate tasks, organize teams that specialize only in specific technologies (attract more specialized specialists), organize parallel development at the very first stages, and also allows you to create a single point of communication - API interface.

Such a single point of communication requires a formal and unambiguous definition, this document is the API specification. Various technologies and languages ​​are used to develop and document API specifications today, for example: OAS (Swagger), Apiary and RAML.

The following three items define the nature of the API first approach:

1. The API interface must be the very first client interface of the application being developed;
2. First of all, the API specification is developed, and then the software part of its clients;
3. The life stages of an API interface must match the life stages of its documentation.

If we consider the process based on the foregoing, then the API specification lies at the center of the development process, and all nodes that make up the system and use this API as an interaction gateway are clients of the API specification. Thus, the server part of the system can be considered the same API client of the specification, like any other node that uses the API to communicate with it. Domain domain application models do not have to match the models described in the API specification. Their possible intentional coincidences with class structures in the code of client applications or database schema structures are introduced rather to simplify the development process, for example, when using a code generator according to the OAS specification. Logically, the above can be brought under the definition of Single contract . Single contract - many clients.

Single Contract. Contract Tools and Libraries

The term Single contract does not claim any participation in criticism for its use in the text of the article. Its use, in this context, is my personal idea.

Expanding the notion of API first , to the more general Single contract, allows us to consider the API specification not only as a formal description of the interaction interface between system components, but also as a single contract used by any number of external libraries and tools as a configuration source. In this case, these tools and libraries can be perceived as clients of the contract along with SPA or mobile applications. Examples of such clients include:

• Documentation Generator
• Mock-server API
• Load Testing Service
• Request / Response Validation Library
• Code generator
• UI interface generator
• etc.

Single contract for such clients is a single configuration file and data source. Contract tools work only on the basis of information obtained from a particular contract. Obviously, for a full-fledged functional of such heterogeneous clients as a mock server API, a single API description is not enough, additional meta information is needed, for example, a description of the links between GET request parameters (resource id) and data that the server should return, prompts indicating the response fields and query parameters used to organize pagination. Further, this example will be discussed in more detail. Specific information for specific instruments, at the same time, must exist and be maintained inextricably from the main document, otherwise it will lead to a violation of the concept of a single contract.

Swagger (OAS) as a Single contract description tool

Swagger (OAS) and Apiary (Blueprint), the most popular on the market, allow you to describe HTTP API interfaces using special languages: Open API based on YAML or JSON, Blueprint based on Markdown, which makes the specifications easy to read. There are also many tools and libraries created by the large open-source community. Swagger is currently widely distributed and can be said to have become the de facto API standard first. Many external systems support importing Swagger specifications, such as SoapUI , Readme.io , Apigee , etc. In addition, the existing SaaS Swagger Hub and Apiary allow users to create projects, upload or create their own specifications, use the built-in documentation generators and mock-servers, as well as publish links to access them from outside.

Swagger along with its OAS 3.0 look pretty confident and its functionality to describe the API (especially simple) is enough in most cases. The following is a list of pros and cons of Swagger:

Pros:

• Clear and easy to read description language;
• Large open-source community;
• Many official and open-source editors, generators, libraries;
• Having a core development team constantly working on developing and improving the format;
• Conditionally free hub for specifications;
• Detailed official documentation;
• Low threshold of entry.

Minuses:

• Weak modularity support;
• Absence of autogeneration of sample response requests based on the description of their structures;
• There are often problems with the poor stability of SmartBear products (the authors of the swagger) and the late reaction of the developer to this (the opinion is based purely on personal experience of use, and on the experience of our team).

But the main limitation of not allowing to use OAS as a Single contract description tool is the inability to attach custom meta information for describing additional parameters of target tools / libraries.
Therefore, all tools that work on the basis of Swagger-specifications must be content with the set of information that can accommodate the basic format.

For example, the implementation of a smart mock api server requires more information than a specification document is capable of, which is why the built-in Swagger Hub mock API is only capable of generating fake data based on data types / structures obtained from the specification document. Undoubtedly, this is not enough and such functionality of the mock server can be satisfied only by a simple API client.

In our company, during the development of one of the projects (React SPA + API server), the following mock-server functionality was required:

• imitation of pagination. The server should not return completely random values ​​of the currentPage, nextPage, pagesTotal fields in response to list requests, but be able to simulate the actual behavior of the pagination mechanism with generating values ​​of these metafields depending on the received page value from the client;
• generation of response bodies containing different data sets depending on the specific parameter of the incoming request;
• ability to build real connections between fake objects: the foo_id field of the Bar entity must refer to the previously generated Foo entity. This can be achieved by adding support for idempotency mock-server;
• imitation of work of various authorization methods: OAuth2, JWT and so on.

Without all this, it is very difficult to develop a SPA in parallel with the development of the server part of the system. And, at the same time, such a mock server, due to what was described earlier, is almost impossible to implement without additional specific meta information that could be stored directly in the API specification and inform it of the required behavior when simulating the next endpoint. This problem can be solved by adding the required parameters as a separate file with configurations parallel to the base OAS specification, but in this case, you will need to support these two different sources separately.

If there are more than one mock server operating in the environment of the development process according to this principle, we will get a “zoo” of tools, each of which, if it has its own unique functionality, is forced to have its own unique configuration file, logically linked to the base API -specifications, but actually located separately and living “its own life”.



Problem: the developer will be forced to keep all configurations up-to-date after changing versions of the base specification, often in completely different places and formats.

Some examples of services that work on a similar principle:

• SoapUI is a system for testing REST & SOAP interfaces. Supports project import from Swagger-spec. When the Swagger base specification is changed, the configuration of the draft based on the API call list continues to exist in parallel and requires manual synchronization;
• Other SmartBear Products;
• Apigee is an API lifecycle management service. Uses Swagger-specifications as templates, based on which, allows you to initialize your configuration of internal services. Automatic synchronization is also missing;
• Readme.io is a service that allows you to create beautiful documentation based on the Swagger-specification, and also has a mechanism for tracking changes in the basic specification and resolving conflicts by updating the project configuration on the service side. Surely, this required the excessive complexity of the development of this service.

Many other services can be added to this list that provide the integration function with the Swagger specification. Integration means for most of them the usual copying of the basic structure of the Swagger-specification and the subsequent auto-completion of the local configuration fields without synchronization with changes in the basic specification.

RAML, annotations, overlays

The desire to find a tool that eliminates the previously mentioned OAS restriction, which makes it possible to view the specification as a single contract for all client tools, led us to become familiar with the RAML language. About RAML is written enough, you can read, for example, https://www.infoq.com/articles/power-of-raml . RAML developers have tried to lay in the language support for modularity at the level of its concept. Now each company or individual developer can create their own or use ready-made public dictionaries when designing an API, redefine and inherit ready-made data models. Starting from version 1.0, RAML supports 5 different types of external modules: include, library, extension, trait, overlay , which allows each of them to be used depending on the task as flexibly as possible.

The time has come to discuss the main possibility of RAML, which, for reasons not entirely clear, has no analogues in OAS and Blueprint - Annotations.

RAML Annotations is the ability to attach custom metadata to basic language structures.

It is this function RAML and became the reason for writing this article.

Example:

#%RAML 1.0 title: Example API mediaType: application/json # Annotation types block may be placed into external file annotationTypes: validation-rules: description: | Describes strict validation rules for the model properties. Can be used by validation library allowedTargets: [ TypeDeclaration ] type: string[] info-tip: description: | Can be used by Documentation generator for showing tips allowedTargets: [ Method, DocumentationItem, TypeDeclaration ] type: string condition: description: | Named example can be returned if condition is evaluated to true. Can be used by Intelligent mock server allowedTargets: [ Example ] type: string types: Article: type: object properties: id: type: integer title: string paragraphs: Paragraph[] createdAt: type: string (validation-rules): ["regex:/\d{4}-[01]\d-[0-3]\dT[0-2]\d:[0-5]\d:[0-5]\d(?:\.\d+)?Z?/"] Paragraph: type: object properties: order: type: integer (validation-rules): ["min:0"] content: string (validation-rules): ["max-length:1024"] /articles/{articleId}: get: (info-tip): This endpoint is deprecated description: Returns Article object by ID responses: 200: body: application/json: type: Article 

User annotation structures themselves must have clear RAML descriptions. For this, a special section annotationTypes is used , the definitions from which can also be transferred to an external module. Thus, it becomes possible to determine the special parameters of an external tool in the form of annotations associated with the basic definition of the RAML API. In order to avoid cluttering up the base specification with a huge number of annotations for various external tools, there is support for the possibility of their removal into separate files - overlays (as well as possible in extensions ), with a classification by application. This is what is said about overlays in the documentation for RAML ( https://github.com/raml-org/raml-spec/blob/master/versions/raml-10/raml-10.md#overlays ):

An overlay adds or overrides the nodes of a RAML API definition while preserving its behavioral, functional aspects. Definition of the APIs: its resources, methods, parameters, bodies, responses, and so on. Cannot be changed by applying an overlay. In contrast, other nodes, such as those that address the functional interface, or These nodes can be changed by applying an overlay.

Overlays are particularly important for separating interface from implementation. Overlays enable you to be controlled tightly, such as a contracting different paces. For example, it can be defined as an API. These things can be controlled through a rigorous version and change management process.

In other words, this functionality allows to “separate the wheat from the chaff,” for example, the basic description of the API specification, from additional meta-information specific to the specific tool using it for work. Meta-information in each individual overlay is “hung” on various specification blocks in the form of annotations.

An example of a basic structure:

 #%RAML 1.0 title: Phrases API mediaType: application/json types: Phrase: type: object properties: content: string /phrases: get: queryParameters: whoSaid: string responses: 200: body: application/json: type: Phrase 

Overlay:

 #%RAML 1.0 Overlay usage: Applies annotations for Intelligent mock server extends: example_for_article_2_1.raml annotationTypes: condition: description: | Named example can be returned if condition is evaluated to true type: string allowedTargets: Example /phrases: get: responses: 200: body: application/json: examples: firstExample: (condition): $whoSaid is Hamlet content: "To be, or not to be?" secondExample: (condition): $whoSaid is Homer Simpson content: "D'oh!" 

As a result, it becomes possible to implement a single contract: all functional, behavioral and meta information is stored and versioned in one single place, and the contractual tools - the contract clients must have the support of the annotations used in this specification. On the other hand, it is the tools themselves that can make their own requirements for annotations that need to be “hung” on the specification - this will provide a wider range of possibilities in developing contractual tools.

The above concept is shown in the figure below:



Among the drawbacks of this approach, one can single out the high complexity of manual synchronization of the base specification file and each of the overlays: when updating the structure of the base specification, it is necessary to apply the required changes in the structures of overlays. This problem becomes more serious with the appearance of more than one overlay.

A possible and most obvious solution would be to develop a special editor or add-on for the existing RAML online editor https://github.com/mulesoft/api-designer . The editing area remains unchanged, but it becomes possible to create tabs: each new tab is a window for editing the overlay assigned to it. When editing the base structure of the specification in the main window, the structures in all the created tabs also change, and if an incompatibility of the new structure with already existing annotations found in the tabs overlays is detected, a warning appears. A more detailed consideration of such an editor is a separate topic and deserves serious consideration.

Existing groundwork

In the search for existing solutions that are close to the implementation of the idea of ​​using annotations as a means of describing meta-information, the following solutions were found:

• https://github.com/raml-org/raml-annotations repository containing official annotations approved by the developer community RAML. In the current version, only OAuth2 annotations are available. Can be used by external tools to obtain meta-information describing aspects of the implementation of OAuth2 for the specification being developed by the API;
• https://github.com/petrochenko-pavel-a/raml-annotations @ petrochenko-pavel-a library of annotations with logical grouping by scopes. The project is rather experimental, but it perfectly illustrates the idea of ​​using annotations. The most interesting annotation groups are:
  
  • additionalValidation.raml - annotations for describing additional rules for the validation of specification models. For example, they can be used by the server library to validate requests for the RAML specification;
  • mock.raml - annotations for describing the details of how a mock server is based on the RAML specification;
  • semanticContexts.raml — annotations indicating the semantic context of individual declared structural blocks of the RAML specification;
  • structural.raml - annotations, clarifying the role of a separate RAML entity in the overall structure of the described domain model;
  • uiCore.raml is an example of annotations that are possible for use by UI generation tools based on the RAML specification;

The repository also contains libraries of service types suitable for use as primitives when describing data structures of the RAML specification.

RAML problems

Despite the functionality, progressiveness of the basic idea, attention from major software vendors (cisco, spotify, vmware, etc.), today RAML has serious problems that can be fatal regarding its successful fate:

• Small and isolated open-source community;
• The incomprehensible strategy of the main RAML developer is mulesoft . The company develops products that are only a copy of existing OAS-based solutions (included in the Anypoint Platform ), instead of creating services that emphasize the advantages of RAML over Swagger;
• The consequence of the first paragraph: a small number of open-source libraries / tools;
• A higher threshold of entry than the OAS (this is strange, but many people think so);
• Due to the large number of bugs and problems with UX / UI, the main service is completely unusable and repulsive to users. The RAML entry point is https://anypoint.mulesoft.com/ .

Conceptual differences. First conclusion

There are contradictions within the community regarding the basic concept. Someone thinks that RAML is a model definition language , and someone that this is an API definition language as OAS or Blueprint (guys who call themselves RAML developers often mention this in various comments). The concept of a model definition language would allow within the RAML specification to describe the domain model of the domain without being strictly tied to the context of the resource API description, thereby expanding the horizons of options for using the specification with external tools (actually creating a foundation for the existence of a real Single Contract !). This definition of the resource concept can be seen on the readhat docs site ( http://restful-api-design.readthedocs.io/en/latest/resources.html , by the way, I recommend everyone to read this wonderful API design guide):

This is a list of available APIs . The Restored mapping of the application data model .

In the RAML, the application data model is the types declared in the types block, and the resource model of an API is what is described in the resources RAML block. Therefore, it is necessary to have the possibility of describing this mapping . But the current implementation of RAML allows you to make such a mapping only 1 to 1, that is, use types “as is” inside the declaration of the resource API.

I think this is the most important problem of the language, the solution of which will allow RAML to go beyond the API definition language and become a full-fledged model definition language : a more common language (rather than OAS or Blueprint) used to describe single system contracts, which are essentially the formal core sets of their components.

The above makes RAML a weak player who is currently unable to win a Swagger competition. Perhaps that is why, as a result, the main developer of RAML went to cardinal measures https://blogs.mulesoft.com/dev/api-dev/open-api-raml-better-together/

Idea Single contract RAML SaaS

Based on the concept of Single Contract , building on the hosting idea of ​​the Swagger hub OAS-based API specifications, and also relying on the capabilities of RAML to declare meta-information and to split the base specification using overlays, the idea of ​​an alternative SaaS solution for hosting and managing specifications based on the RAML language that is capable of surpass Swagger Hub and Apiary with the volume and quality of possible functionality.

The new service, by analogy with Swagger hub, will be hosting user contracts with the provision of an online editor and the possibility of viewing documentation-previews with real-time updates. The main difference should be the presence of a catalog of contract plug-ins built into the service, any of which the user can install in his current project an API specification. For installation, you will need to implement the required RAML annotations specified in the plugin documentation. After adding a new plug-in to the project, a new tab will be added to the code editor window. When you switch to this tab, editing annotations of the installed plug-in will become available. The structure of the base specification should be automatically duplicated in all tabs corresponding to the plugins. In the event of conflicts between the basic structure and already existing annotations, a special mechanism should offer options for its solution, or resolve it automatically.


Technically, each tab will be an abstraction over RAML overlay containing annotations of each specific plug-in. This guarantees the compatibility of the specification with any tool that supports RAML 1.0.

The plug-in directory must be open to the open source community. It is also possible to implement paid plug-ins, which can serve as an incentive for developing new ones.

Possible plugins: API documentation with support for a large number of annotations for flexible parameterization of its rendering, a smart mock server (from the example above), downloadable libraries for validating requests or code generation, debugging tools for outgoing API requests for mobile applications (caching proxy), load tests with test flow setting through annotations, various plugins for integration with external services.

This idea of ​​the service contains clear advantages over existing services for managing API specifications and its implementation lays the beginning of a possible change in the approach to the implementation of any external systems one way or another connected with the API.

Second conclusion

The purpose of this article is not to criticize Swagger, Apiary, or other de facto standard API development tools, but rather to consider the conceptual difference c by the design approach of specifications promoted by RAML, an attempt to introduce the concept Contract first and consider the possibility of its implementation on the basis of RAML. Another goal was the desire to attract to RAML well-deserved attention of developers for the further possible development of its community.

RAML official website
Slack channel
Specification

Thanks for attention.