Service control panel. Part 1: Introduction

Introduction

My name is Maxim and I am the developer of the front end of the Miran service control panel. In this series of articles, I will tell you how the system is evolving and what's new to expect in the near future. Welcome under cat.

The era of dinosaurs.

The old panel, or panel_v1, is based on the Django side of the backend + TastyPie as an API + AngularJS in the frontend. The choice was based on good documentation and the ability to do a small file to bring the framework to the state we need. A bunch of Django and Tastypie provided all the necessary tools for our requests, and AngularJS was quite convenient for displaying data.

As you know, the requirements are constantly expanding and when they began to exceed the capabilities of the panel, it was decided to completely change the architecture and evolution of the technology stack. The “inventory” of the code was carried out and after that the work on it was frozen and limited to only minimal changes. And we got the opportunity to choose what will be the new control panel.

Flour choice

The main issue was the choice of architecture. Taking into account the existing experience of panel_v1 and the possibility of tastypie, we decided to develop the idea and break the entire panel into parts, link it through api and manage all parts individually. Therefore, the choice was microservices. After all, it is extremely convenient to have small modules that perform strictly their task and when changing their work? changes in their work is enough to update the interfaces for the rest of microservices.

The decision is made, super. For the backend there will be enough lightweight but powerful Flask, the frontend will move to the new Angular2. It seems everything looks good, but how can they communicate?
It requires an API that is flexible enough and suitable for the backend and frontend with minimal changes. Moreover, it will need to be opened to customers, which means you will not be able to do without detailed documentation with examples of use. This is where we became thoughtful.

Insight

The solution to our problem was the OpenAPI specification in the implementation of Swagger. Here is a quick list of Swagger features:

• Single API description file for backend and frontend.
• Work with yaml and json formats.
• A python connexion package that runs with an API specification file and builds routes based on it.
• With the help of swagger ui, all documentation is stored in the description of the methods of the API itself, so there is no need for any additional tools. There is a question: What does the method do and what data does it require to enter? - Welcome to ui! You can also perform a query and look at the result. As they say, it's better to see once than to hear 100 times.
• Strict control of incoming data. For example, let's take the standard situation - when the user fills out some form, then a number of checks occur:
  - at first simple validation on the frontend side;
  - further connexion in accordance with the specification file checks incoming data from the frontend;
  - and only then the backend can additionally perform any specific check.
  It is this multi-step control over the entered data that insures the user against an error.
• Specifically for the frontend: the swagger-codegen utility, which takes the specification file as input and provides all the necessary services for Angular2. And both in the form of Javascript and Typescript (in fact, the list of available languages ​​is more extensive and can be found here ).
• Non dull UI wallpapers:
  
  Considering all these points, we got a single source for all changes.
  Need to add functionality? Fine! First, the description is entered in the specification file, then the logic is described in the backend, after which api is rebuilt with new methods for the frontend and it remains to use these methods.
  Is there a need to change the returned data from a method? Easily! We fixed the changes to the specification file, changed the backend according to it, reassembled the api for the frontend and minimally fixed the affected methods.
  As a result, all routing, all requests and responses to them are described in one file. And the backend and the frontend work in concert.

It's not so simple

But everything does not happen smoothly. Below is a short list of pitfalls that we encountered.

Specification file

By default, the specification file can be split and stored, for example, definitions (an object containing a description of the data circulating in operations) separately and access it via a link. But this option did not work because of connexion. The current version of the package does not provide for this feature (sincerely I hope that in the next versions it will be added). For now it was necessary to get out as follows: all objects of entities (definitions, paths, responses, etc.) are selected in files and collected by means of grunt. The result was a set of tasks that can be easily added and changed. In my opinion, this option is better than finding the desired line among ~ 2500 lines of the final file.
Here is an excerpt from the task for the client part:

1. Cleaning before assembly
   

   clean: client: [ '<%= client_interim_dir %>' '<%= client_root %>/definitions.yml' '<%= client_root %>/paths.yml' ] 

2. Add indents

   
   

    indent: client_common_definitions: src: ['<%= client_common %>/definitions/*.yml'] dest: '<%= client_interim_dir %>/common/definitions/' options: style: 'space' size: 2 change: 1 client_dedic_definitions: src: ['<%= client_dedic_project %>/definitions/*.yml'] dest: '<%= client_interim_dir %>/dedic/definitions/' options: style: 'space' size: 2 change: 1 client_common_paths: src: ['<%= client_common %>/paths/*.yml'] dest: '<%= client_interim_dir %>/common/paths/' options: style: 'space' size: 2 change: 1 client_dedic_paths: src: ['<%= client_dedic_project %>/paths/*.yml'] dest: '<%= client_interim_dir %>/dedic/paths/' options: style: 'space' size: 2 change: 1 client_misc: src: [ '<%= client_root %>/projects.yml' ] dest: '<%= client_interim_dir %>/' options: style: 'space' size: 2 change: 1 

   
   

3. “Gluing” intermediate and final files

   
   

    concat: client_common_definitions: src: ['<%= client_interim_dir %>/common/definitions/*.yml'] dest: '<%= client_interim_dir %>/common_definitions.yml' client_dedic_definitions: src: ['<%= client_interim_dir %>/dedic/definitions/*.yml'] dest: '<%= client_interim_dir %>/dedic_definitions.yml' client_common_paths: src: ['<%= client_interim_dir %>/common/paths/*.yml'] dest: '<%= client_interim_dir %>/common_paths.yml' client_dedic_paths: src: ['<%= client_interim_dir %>/dedic/paths/*.yml'] dest: '<%= client_interim_dir %>/dedic_paths.yml' client_misc_paths: src: [ '<%= client_interim_dir %>/common_paths.yml', '<%= client_interim_dir %>/projects.yml' ] dest: '<%= client_interim_dir %>/common_paths.yml' client_target_paths: src: [ '<%= client_root %>/paths_header.yml' '<%= client_interim_dir %>/common_paths.yml' '<%= client_interim_dir %>/dedic_paths.yml' ] dest: '<%= client_root %>/paths.yml' client_target_definitions: src: [ '<%= client_root %>/definitions_header.yml' '<%= client_interim_dir %>/common_definitions.yml' '<%= client_interim_dir %>/dedic_definitions.yml' ] dest: '<%= client_root %>/definitions.yml' client_swagger: src: [ '<%= client_root %>/info.yml' '<%= client_root %>/tags.yml' '<%= client_root %>/security.yml' '<%= client_root %>/definitions.yml' '<%= client_root %>/parameters.yml' '<%= client_root %>/responses.yml' '<%= client_root %>/paths.yml' ] dest: '<%= client_output_path %>/client_swagger.yaml' 

   
   

Build api-layer for the frontend

This problem was discovered at the moment when I investigated how swagger-codegen can perform the assembly. There are two options: provide a specification file or download it. And here we decided to collect from the file, since the API service can refuse and this will affect the work of the frontend. Therefore, we chose the finished file. The repository was connected as git submodule, which ensured constant relevance. The algorithm is the following:

1. Update dependent repository:
   

    git submodule update 

2. Run the Grunt build task:
   

    grunt compile 

3. Run swagger-codegen with the resulting file.

But the detail pops up that the input swagger-codegen accepts JSON. And we have YAML. And then we realized that we guessed with Grunt. By connecting the plug YAML to JSON pants turning ... Ugh. YAML turns into JSON. And we get the required specification file.

Name problem

This was affected by the settings of swagger-codegen when naming methods and variables in typescript. For example, the property name is “total_count”. Without compilation settings, the typescript variable name became totalCount. While trying to solve this problem, it was found that it is possible to send an additional settings file in JSON format along with the configuration file. And this line:

 { "modelPropertyNaming": "original" } 

leaves names without change.

In the next issue, I will tell you how the thorny path of developing the frontend began and how we managed to optimize the loading of the panel. Thanks for attention!)