DevOps in Development: Automating Web Application Code Writing

Good day, dear Habrazhiteli!

Today DevOps is on the wave of success. Practically at any conference devoted to automation, one can hear from the speaker that “we have implemented DevOps here and there, applied this and that, it became much easier to conduct projects, etc., etc.”. And this is commendable. But, as a rule, the introduction of DevOps in many companies ends at the automation stage of IT Operations, and very few people talk about the introduction of DevOps directly into the development process itself.

I would like to correct this little misunderstanding. DevOps in development can come through the formalization of the code base, for example, when writing a GUI for the REST API.
')
In this article, I would like to share with you the solution of a non-standard case that our company faced - we had the opportunity to automate the creation of a web application interface. I will tell you about how we came to this task and what we used to solve it. We do not believe that our approach is the only correct one, but we like it very much.

I hope this material will be interesting and useful to you.

Well, let's start!

Prehistory

This story began about a year ago: it was a beautiful summer day and our development department was engaged in creating the next web application. On the agenda was the task of introducing a new feature into the application - it was necessary to add the ability to create custom hooks.



At that time, the architecture of our web application was built in such a way that in order to implement a new feature, we had to do the following:

1. On the back-end: to create a model for a new entity (hooks), describe the fields of this model, describe all the logic of actions (actions) that the model can perform, etc.
   
2. On the front-end: to create a presentation class corresponding to the new model in the API, manually describe all the fields that this model has, add all types of actions that the given presentation can launch, etc.
   

It turns out that we were simultaneously in two places at once, it was necessary to make very similar changes in the code, one way or another, “duplicating” each other. And this, as you know, is not good, because with further changes, developers would need to make edits in two places at the same time.

Suppose we need to change the type of the field “name” from “string” to “textarea”. To do this, we will need to make this edit in the model code on the server, and then make similar changes in the presentation code on the client.

Is it too complicated?

Previously, we put up with this fact, since many applications were not very large and the approach with “duplication” of code on the server and on the client was the place to be. But on that same summer day, before the start of the introduction of a new feature, something clicked inside us, and we realized that it was impossible to continue this work. The current approach was very unreasonable and demanded a lot of time and labor. In addition, “duplication” of the code on the back-end and the front-end could lead to unexpected bugs in the future: developers could make changes on the server and forget to make similar changes on the client, and then everything would not go according to plan.

How to avoid code duplication? Search for a solution

We began to wonder how we can optimize the process of introducing new features.

We asked ourselves the question: “Can we now avoid duplicating changes in the model’s presentation on the front-end, after any change in its structure on the back-end?”

We thought and answered: "No, we can not."

Then we asked ourselves another question: “Okay, what is the reason for this duplication of code?”

And here it hit us: the problem, in fact, is that our front-end does not receive data on the current API structure. The front-end does not know anything about the models that exist in the API, until we ourselves inform him about this.

And then we had an idea: what if we build the architecture of the application so that:

• The front-end received from the API not only the data of the models, but also the structure of these models;
• The front-end dynamically formed views based on the structure of the models;
• Any change in the structure of the API is automatically displayed on the front-end.

Introduction of a new feature will take much less time, since it will only require changes on the back-end side, and the front-end will automatically pick up everything and present it to the user properly.

Universality of new architecture

And then, we decided to think a little more widely: is the new architecture suitable only for our current application, or can we use it somewhere else?



After all, one way or another, almost all applications have a part of the similar functionality:

• there are users for almost all applications, and therefore it is necessary to have the functionality associated with the registration and authorization of the user;
• almost in all applications there are several types of representations: there is a view for viewing the list of objects of a certain model, there is a view for viewing a detailed record of one single object of the model;
• almost all models have attributes similar in type: string data, numbers, etc., and therefore, you need to be able to work with them both on the back-end and on the front-end.

And since our company often performs custom web application development, we thought: why do we need to reinvent the wheel every time and develop similar functionality from scratch each time, if you can write a framework once that would describe all the basic, common to many applications, things, and then, creating a new project, use ready-made developments as dependencies, and, if necessary, declaratively change them in a new project.

Thus, in the course of a long discussion, we had the idea of ​​creating VSTUtils - a framework that:

1. It contained the basic, most similar for most applications, functionality;
2. Allowed to generate a front-end on the fly based on an API structure.

How to make friends back-end and front-end?

Well, we must do, we thought. We already had a certain back-end, a certain front-end, too, but neither the server nor the client had a tool that could inform or get data about the API structure.

In the search for a solution to this problem, our eye fell on the OpenAPI specification, which, based on the description of the models and the relationships between them, generates a huge JSON containing all this information.

And we thought that, in theory, when the application is initialized on the client, the front-end can receive this JSON from the API and build all the necessary views based on it. It remains only to teach our front-end to do it all.

And after a while we did teach him.

Version 1.0 - what was the result

The architecture of the first version of the VSTUtils framework consisted of 3 conditional parts and looked like this:

1. Back-end:
   
   • Django and Python - all the logic associated with the models. Based on the basic model of the Django Model, we have created several classes of basic models of VSTUtils. All actions that these models can perform we have implemented using Python;
     
   • Django REST Framework - REST API generation. Based on the description of the models, a REST API is formed, thanks to which the server and the client communicate;
     
2. Layer between back-end and front-end:
   
   • OpenAPI - JSON generation with a description of the API structure. After all models have been described on the back-end, views are created for them. Adding each of the views adds the necessary information to the final JSON:
     
     JSON'a Example - OpenAPI Scheme

     { // ,     (, ), //   -  , //  -     . definitions: { //    Hook. Hook: { // ,     (, ), //   -   , //  -       (,    ..). properties: { id: { title: "Id", type: "integer", readOnly: true, }, name: { title: "Name", type: "string", minLength:1, maxLength: 512, }, type: { title: "Type", type: "string", enum: ["HTTP","SCRIPT"], }, when: { title: "When", type: "string", enum: ["on_object_add","on_object_upd","on_object_del"], }, enable: { title:"Enable", type:"boolean", }, recipients: { title: "Recipients", type: "string", minLength: 1, } }, // ,     ,    . required: ["type","recipients"], } }, // ,     (, ), //   -   ( URL), //  -     . paths: { //      '/hook/'. '/hook/': { //    get    /hook/. //  ,       Hook. get: { operationId: "hook_list", description: "Return all hooks.", // ,       ,     . parameters: [ { name: "id", in: "query", description: "A unique integer value (or comma separated list) identifying this instance.", required: false, type: "string", }, { name: "name", in: "query", description: "A name string value (or comma separated list) of instance.", required: false, type: "string", }, { name: "type", in: "query", description: "Instance type.", required: false, type: "string", }, ], // ,     (, ), //   -   ; //  -   . responses: { 200: { description: "Action accepted.", schema: { properties: { results: { type: "array", items: { //   ,       . $ref: "#/definitions/Hook", }, }, }, }, }, 400: { description: "Validation error or some data error.", schema: { $ref: "#/definitions/Error", }, }, 401: { // ... }, 403: { // ... }, 404: { // ... }, }, tags: ["hook"], }, //    post    /hook/. //  ,       Hook. post: { operationId: "hook_add", description: "Create a new hook.", parameters: [ { name: "data", in: "body", required: true, schema: { $ref: "#/definitions/Hook", }, }, ], responses: { 201: { description: "Action accepted.", schema: { $ref: "#/definitions/Hook", }, }, 400: { description: "Validation error or some data error.", schema: { $ref: "#/definitions/Error", }, }, 401: { // ... }, 403: { // ... }, 404: { // ... }, }, tags: ["hook"], }, } } } 

3. Front-end:
   
   • JavaScript is a mechanism that pans the OpenAPI schema and generates views. This mechanism is run once, when the application is initialized on the client. By sending a request to the API, it receives in response the requested JSON with a description of the API structure and, analyzing it, it creates all the necessary JS objects containing the parameters of the model views. This API request is quite weighty, so we cache it and re-request it only when the application version is updated;
   • JavaScript SPA libs - rendering views and routing between them. These libraries were written by one of our front-end developers. When a user accesses a particular page, the rendering engine produces page rendering based on the parameters previously stored in JS view objects.

Thus, what we have: we have a back-end, which describes all the logic associated with the models. Then OpenAPI comes into play, which, based on the model description, generates JSON with a description of the API structure. Next, the baton is transmitted to the client, which, by analyzing the generated JAP generated by OpenAPI, automatically generates a web interface.

The introduction of features in the application on the new architecture - how it works

Remember the task about adding custom hooks? Here is how we would implement it in an application based on VSTUtils:



Now, thanks to VSTUtils, we don’t need to write anything from scratch. Here is what we do to add the ability to create custom hooks:

1. On the back-end: we take and inherit from the most appropriate class in VSTUtils, add a new functionality, characteristic of the new model;
2. On the front-end:
   
   • if the view for this model is no different from the basic VSTUtils view, then we do nothing, everything is automatically displayed properly;
   • if you need to somehow change the behavior of the view, using the mechanism of signals, we declaratively extend or completely change the basic behavior of the view.
     

As a result, we got a pretty good solution, we achieved our goal, our front-end became auto-generated. The process of introducing new features into existing projects has noticeably accelerated: releases began to appear once every 2 weeks, whereas previously we released releases every 2-3 months with a much smaller number of new features. I would like to note that the development team has remained the same; it is the new application architecture that has given us such fruits.

Version 1.0 - our hearts demand change

But, as you know, there is no limit to perfection, and VSTUtils is no exception.

Despite the fact that we were able to automate the formation of a front-end, the solution we initially wanted was not straightforward.

The client-side application architecture was not thoroughly thought out, and turned out to be not as flexible as it could be:

• the process of introducing functional overloads was not always convenient;
• The OpenAPI schema parsing mechanism was not optimal;
• views and routing between them were rendered using self-written libraries, which also did not suit us for a number of reasons:
  
  • these libraries were not covered by tests;
  • there was no documentation for these libraries;
  • they didn’t have any community - in the case of finding bugs in them or leaving the employee who wrote them, the support of such a code would be very difficult.

And since in our company we adhere to the DevOps approach and try to standardize and formalize our code as much as possible, then in February of this year we decided to conduct a global refactoring of the front-end framework VSTUtils. We had several tasks:

• to form not only classes of representations on the front-end, but also classes of models — we realized that it would be more appropriate to separate the data (and their structure) from their presentation. In addition, the presence of several abstractions in the form of representation and model would greatly facilitate the addition of overloads of the basic functionality in projects based on VSTUtils;
• use a tested framework for rendering and routing, with a large community (Angular, React, Vue) - this will allow us to give the whole headache with the support of the code associated with rendering and routing within our application.

Refactoring - choosing a JS framework

Among the most popular JS frameworks: Angular, React, Vue, our choice fell on Vue because:

• Vue codebase weighs less than React and Angular;
  
  Comparative table of framework sizes Gzipped version
  

  Framework	Size, kb
  Angular 2	111
  Angular 2 + RX	143
  Angular 1.4.5	51
  React 0.14.5 + React DOM	40
  React 0.14.5 + React DOM + Redux	42
  React 15.3.0 + React DOM	43
  Vue 2.4.2	21

• the page rendering process takes less time for Vue than for React and Angular;
  
• The Vue entry threshold is much lower than in React and Angular;
• Native syntax patterns;
• Elegant, detailed documentation available in several languages, including Russian;
• A developed ecosystem that provides, in addition to the Vue base library, libraries for routing and for creating a reactive data warehouse.

Version 2.0 - the result of front-end refactoring

The process of global refactoring of front-end VSTUtils took about 4 months and this is what we ended up with:



The VSTUtils front-end framework still consists of two large blocks: the first is parsing the OpenAPI schema, the second is rendering views and routing between them, but both of these blocks have a number of significant changes.

The mechanism that casts the OpenAPI scheme has been completely rewritten. The approach to parsing this scheme has changed. We tried to make the front-end architecture as similar as possible to the back-end architecture. Now, on the client side, we have not just a single abstraction in the form of representations, now we also have abstractions in the form of models and QuerySets:

• Objects of the Model class and its descendants are objects corresponding to the Django Models abstraction on the server side. Objects of this type contain data on the structure of the model (model name, model fields, etc.);
• objects of the QuerySet class and its descendants are objects corresponding to the Django QuerySets abstraction on the server side. Objects of this type contain methods that allow you to perform requests to the API (add, modify, retrieve, delete data of model objects);
• View objects are objects that contain data about how to present a model on a particular page, what template to use to “render” the page, what other model representations this page can refer to, etc.

The unit responsible for rendering and routing has also changed noticeably. We have abandoned samopisny JS SPA libraries in favor of the framework Vue.js. We developed our own Vue components that make up all the pages of our web application. Routing between views is performed using the vue-router library, and we use vuex as the reactive storage of the application state.

I would also like to note that on the front-end side, the implementation of the Model, QuerySet and View classes does not depend on the means of rendering and routing, that is, if we suddenly want to switch from Vue to some other framework, for example, to React or something new, all we need to do is rewrite the Vue components into the components of the new framework, rewrite the router, the repository, and that’s it - the VSTUtils framework will work again. The implementation of the Model, QuerySet and View classes will remain the same, since it does not depend on Vue.js. We believe that this is a very good help for possible future changes.

Let's sum up

Thus, the reluctance to write “duplicate” code resulted in the task of automating the formation of the front-end of a web application, which was solved by creating the VSTUtils framework. We managed to build a web application architecture in such a way that the back-end and front-end complement each other harmoniously and any change in the API structure is automatically picked up and displayed properly on the client.

The advantages that we have received from the formalization of the web application architecture:

• Releases of applications running on the basis of VSTUtils began to come out 2 times more often. This is due to the fact that now for the introduction of a new feature, often, we need to add code only on the back-end, the front-end is automatically generated - this saves time;
• Simplified update base functionality. Since now all the basic functionality is assembled in one framework, in order to update some important dependencies or make improvements to the basic functionality, we need to make changes only in one place - in the VSTUtils code base. When updating VSTUtils version in child projects, all innovations will automatically pick up;
• Finding new employees has become easier. Agree, it is much easier to find a developer under a formalized technology stack (Django, Vue) than to look for a person who agrees to work with an unknown samopis. Search results for developers who mentioned Django or Vue in their CVs on HeadHunter (across all regions):
  
  • Django - 3,454 resumes were found for 3,136 applicants;
  • Vue - 4,092 CVs of 3,747 job seekers found.

The disadvantages of this formalization of the web application architecture include the following:

• Due to the parsing scheme OpenAPI application initialization on the client takes a little longer than before (about 20-30 milliseconds longer);
• Bad search indexing. The fact is that at the moment we are not using the server rendering in the framework of VSTUtils, and the entire content of the application is generated in the final form already on the client. But our projects often do not need high search results and for us it is not so critical.

At this my story comes to an end, thank you for your attention!

useful links

• OpenAPI specification ;
• VSTUtils repository ;
• Sample project created on the basis of VSTUtils .