Understanding DevOps and Js on the example of Dillinger.io
I am a Java developer backend, and very rarely reach the front end. This is a big omission. Lack of understanding of the frontend does not allow me to see the full picture of the world. I do not know what they are paying attention to and how they use all the fashionable tools like Node.js, Gulp, Webpack, that is, the fact that they are so frightened by the backend of developers with their diversity and constant changes.
And my firstvictim was the goal of Dillinger.io . It is an open text editor for Markdown, it has syntax highlighting and export of written text to PDF, HTML, Dropbox, Github and Google Drive, you can also add your own way, for example, static website generator or Wordpress.
A set of features sounds quite modest. But in order to understand it, I had to spend a couple of evenings and read about the whole zoo of technologies. And this project has a lot to learn.
There is a modest list of technologies used in the project repository :
From myself I can add a couple more:
This technology zoo surprised me, to say the least. From all this, I used only AngularJs, Bootstrap and Jquery. Of course, I heard about most technologies, but everything was limited to rumors.
')
When I looked through the list of technologies, I wondered why: “Why does a text editor, which works in my browser, have a backend?”. And all the more I did not understand why, in this case, Docker and the Enterprise platform for managing Node.js are needed. But I learned more about the creator of the application and the second question was solved by itself. Joe McCann is the co-founder of NodeSource , the company that released N | Solid (a framework for managing and monitoring Node.js applications). This means that he is well acquainted with the creation of the application infrastructure and in this area he has something to learn.
If you just want to test the application, you can choose a simple option. Only Node.js and Gulp are needed. Node.js brings up and configures the Express web server. And Gulp monitors changes in the Js and Sass files, converts them into a view that is understandable to the browser and reloads the page if any of the files have been changed. In the readme guides can be found on the installation and launch the server.
Node.js settings are in the app.js file. This file includes external dependencies and internal modules (routing of the main page, settings and plugins), as well as a web server. It gives static pages, compiles ejs templates, binds cookies to the user, compresses http-responses, sets up logging and finds handlers for dynamic requests.
The scripts for Gulp are in the / gulp folder. The main interest here are tasks : just by their names you can understand why Gulp is used in this project: watch, build, webpack, sass, browserSync. Or in Russian: we tell Gulp what files to monitor. If one of them is updated, then with the help of Webpack, it will assemble a script from dependencies and pieces of Js-code and turn Sass into Css. Further, he will ask the browser to refresh the page using browserSync.
Also in these files you can find different code execution profiles (development mode and production mode), they are different in that there are more debug information in the development mode (mapping of minified files, logs, etc.).
But as soon as you are done with developing and testing the application locally, the question arises of transferring it to the server. Of course, the easiest option is to do everything and install all the necessary utilities on the remote computer, make sure that it is configured like the local machine, run the application and believe that nothing will change. But to be sure that the environment is configured correctly and nothing changes, you need to somehow isolate your application. There are two principal ways: virtual machines and containers. And, of course, it’s a sin not to collect all the hipster technologies, if you write on JS =) And the choice towards Docker is obvious.
There is nothing supernatural in the container Docker configuration . The base container uses the image on which Node.Js is installed and all the necessary utilities. The script contains instructions for installing dependencies and starting the server, port 80 is given out, and environment variables are specified.
Fashion is not the only reason for using Docker in this project. As I said, the author of the project was somehow involved in creating N | Solid - an enterprise platform for applications on Node.Js. In the last paragraph, I did not mention, but the image of N | Solid is used as the base image, and it is not yet clear what advantages this gives. But there are two more suspicious files that should be studied: nsolid.yml and docker-compose.yml .
These two files are needed to configure the Docker Compose utility. They describe how to raise this application in cluster mode and configure the N | Solid platform. All containers rise in one network and as a result 8 containers start:
The repository did not forget about the NGinx configuration . It is used as a proxy. NGinx, like Express.js, compresses the data and gives out static files. It processes requests for static files independently, and proxies other requests between the cluster nodes. It turns out that there are two implementations of Proxy (N | Solid Proxy and NGinx) in the repository, and NGinx is used by default. Static files can also be sent out in two ways: via Express.js server or using NGinx. Both of these methods are used: the first - in the case of local development, and the second - if you run the application in cluster mode and use NGinx as a server.
Due to the fact that the capabilities of Service Registry, Proxy and Console are not used in this configuration, we simply close our eyes on them and consider only the request proxying in the figure. And, in general, I did not feel the importance of N | Solid for this project, most of the functionality can be recreated with the help of NGinx, maybe the whole thing is in the convenience of control, but my hands did not reach him.
But this automation does not end there =) At the root are two more interesting files: deploy.yml and dillinger.service . They are also associated with the creation of infrastructure.
The first one is the script for Ansible . It automates the installation of an application on Debian. Or rather, it connects extra reps, installs rsync and docker, installs the Dillinger image, creates and runs Debian-service with this image.
The second file describes this service. He describes that Dillinger depends on the docker, that the service will be available to several users, and also indicates how to start the service and that every time before starting it is necessary to download a new version of the image.
True, this script raises only one image of Dillinger, and not a whole cluster, but the line of thought is clear and you can modify this script so that it runs the entire cluster.
One of the important approaches of the first version of AngularJs is double-sided binding. It allows you to describe part of the logic directly in HTML, and this feature is actively used in this project. The code turned out to be blurred between the JS-files and ejs-templates, but the project is small and the whole picture can be kept in mind.
For writing HTML, Ejs template engine was used. As a result, the files turned out small and easy to read. All templates can be viewed in the / view folder. But if you just start reading all the files in a row, you will see tags that are not in normal Html, for example, the switch tag in the file / views / dropdowns / settings . This is a directive described by AngularJs, it is connected with some kind of logic, and this is a good way to use an angular, in this case, the code has become more readable and understandable. Although, if you use directives too often, you will have to learn your own Domain Specific Language (DSL) for each site. But with the advent of code, the templates become less readable, for example, in the file /views/sidebar.ejs you need to understand what the document means in this context and why you cannot delete the only remaining document.
The templates also contain a separation of scripts for development mode and production: in the /views/footer.ejs file, asynchronous script loading and Google analytics are included.
We have already managed to touch on AngularJs , and the time has come to find out in more detail how and why it is used in this project. If someone is unfamiliar with AngularJs, it's worth mentioning that this is a whole framework like, for example, Spring for Java. That is, it contains not only auxiliary functions, but also programming approaches that need to be used with it: Dependency Injection (description of dependencies when creating an angular component), modularity (programming pattern, allowing to separate the visibility of variables), double-sided binding (with changing a variable changes the component on the screen, changing the component on the screen (entering text, selecting a checkbox, etc.) changes the state of the variable). More details can be found here (look under the spoiler).
The application code can be found in the / public / js folder. The file /public/js/app.js describes the dependencies, the main module of the angular and the startup screen with the logo and the signature on the download.
All main application logic is located in the / public / js / services folder.
It describes the debounce function (a simple query filter that allows you to collapse identical commands and execute the code only once). Debounce is a programming pattern on js, it is used, for example, so that, without thinking about anything, when changing text, call the save function and be sure that it will not work too often (after each keystroke). Now this function works very naively and calls the command only some time after the last function call. If the user prints the text and accidentally closes the tab or reloads the page, then all his precious work may disappear (I once fell for this bug).
Also there are services for storing documents and users . They store information in the browser’s local memory and read it at initialization. The document service, oddly enough, stores the documents, that is, the text, the name and other metadata of the document entered by you. A user service stores settings.
There are two more services in this folder: wordscount and notification service . With the first service everything is clear, it is simple. The amount of code in the second service may scare, but there, too, everything is quite simple: it describes the function with which you can show the user a pop-up window. The main part of the code calculates the mutual position of the notifications, displays them in order and processes the closing of the window. An html file is used as a notification template, it displays a logo and a message.
With controllers it is still easier. They handle the events described in ejs templates. For example, the user controller handles the handling of buttons on the drop-down menu , and the document controller handles the events of the left menu, creates a new file, and autosaves.
There is also a file export controller, it uses the form on the main page for downloading files. And goes to the server in the core plugin for rendering Markdown.
The controllers also configure the text editor itself. To my disappointment, little is written about the text editor itself. The main functions are delegated to the Ace Editor , and all I managed to find is the theme and configuration of the text editor twice (due to code duplication, there may be bugs as well).
I already mentioned the directives when I talked about ejs templates. A directive is an arbitrary html element or attribute that has its own logic. There are many of them in the project, and I will not dwell on them in detail, they are very simple, and those who are interested can search for them in the / public / js / components folder and in other parts of the project.
About backend, I will tell not so much. I was surprised to meet him in this project. The backend in the project is implemented as plug-ins and lies in the / plugins folder. Among them is the core plugin, which renders Markdown in PDF, Html and Markdown. And 4 plug-in integration with external services, it makes sense to read them in conjunction with the corresponding plug-ins on the front end . These are just adapters that are engaged in authorization, render form files and collect additional information from the user. So let's say, reading for an amateur, if I find someone interesting, I will not stop.
Perhaps enough about this project. Today we understand the architecture of a simple application on Node.js, including infrastructure, front end and back end. I hope you learned something new and it became clearer how this project works and, maybe, even a clearer vision appeared on how to make your own better. I could forget to highlight some important topic or not to notice something, write, I will be happy to fix it.
Unfortunately, in the framework of the current version of the text editor, not all the ideas of Martin Broder are implemented, but there is work for the Open-Source community. When I understood this project, I wrote a couple issue and fixed one bug and urge you to do the same. Do it at least for your own sake, you will surely find something else interesting. See how much work awaits you.
Since the writing of the article could have changed a lot. Last commit at the time of writing this article: c3782a8dc0b91e5a6ae2c2ecd528daa1f42b3a9a .
And my first
A set of features sounds quite modest. But in order to understand it, I had to spend a couple of evenings and read about the whole zoo of technologies. And this project has a lot to learn.
Technology
There is a modest list of technologies used in the project repository :
- AngularJs - a front-end framework that allows you to bring logic in HTML (yes, first);
- Ace Editor - text editor;
- Marked - Markdown parser and compiler;
- Twitter Bootstrap - 'Nough said;
- Node.js - reactive js on the backend;
- Express - web-framework for Node.js;
- Gulp - streaming build system for the frontend;
- Keymaster.js - library for handling hotkeys;
- jQuery - well, where without it?
From myself I can add a couple more:
- Docker - containers (everyone talks about them anyway);
- Docker Compose - tools for running containers in batches;
- Ansible - tools for automating configuration management;
- Heroku - cloud application server;
- N | Solid - monitoring, proxy, orchestrating and managing applications on Node.Js;
- NGinx - http and proxy server (in this project);
- Npm is a package manager for Node.js;
- Webpack - JS file collector for frontend;
- Karma - test launcher;
- Ejs - template engine;
This technology zoo surprised me, to say the least. From all this, I used only AngularJs, Bootstrap and Jquery. Of course, I heard about most technologies, but everything was limited to rumors.
')
Infrastructure
When I looked through the list of technologies, I wondered why: “Why does a text editor, which works in my browser, have a backend?”. And all the more I did not understand why, in this case, Docker and the Enterprise platform for managing Node.js are needed. But I learned more about the creator of the application and the second question was solved by itself. Joe McCann is the co-founder of NodeSource , the company that released N | Solid (a framework for managing and monitoring Node.js applications). This means that he is well acquainted with the creation of the application infrastructure and in this area he has something to learn.
Can I just run the application?
If you just want to test the application, you can choose a simple option. Only Node.js and Gulp are needed. Node.js brings up and configures the Express web server. And Gulp monitors changes in the Js and Sass files, converts them into a view that is understandable to the browser and reloads the page if any of the files have been changed. In the readme guides can be found on the installation and launch the server.
Node.js settings are in the app.js file. This file includes external dependencies and internal modules (routing of the main page, settings and plugins), as well as a web server. It gives static pages, compiles ejs templates, binds cookies to the user, compresses http-responses, sets up logging and finds handlers for dynamic requests.
The scripts for Gulp are in the / gulp folder. The main interest here are tasks : just by their names you can understand why Gulp is used in this project: watch, build, webpack, sass, browserSync. Or in Russian: we tell Gulp what files to monitor. If one of them is updated, then with the help of Webpack, it will assemble a script from dependencies and pieces of Js-code and turn Sass into Css. Further, he will ask the browser to refresh the page using browserSync.
Also in these files you can find different code execution profiles (development mode and production mode), they are different in that there are more debug information in the development mode (mapping of minified files, logs, etc.).
How to transfer the application to the server?
But as soon as you are done with developing and testing the application locally, the question arises of transferring it to the server. Of course, the easiest option is to do everything and install all the necessary utilities on the remote computer, make sure that it is configured like the local machine, run the application and believe that nothing will change. But to be sure that the environment is configured correctly and nothing changes, you need to somehow isolate your application. There are two principal ways: virtual machines and containers. And, of course, it’s a sin not to collect all the hipster technologies, if you write on JS =) And the choice towards Docker is obvious.
There is nothing supernatural in the container Docker configuration . The base container uses the image on which Node.Js is installed and all the necessary utilities. The script contains instructions for installing dependencies and starting the server, port 80 is given out, and environment variables are specified.
But why exactly docker?
Fashion is not the only reason for using Docker in this project. As I said, the author of the project was somehow involved in creating N | Solid - an enterprise platform for applications on Node.Js. In the last paragraph, I did not mention, but the image of N | Solid is used as the base image, and it is not yet clear what advantages this gives. But there are two more suspicious files that should be studied: nsolid.yml and docker-compose.yml .
These two files are needed to configure the Docker Compose utility. They describe how to raise this application in cluster mode and configure the N | Solid platform. All containers rise in one network and as a result 8 containers start:
- N | Solid service registry (registers infrastructure applications),
- N | Solid proxy (based on Service Registry distributes requests between applications),
- N | Solid console (application management tool),
- 4 containers Dillinger (stateless, are registered in the service registry and are ready to accept requests),
- NGinx (web server, not on Node.JS).
The repository did not forget about the NGinx configuration . It is used as a proxy. NGinx, like Express.js, compresses the data and gives out static files. It processes requests for static files independently, and proxies other requests between the cluster nodes. It turns out that there are two implementations of Proxy (N | Solid Proxy and NGinx) in the repository, and NGinx is used by default. Static files can also be sent out in two ways: via Express.js server or using NGinx. Both of these methods are used: the first - in the case of local development, and the second - if you run the application in cluster mode and use NGinx as a server.
Due to the fact that the capabilities of Service Registry, Proxy and Console are not used in this configuration, we simply close our eyes on them and consider only the request proxying in the figure. And, in general, I did not feel the importance of N | Solid for this project, most of the functionality can be recreated with the help of NGinx, maybe the whole thing is in the convenience of control, but my hands did not reach him.
More automation
But this automation does not end there =) At the root are two more interesting files: deploy.yml and dillinger.service . They are also associated with the creation of infrastructure.
The first one is the script for Ansible . It automates the installation of an application on Debian. Or rather, it connects extra reps, installs rsync and docker, installs the Dillinger image, creates and runs Debian-service with this image.
The second file describes this service. He describes that Dillinger depends on the docker, that the service will be available to several users, and also indicates how to start the service and that every time before starting it is necessary to download a new version of the image.
True, this script raises only one image of Dillinger, and not a whole cluster, but the line of thought is clear and you can modify this script so that it runs the entire cluster.
Front end
One of the important approaches of the first version of AngularJs is double-sided binding. It allows you to describe part of the logic directly in HTML, and this feature is actively used in this project. The code turned out to be blurred between the JS-files and ejs-templates, but the project is small and the whole picture can be kept in mind.
Templates
For writing HTML, Ejs template engine was used. As a result, the files turned out small and easy to read. All templates can be viewed in the / view folder. But if you just start reading all the files in a row, you will see tags that are not in normal Html, for example, the switch tag in the file / views / dropdowns / settings . This is a directive described by AngularJs, it is connected with some kind of logic, and this is a good way to use an angular, in this case, the code has become more readable and understandable. Although, if you use directives too often, you will have to learn your own Domain Specific Language (DSL) for each site. But with the advent of code, the templates become less readable, for example, in the file /views/sidebar.ejs you need to understand what the document means in this context and why you cannot delete the only remaining document.
The templates also contain a separation of scripts for development mode and production: in the /views/footer.ejs file, asynchronous script loading and Google analytics are included.
AngularJs
We have already managed to touch on AngularJs , and the time has come to find out in more detail how and why it is used in this project. If someone is unfamiliar with AngularJs, it's worth mentioning that this is a whole framework like, for example, Spring for Java. That is, it contains not only auxiliary functions, but also programming approaches that need to be used with it: Dependency Injection (description of dependencies when creating an angular component), modularity (programming pattern, allowing to separate the visibility of variables), double-sided binding (with changing a variable changes the component on the screen, changing the component on the screen (entering text, selecting a checkbox, etc.) changes the state of the variable). More details can be found here (look under the spoiler).
The application code can be found in the / public / js folder. The file /public/js/app.js describes the dependencies, the main module of the angular and the startup screen with the logo and the signature on the download.
Services
All main application logic is located in the / public / js / services folder.
It describes the debounce function (a simple query filter that allows you to collapse identical commands and execute the code only once). Debounce is a programming pattern on js, it is used, for example, so that, without thinking about anything, when changing text, call the save function and be sure that it will not work too often (after each keystroke). Now this function works very naively and calls the command only some time after the last function call. If the user prints the text and accidentally closes the tab or reloads the page, then all his precious work may disappear (I once fell for this bug).
Also there are services for storing documents and users . They store information in the browser’s local memory and read it at initialization. The document service, oddly enough, stores the documents, that is, the text, the name and other metadata of the document entered by you. A user service stores settings.
There are two more services in this folder: wordscount and notification service . With the first service everything is clear, it is simple. The amount of code in the second service may scare, but there, too, everything is quite simple: it describes the function with which you can show the user a pop-up window. The main part of the code calculates the mutual position of the notifications, displays them in order and processes the closing of the window. An html file is used as a notification template, it displays a logo and a message.
Controllers
With controllers it is still easier. They handle the events described in ejs templates. For example, the user controller handles the handling of buttons on the drop-down menu , and the document controller handles the events of the left menu, creates a new file, and autosaves.
There is also a file export controller, it uses the form on the main page for downloading files. And goes to the server in the core plugin for rendering Markdown.
The controllers also configure the text editor itself. To my disappointment, little is written about the text editor itself. The main functions are delegated to the Ace Editor , and all I managed to find is the theme and configuration of the text editor twice (due to code duplication, there may be bugs as well).
Directives
I already mentioned the directives when I talked about ejs templates. A directive is an arbitrary html element or attribute that has its own logic. There are many of them in the project, and I will not dwell on them in detail, they are very simple, and those who are interested can search for them in the / public / js / components folder and in other parts of the project.
Backend
About backend, I will tell not so much. I was surprised to meet him in this project. The backend in the project is implemented as plug-ins and lies in the / plugins folder. Among them is the core plugin, which renders Markdown in PDF, Html and Markdown. And 4 plug-in integration with external services, it makes sense to read them in conjunction with the corresponding plug-ins on the front end . These are just adapters that are engaged in authorization, render form files and collect additional information from the user. So let's say, reading for an amateur, if I find someone interesting, I will not stop.
Perhaps enough about this project. Today we understand the architecture of a simple application on Node.js, including infrastructure, front end and back end. I hope you learned something new and it became clearer how this project works and, maybe, even a clearer vision appeared on how to make your own better. I could forget to highlight some important topic or not to notice something, write, I will be happy to fix it.
Unfortunately, in the framework of the current version of the text editor, not all the ideas of Martin Broder are implemented, but there is work for the Open-Source community. When I understood this project, I wrote a couple issue and fixed one bug and urge you to do the same. Do it at least for your own sake, you will surely find something else interesting. See how much work awaits you.
Since the writing of the article could have changed a lot. Last commit at the time of writing this article: c3782a8dc0b91e5a6ae2c2ecd528daa1f42b3a9a .
Source: https://habr.com/ru/post/280968/
All Articles