We build Docker images for CI / CD quickly and conveniently along with dapp (review and video)
This is the second publication, based on my speeches at conferences. The first was a general one and was dedicated to the review of Continuous Delivery practices with Docker . The new one is based on the more applied report “We assemble Docker images quickly and conveniently”, which was presented on November 8 at the HighLoad ++ 2016 conference in the “DevOps and Operation” section.
Like last time, if you have the opportunity to spend ~ an hour on video, we recommend watching it in full (see the end of the article) . Otherwise - we represent the main essence in text form.
')
Our requirements in the context of CI / CD processes (Continuous Integration, Continuous Delivery and Continuous Deployment) are:
To meet these requirements, the standard Docker mechanism — Dockerfile — is not enough. Docker authors have official reasons for this, which are accepted as fundamental and very logical principles in the project, such as a focus on solving common (rather than private) problems and a high level of portability.
Therefore, we wrote our utility - dapp (in Ruby, distributed under the MIT License). At this stage, she is able to deal only with image assembly, and her development plans include support for the full CI / CD cycle. In the design and implementation of dapp, preference is given to ease of use and speed / efficiency.
The configuration for images collected from dapp is described in the Dappfile on the principle of One repository → One project → One Dappfile. The format of this file is currently Ruby DSL, but we plan to switch to a simpler and more familiar YAML.
What opportunities does a dapp bring?
In dapp, a pattern is implemented with four stages of building a Docker image:
The results of these stages are cached, which leads to a significant increase in the speed of image re-assemblies.
For containers at the time of assembly, the so-called "external context" is available - these are mounted directories that are used at the time of assembly, but are excluded from the final image. In these directories, you can save information and use it in the following assemblies.
An example of use is the / var / lib / apt directory: its contents after running apt-get update are also needed for the following builds, but are not needed inside the image itself (additional data).
Support for changes from Git is also made in accordance with ideas of optimization and flexibility. When you first build an image, all application sources (git archive) are added to it, and later, only deltas are added; Git patches (git patch apply). The content of the patches is cached for better performance.
It is possible to specify files / directories, in case of which changes it is necessary to perform the install stage.
Sometimes, building a project (“compiling” some of its components) requires large third-party tools that are not used by the final application (i.e., do not need to be stored in the image). This applies not only to building sources in languages ​​like C, but also, for example, generating assets using Node.js. The problem is realized with the help of so-called "artifacts". When the dapp build command is executed, an additional Docker image is created with an artifact (that is, a third-party tool required to build the image). Files of this artifact are added to a real (final) image from an additional one using an external context. For artifacts, cache is also supported.
Modularity in assembling images brings significant benefits, but its implementation within the framework of the shell (Bash) is a thankless task. But it is perfectly made in configuration management systems: Chef → Berkshelf, Puppet → Librarian ... We use Chef, therefore we added its support in dapp. This support means the ability to execute recipes within the generated Docker image.
Technically, everything is organized in such a way that a Chef cookbook is placed in a special directory inside the Git repository (.dapp_chef). When you run the dapp build command, everything you need is collected in a directory that is mounted inside the Docker container. Additionally, a fully installed Chef (chefdk) is mounted in the container. Next, Chef runs, which configures the container for the cookbook. The resulting image is customized by recipe, but does not contain either chefdk or cookbook.
The format adopted in Dappfile allows you to describe many images at once in one file.
We have been using and developing dapp for almost 2 years and really want to make from it a useful Open Source solution that will help you customize your CI / CD processes and solve related tasks. We remind you that the source code is available on GitHub , issues and pull requests are also welcome there . Documentation on dapp in Russian is available here .
For dapp, we are looking for a unique enthusiast who dreams of becoming a technological evangelist. If you have a real interest in such tools, experience in DevOps, project management and writing competent technical texts - do not postpone and be sure to email us at info@flant.ru.
The video from the speech (about an hour) was published on YouTube (click on the link to start playing from the 13th minute, where the technical problems with the microphone stopped and the introductory part was completed, repeating the general CI report ) .
Presentation of the report:
Like last time, if you have the opportunity to spend ~ an hour on video, we recommend watching it in full (see the end of the article) . Otherwise - we represent the main essence in text form.
')
What do we want from Docker images?
Our requirements in the context of CI / CD processes (Continuous Integration, Continuous Delivery and Continuous Deployment) are:
- Compact volume. The reason is that within the CD you need to collect very often and a lot (each commit), which may soon lead to the need for huge repositories. We accepted the image rate of <200 MB (the base image with the Ubuntu system takes 130 MB).
- With a commit of 10 KB in volume, we want to see a similar addition in the size of the image, rather than the added full size of the image.
- Quick build images - in 10 seconds.
- The ability to use the generated images as the final product for different sites (from test to production).
dapp instead of dockerfile
To meet these requirements, the standard Docker mechanism — Dockerfile — is not enough. Docker authors have official reasons for this, which are accepted as fundamental and very logical principles in the project, such as a focus on solving common (rather than private) problems and a high level of portability.
Therefore, we wrote our utility - dapp (in Ruby, distributed under the MIT License). At this stage, she is able to deal only with image assembly, and her development plans include support for the full CI / CD cycle. In the design and implementation of dapp, preference is given to ease of use and speed / efficiency.
The configuration for images collected from dapp is described in the Dappfile on the principle of One repository → One project → One Dappfile. The format of this file is currently Ruby DSL, but we plan to switch to a simpler and more familiar YAML.
What opportunities does a dapp bring?
1. Stage and cache
In dapp, a pattern is implemented with four stages of building a Docker image:
- before_install: OS settings, etc., which (according to the results of our analysis of dozens of different projects) accounts for <1% of commits;
- install: application dependencies - about 5% of commits;
- before_setup;
- setup: configs - about 2%.
The results of these stages are cached, which leads to a significant increase in the speed of image re-assemblies.
2. External context
For containers at the time of assembly, the so-called "external context" is available - these are mounted directories that are used at the time of assembly, but are excluded from the final image. In these directories, you can save information and use it in the following assemblies.
An example of use is the / var / lib / apt directory: its contents after running apt-get update are also needed for the following builds, but are not needed inside the image itself (additional data).
3. Git
Support for changes from Git is also made in accordance with ideas of optimization and flexibility. When you first build an image, all application sources (git archive) are added to it, and later, only deltas are added; Git patches (git patch apply). The content of the patches is cached for better performance.
It is possible to specify files / directories, in case of which changes it is necessary to perform the install stage.
4. Artifacts
Sometimes, building a project (“compiling” some of its components) requires large third-party tools that are not used by the final application (i.e., do not need to be stored in the image). This applies not only to building sources in languages ​​like C, but also, for example, generating assets using Node.js. The problem is realized with the help of so-called "artifacts". When the dapp build command is executed, an additional Docker image is created with an artifact (that is, a third-party tool required to build the image). Files of this artifact are added to a real (final) image from an additional one using an external context. For artifacts, cache is also supported.
5. Chef support
Modularity in assembling images brings significant benefits, but its implementation within the framework of the shell (Bash) is a thankless task. But it is perfectly made in configuration management systems: Chef → Berkshelf, Puppet → Librarian ... We use Chef, therefore we added its support in dapp. This support means the ability to execute recipes within the generated Docker image.
Technically, everything is organized in such a way that a Chef cookbook is placed in a special directory inside the Git repository (.dapp_chef). When you run the dapp build command, everything you need is collected in a directory that is mounted inside the Docker container. Additionally, a fully installed Chef (chefdk) is mounted in the container. Next, Chef runs, which configures the container for the cookbook. The resulting image is customized by recipe, but does not contain either chefdk or cookbook.
6. Several images
The format adopted in Dappfile allows you to describe many images at once in one file.
dapp as open source
We have been using and developing dapp for almost 2 years and really want to make from it a useful Open Source solution that will help you customize your CI / CD processes and solve related tasks. We remind you that the source code is available on GitHub , issues and pull requests are also welcome there . Documentation on dapp in Russian is available here .
By the way, we have a job!
For dapp, we are looking for a unique enthusiast who dreams of becoming a technological evangelist. If you have a real interest in such tools, experience in DevOps, project management and writing competent technical texts - do not postpone and be sure to email us at info@flant.ru.
Video and slides
The video from the speech (about an hour) was published on YouTube (click on the link to start playing from the 13th minute, where the technical problems with the microphone stopped and the introductory part was completed, repeating the general CI report ) .
Presentation of the report:
Source: https://habr.com/ru/post/324274/
All Articles