Practice with dapp. Part 1: Build Simple Applications

This article is an introductory guide to building Docker images of applications using our open source dapp utility (read more about it in the announcement ) . Using the example of two simple applications (with one image), we consider how some of the main features of dapp can be used and what results they give.



At once I will make a reservation that dapp was not conceived as a utility that simplifies the local development of the application. Recently, however, we have a vision of how we could make the life of a regular developer easier, and another article will definitely come out about this. And now - about the simplification of the process CI / CD.
')

Build the application

What part does the assembly take in the CI / CD process? Let's take another look at the diagram from the distol colleague report :



The build turns the source code of programs from the Git repository (the git stage) into Docker images, ready to run in various environments (the build stage) . The question is, what is there to simplify, if you already have a Dockerfile and docker build ? Indeed, there are lots of articles about building Docker images for applications in various languages. But basically, these are articles about building and launching an application from a source code slice.

What problems will you face when there are several dozen apps? When it turns out that the build process has a lot of common parts and need to copy Dockerfile pieces? When will you want to speed up the build (after all, only two files have changed in the application)? These questions cannot be predicted in advance until you meet them in practice. Our answers to such questions and the experience of solving them is embodied in the dapp utility.

Build with dapp and Dappfile

Let us analyze the “live” example that dapp gives. As the first test, take a simple PHP application symfony-demo .

For the most impatient in our GitHub , a Dappfile has already been added and a Vagrantfile is made, so it’s enough to run vagrant up , after which you can build the application without installing Docker and dapp in your system.

For those who are not in a hurry and would like to “plunge into the head”, you need to install dapp ( see the documentation ) and repress the application repository for yourself:

 $ git clone https://github.com/symfony/symfony-demo.git $ cd symfony-demo $ vi Dappfile 

Describing the Dappfile assembly is a file with a DSL syntax for Ruby (somewhat similar to Vagrantfile , but we plan to switch to YAML while maintaining support for the old format). Its contents are:

 dimg 'symfony-demo-app' do docker.from 'ubuntu:16.04' git do add '/' do to '/demo' end end shell do before_install do run 'apt-get update', 'apt-get install -y curl php7.0', #  phpapp 'groupadd -g 242 phpapp', 'useradd -m -d /home/phpapp -g 242 -u 242 phpapp' end install do run 'apt-get install -y php7.0-sqlite3 php7.0-xml php7.0-zip', #  composer 'curl -LsS https://getcomposer.org/download/1.4.1/composer.phar -o /usr/local/bin/composer', 'chmod a+x /usr/local/bin/composer' end before_setup do #        composer install run 'chown phpapp:phpapp -R /demo && cd /demo', "su -c 'composer install' phpapp" end setup do #       run 'echo `date` > /demo/version.txt', 'chown phpapp:phpapp /demo/version.txt' end end #    ,   start.sh docker.expose 8000 end 

You also need to add start.sh and make it executable ( chmod +x start.sh ):

 #!/bin/sh cd /demo su -c 'php bin/console server:run 0.0.0.0:8000' phpapp 

You can assemble the application image with the command:

 $ dapp dimg build 

And run like this:

 $ dapp dimg run -d -p 8000:8000 -- /demo/start.sh 

Features: cache, git patch, build stages

The assembly produces a long log with all the actions performed. The listing is very large, so its part is laid out separately on GitHub . If you now run the dapp dimg build again, then these actions will not be executed again, because The result of their work is cached. The restart log can be seen here . His fragment:



You can see the lines with the name of the stage and the result of [USING CACHE] - this means that the dapp did not perform the described actions, creating a new image layer, and used the existing one.

Now let's pretend to be a developer and make changes - for example, change the link text in the page header in the app/Resources/views/base.html.twig . Make a commit and try to build the application. It can be seen that only git patch was superimposed, i.e. A new image was created on the basis of cached layers, to which changes were added to the project files:

 ... Setup [USING CACHE] signature: dimgstage-symfony-demo:3705edf770dd88ac714a7001fd24f395c87b2110005025eff48019d5973846ce date: 2017-08-16 04:16:46 +0000 difference: 0.0 MB Git artifacts dependencies [USING CACHE] signature: dimgstage-symfony-demo:f3f1c3e1ce5f0f5b880b1ec693b194d7e6a841a4166b29982d11b4e4c4cbe360 date: 2017-08-16 04:16:49 +0000 difference: 0.0 MB Git artifacts: apply patches (after setup) [USING CACHE] signature: dimgstage-symfony-demo:15e56865dd8b2a1cc55d5381a4e6f2cbcdc3a718509de29b15df02e8279b42c3 date: 2017-08-16 04:16:52 +0000 difference: 0.0 MB Git artifacts: latest patch ... [OK] 3.22 sec signature: dimgstage-symfony-demo:a9c21d0e36218563c8fd34b51969ed2f3b6662ca7775acae49488c5ebbbf25e1 Docker instructions ... [OK] 3.16 sec signature: dimgstage-symfony-demo:2eae4537c4210aaf4a153c7b8d3036343abf98b4ac4a3b99a2eb1967bea61378 instructions: EXPOSE 8000 

This build acceleration works well for files that do not need any special processing. What to do if it changes, for example, composer.json and you need to call composer install when building?

In this case, dapp supports 4 custom build stages (described in detail in the documentation ) . The first stage is before_install , during which the source is not available. This is usually the installation of rarely changing packages and OS settings. Further 3 stages: install , before_setup and setup - already have access to the source code. How to manage git patch overlay?

In order to indicate that changes in the files should restart the build from a certain stage, you need to specify the stage_dependencies directive in the git directive. In the case of our application, a change to the composer.json or composer.lock file should lead to a reassembly starting from the before_setup stage, at which composer install is started. Therefore, the git directive will look like this:

  git do add '/' do to '/demo' stage_dependencies.before_setup 'composer.json', 'composer.lock' end end 

Log build here . It can be seen that git path applied after the install stage and the image was rebuilt from the before_setup stage:

 ... Install [USING CACHE] signature: dimgstage-symfony-demo:a112d1abf364602c3595990c3f043d88e041a2a6f3cbcf13b6fc77a9fb3fd190 date: 2017-08-16 04:14:19 +0000 difference: 5.0 MB Git artifacts dependencies ... [OK] 2.75 sec signature: dimgstage-symfony-demo:9f0600ab6fb99356110c50454fc31e5fdc6ac3028e4ba8f200e789d140514bf9 Git artifacts: apply patches (after install) ... [OK] 2.18 sec signature: dimgstage-symfony-demo:f139188f9b0662d8177d41689b57c700e2276d997139673c3384731f6851d72e Before setup [BUILDING] ... 

Such links between file changes in the repository and stages reduce the overall build time. In most cases, application dependencies are not added or changed very often. For example, the developer implements a new feature, due to which it was necessary to add a dependency to composer.json . The first commit, which will be a new dependency, will be re-compiled from the before_setup stage — this will take some time. But subsequent commits, in which composer.json no longer change, are executed quickly. This is exactly what allows in our CI / CD configuration to automatically launch an assembly for each commit in the branches of developers and DevOps engineers (see “ GitLab CI for continuous integration and delivery in production. Part 1: our pipeline ”) .

Features: multi-stage or artifact image?

Not so long ago, in Dockerfile , it became possible to assemble parts of the final image using other images (multi-stage builds). This is done in order not to drag golang, webpack, gcc or other tools into the final image that are needed only for assembly and are completely unnecessary while the application is running. Dapp supports this build initially, using the artifact sections.

Take for the following example a golang web application . As with the first application, the repository ready for experimentation with Dappfile and Vagrantfile can be Vagrantfile from our GitHub .

Steps:

 $ git clone https://github.com/revel/examples revel-examples $ cd revel-examples/booking $ vi Dappfile 

Dappfile will be like this:

 dimg_group do artifact do docker.from 'golang:1.8' git do add '/' do to '/go/src/github.com/revel/examples' end end shell.before_install do run 'apt-get update', 'apt-get install -y sqlite3 libsqlite3-dev tree' end shell.install do run 'go get -v github.com/revel/revel', 'go get -v github.com/revel/cmd/revel' end shell.build_artifact do run '(go get -v github.com/revel/examples/booking/... ; true)' run 'revel build github.com/revel/examples/booking /app' end export '/app' do to '/app' after 'install' end end dimg 'booking-app' do docker.from 'ubuntu:16.04' end end 

You can build and run all the same commands:

 $ dapp dimg build $ dapp dimg run -p 9000:9000 --rm -d -- /app/run.sh 

In order not to get lost among the docker images output, we run the final image:

 $ dapp dimg tag booking-app:v1.0 

Now you can notice that the size of the final image does not depend on the size of the image with the golang build tools:

 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE booking-app v1.0 57d564633ecb 4 minutes ago 136MB ubuntu 16.04 ccc7a11d65b1 7 days ago 120MB golang 1.8 6ce094895555 3 weeks ago 699MB 

Now I will describe Dappfile in more detail using artifact (I deliberately omit the features of revel application assembly — if you are interested, you can discuss it in comments). In general, the structure will be as follows:

 dimg_group artifact do docker.from git   export end dimg 'dimg_name' do docker.from   end end 

• The artifact section indicates that part of the application should be assembled in a separate image, and using the git directive inside the artifact you can specify where to put the source texts.
• export indicates that you need to copy from a separate image to the final image after executing the commands of the build_artifact stage.
• In export , you can also specify at what stage the final image needs the results of the artifact. In our case, the specified after 'install' , i.e. after the install stage commands, the /app directory from the artifact image will be copied to the final image.

At first glance, all this is similar to Dockerfile using multi-stage, but the advantage of the artifact is that other dapp features are applied to it - caching and stage dependencies on changes in Git (these dependencies can be described in the git section).

Results

The described dapp features: splitting into stages, dependence of stages on changes in files in Git repositories, using artifact images — can simplify and speed up the assembly of almost any application both in the CI / CD process and for local development.

But this is only the beginning. In the Dappfile you can describe several images at once, and the dapp dimg build will collect them, and the dapp dimg push --tag tag will tag and “push” the cache images and the final image to the Registry. These features will be better illustrated in the following articles - in conjunction with the description of the support for the deployment in Kubernetes, which recently appeared in dapp.