The article presents (and demonstrated in short videos) a toolkit that facilitates the development and debugging of configurations with a dapp - Open Source utility, which we use every day when building and maintaining CI / CD processes.

Note : Recently, support for YAML syntax in dapp was announced, the features of which can be found in this publication . By default, all the tools described below will be valid both for it and for configuration in Ruby DSL (used in previous versions of dapp), and if this is not the case, please specify separately.

Introspection Stages

Writing the configuration in the first steps is complicated by the lack of understanding of what is contained in the assembly container when executing instructions.

Building an application and an artifact consists of a set of related stages . Each stage is assembled in an assembly container, which is based on the image of the previous stage (the base image in the case of the from stage), and then stored in the Docker image, application cache. All stages perform a specific function and are associated with the relevant directives in the configuration. You can learn more about the nature of the stages, their functions, features and possible states in the dapp documentation .

During the assembly process, it is possible to gain access to a certain stage using the introspection options . During introspection, the assembly container, the environment, contains the service tools and environment variables. The toolkit is presented as a set of utilities, which is required at the time of assembly. Its addition is carried out by mounting directories from the service containers of our dappdeps distributions (in the dappdeps container, they are accessible via the path /.dapp/deps ). More information about the idea of ​​these distributions, their assembly process and composition can be found in the article about dappdeps .

When developing introspection allows you to come to the desired result in the assembly container, and then transfer all the steps, instructions, to the configuration of the corresponding stage. This approach can be useful when the task is clear, but the steps to solve it are not obvious and require experimentation.

The process of writing a configuration in the introspection mode (using the example of a symfony-demo application) is demonstrated in this video:

When debugging, introspection allows you to see why the build ended with an error or the result did not meet expectations, check for dependent files, system status:

Finally, when using introspection with the Ansible-collector (for more details on Ansible dapp support, see this article ) , you can debug Ansible playbooks in the build container and then transfer the Ansible tasks to the appropriate configuration stages:

The following introspection options are supported during assembly:

 #         dapp dimg build --introspect-error dapp dimg build --introspect-before-error #    STAGE dapp dimg build --introspect-stage STAGE dapp dimg build --introspect-artifact-stage STAGE #      STAGE dapp dimg build --introspect-before STAGE dapp dimg build --introspect-artifact-before STAGE 

Work with a local Git-repository in a dev-mode

When developing a configuration in standard build mode, formal commits are required in order for dapp to take into account changes to the files during build. By analogy with mounting the working directory when building the Dockerfile want to work with the current state of the local repository .

This concept is implemented in the dev-build mode: the build takes into account uncommitted changes to the local Git repository, the corresponding configuration. To be precise, the paths from the following git subdirections are taken into account: add , includePaths , excludePaths , stageDependencies .

When adding git-submodules and nested Git repositories, .gitignore files at all levels are taken into account.

 dapp dimg build --dev #  dev-    dapp dimg mrproper --improper-dev-mode-cache 

Alternative caching scheme with asLayers directive

The before_install , install , before_setup , setup steps depend on the corresponding instructions in the configuration. Any change to the instructions results in a reassembly of the corresponding stage with all the instructions. Thus, with heavy, time-demanding instructions, development may be delayed.

Add to all this the situation when, when executing commands, the assembly falls on one of the last instructions of the stage. Apart from the fact that the assembly must be performed anew, there is no way to get the state of the environment before the instruction fell, to check the correctness of the previous ones.

For ease of development and debugging, the asLayers directive was introduced, which is indicated for a specific application or its artifact in the configuration ( dappfile.yaml ). During assembly, instructions are cached separately , and reassembly is performed only when their order is changed.

If there is no asLayers directive (or in the case of asLayers: false ), default caching is used, i.e. one Docker image for all instructions of the stage, and if you specify asLayers: true , then the new caching mode will be asLayers: true - one Docker image per one shell command or one task for Ansible.

Switching between build modes is governed only by the asLayers directive — the remaining configuration instructions remain unchanged. After the assembly instructions are debugged, asLayers must be turned off. Video demonstration of using asLayers in dapp:

The asLayers directive allows you to cache instructions individually. When using the introspection options --introspect-error and --introspect-before-error user can get the environment before or after the execution of the problem instruction.

Important notes:

• New dapp features are implemented only in the YAML configuration, so asLayers no longer supported in variants with Ruby DSL.
• It is important not to use this instruction with a regular assembly of images: this mode generates an excessive number of Docker images and is not designed for incremental build (as the waiting time and size of the build cache increase).

Summarizing

Because dapp remains one of the main tools in the daily work of our DevOps engineers, we are highly motivated to make it truly convenient, satisfying all their needs. At the same time, this is an open source project, and we are pleased to also see issues from third-party users, whose work scenarios may differ from our experience. Also ready to answer any of your questions on the use of dapp in the comments to this article - welcome!

PS

Read also in our blog:

• “ Waited: support for YAML and Ansible (without cows) in dapp ”;
• “The Linux distribution from scratch for building Docker images is our experience with dappdeps ”;
• “ Build projects with dapp. Part 1: Java ;
• " Practice with dapp. Part 1: Build simple applications ";
• " Practice with dapp. Part 2. Deploying Docker images in Kubernetes with the help of Helm ”;
• “ We assemble Docker images for CI / CD quickly and conveniently along with dapp (review and video of the report) ”.