Advanced multi-stage build templates

The multistage assembly function in Dockerfile files allows you to create small images of containers with a higher level of caching and a smaller amount of protection. In this article, I’ll show several advanced templates — something more than copying files between build and execute steps. They allow you to maximize the effectiveness of the function. However, if you are a beginner in the field of multi-stage assembly, then first, probably, it would not be superfluous to read the usage guide .

Version Compatibility

Support for multi-stage build was added to Docker in version v17.05. All templates work with any subsequent version, but some are much more efficient, thanks to the build using server-side BuildKit . For example, BuildKit effectively skips unused stages and, if possible, creates stages at the same time (I singled out these examples separately). Currently, BuildKit is being added to Moby as an experimental server part of the build and should be available in Docker CE v18.06. It can also be used offline or as part of the img project.

---

Inheritance from the stage

Multi-stage build adds several new syntax concepts. First of all, you can assign the stage beginning with the FROM command the name AS stagename and use the option --from=stagename in the COPY to copy the files from this stage. In fact, the FROM command and the --from label have much more in common, it’s not for nothing that they have the same name. Both take the same argument, recognize it and either start a new phase from this point, or use it as a source to copy the file. That is, to use the previous stage in the original image quality for the current stage, you can take not only --from=stagename , but also the name of the stage FROM stagename . It is useful if you use the same common parts in several commands in the Dockerfile: reduces the common code and simplifies its maintenance, keeping the child stages separate. Thus, rebuilding one stage does not affect the build cache for others. Accordingly, each stage can be assembled individually using the --target label when calling docker build .

 FROM ubuntu AS base RUN apt-get update && apt-get install git FROM base AS src1 RUN git clone … FROM base as src2 RUN git clone … 

In this example, the second and third stages in BuildKit are built simultaneously.

Direct use of images

Instead of using assembly step names in FROM commands that previously only supported image references, you can directly use images with the --from label. It turns out to copy files directly from these images. For example, linuxkit/ca-certificatesimage in the following code directly copies the TLS CA roots to the current stage.

 FROM alpine COPY --from=linuxkit/ca-certificates / / 

Common nickname

The build step does not necessarily include any commands; it may consist of a single FROM string. If an image is used in several places, it will make reading easier and make it so that if you need to update a shared image, you only need to change one line.

 FROM alpine:3.6 AS alpine FROM alpine RUN … FROM alpine RUN … 

In this example, every place that uses the alpine image is actually fixed to alpine:3.6 , and not alpine:latest . When it comes time to upgrade to alpine:3.7 , you will need to change a single line, and you can be sure: now the updated version is used in all elements of the assembly.

This is all the more important when the assembly argument is used in the alias. The following example is similar to the previous one, but allows the user to redefine all instances of the assembly in which the alpine image is involved, by setting the option --build-arg ALPINE_VERSION=value . Remember: any arguments used in the FROM commands must be defined before the first stage of the assembly .

 ARG ALPINE_VERSION=3.6 FROM alpine:${ALPINE_VERSION} AS alpine FROM alpine RUN … 

Using assembly arguments in "- from"

The value specified in the --from label of the COPY must not contain assembly arguments. For example, the following example is invalid.

 // THIS EXAMPLE IS INTENTIONALLY INVALID FROM alpine AS build-stage0 RUN … FROM alpine ARG src=stage0 COPY --from=build-${src} . . 

This is due to the fact that the dependencies between the stages need to be determined even before the assembly begins. Then the constant evaluation of all teams is not required. For example, the environment variable defined in the alpine image may affect the estimated value from --from . The reason we can evaluate the arguments of the FROM command is because these arguments are defined globally before the start of any stage. Fortunately, as we found out earlier, it is enough to define the stage of the alias using one FROM command and refer to it.

 ARG src=stage0 FROM alpine AS build-stage0 RUN … FROM build-${src} AS copy-src FROM alpine COPY --from=copy-src . . 

Now, if you override the src assembly argument, the initial stage for the final COPY element is switched. Note: if some stages are no longer used, then only BuildKit-based linkers will be able to skip them.

Conditions using assembly arguments

We were asked to add support for IF/ELSE style conditions in Dockerfile. We still do not know whether we will add something similar, but in the future we will try - using the support of the client part in BuildKit. Meanwhile, to achieve a similar behavior, you can use the current multi-stage concept (with some planning).

 // THIS EXAMPLE IS INTENTIONALLY INVALID FROM alpine RUN … ARG BUILD_VERSION=1 IF $BUILD_VERSION==1 RUN touch version1 ELSE IF $BUILD_VERSION==2 RUN touch version2 DONE RUN … 

The previous example shows pseudo-code for recording conditions using IF/ELSE . To achieve similar behavior with the current multi-stage builds, you may need to define different branch conditions as separate steps and use an argument to choose the right dependency path.

 ARG BUILD_VERSION=1 FROM alpine AS base RUN … FROM base AS branch-version-1 RUN touch version1 FROM base AS branch-version-2 RUN touch version2 FROM branch-version-${BUILD_VERSION} AS after-condition FROM after-condition RUN … 

The last stage in the Dockerfile is based on the after-condition stage, which is an image alias (recognized based on the BUILD_VERSION assembly BUILD_VERSION ). Depending on the value of BUILD_VERSION , one or another stage of the middle section is selected.

Please note: only BuildKit-based linkers can skip unused branches. In previous versions of linkers, all stages would be built, but before creating the final image, their results would be discarded.

Development / Testing Assistant for Minimum Production Stage

Finally, let's take a look at an example of combining previous templates to demonstrate how to create a Dockerfile that creates a minimal production image and can then use its contents to test and create a development image. Let's start with the basic Dockerfile example:

 FROM golang:alpine AS stage0 … FROM golang:alpine AS stage1 … FROM scratch COPY --from=stage0 /binary0 /bin COPY --from=stage1 /binary1 /bin 

When a minimal production image is created, this is a fairly common option. But what if you also need to get an alternative developer image or run tests with these binaries at the final stage? The first thing that comes to mind is simply to copy similar binaries during the testing and development phases. The problem is this: there is no guarantee that you will test all production binaries in the same combination. At the final stage, something may change, and you will forget to make similar changes at other stages or make an error in the path to copy binary files. In the end, we are not testing a separate binary file, but a final image.

An alternative is to define the development and testing phase after the production phase and copy the entire contents of the production phase. Then use the FROM command for the production phase to again make the default production phase the last step.

 FROM golang:alpine AS stage0 … FROM scratch AS release COPY --from=stage0 /binary0 /bin COPY --from=stage1 /binary1 /bin FROM golang:alpine AS dev-env COPY --from=release / / ENTRYPOINT ["ash"] FROM golang:alpine AS test COPY --from=release / / RUN go test … FROM release 

By default, this Dockerfile will continue to build the minimum default image, while, for example, an assembly with the option --target=dev-env will create an image with a shell containing all the binaries of the final version.

---

I hope this was useful and suggested how to create more efficient multistage Dockerfile files. If you participate in DockerCon2018 and want to learn more about multi-stage builds, Dockerfiles, BuildKit, or any related topics, subscribe to the Hallway track linker or track Docker’s internal meetings on the Contribute and Collaborate tracks or Black Belt .