12 Fractured Apps and Docker

Over the years I have witnessed how more and more people are supporting the manifesto of the 12 Factor App and are starting to implement the provisions described there. This has led to the emergence of applications that have been greatly simplified in deployment and management. However, examples of practical application of these 12 factors were quite rare in the vast Internet.


While working with Docker, the benefits of the 12 Factor App (12FA) have become more tangible for me. For example, 12FA recommends that logging be configured for standard output and processed as a general event flow. Have you ever used the docker logs ? This is the 12FA in action!

12FA also recommends using environment variables to configure the application. Docker does this trivially, providing the ability to set environment variables programmatically when creating containers.
')

The Docker and 12 Factor App is a killer combination that provides a quick overview of the design and deployment of future applications.

Docker also partly simplifies the movement of legacy applications into a container. I say “partly”, because ultimately you have to slightly edit Docker containers, with the result that 2 GB images of containers are created on top of a full-fledged Linux distribution.

Unfortunately, legacy applications that you may be working with right now have many flaws, especially around the startup process. Applications, even modern ones, have too many dependencies and because of this they cannot provide a clean launch. Applications that require access to an external database usually initiate a connection to the database during startup. However, if this database was unavailable, or temporarily unavailable, then many of the applications simply will not start. If you are lucky, you can get an error message with details that will help you in troubleshooting.

Many applications that are packaged in Docker have some minor flaws. This is more like microcracks - applications continue to work, but can cause hellish torment when working with them.

This behavior of applications forces to resort to complex deployment processes and contributes to the development of tools such as Puppet or Ansible. Configuration management tools help solve various problems, for example, the inaccessibility of the database. They run the database on which this application depends before starting the application itself. Most likely it looks like sticking adhesive tape on a lacerated wound. The application should simply repeat the connection to the database, using a sort of classification for the returned errors and of course error logging. In this case, there will be two options: either you can return the database online, or your company will simply go bankrupt.

Another problem for applications moved to Docker is in the configuration files. Many applications, even modern ones, still rely on configuration files located locally on disks. The most commonly used solution is to deploy additional new containers that link configuration files to a container image.

Do not do this.

If you choose this solution, you will end up with an infinite number of container images, named something like this:

• application-v2 – prod-01022015
• application-v2-dev-02272015

Soon you will need to look for tools to manage so many images.

Moving to Docker gave people the mistaken belief that they no longer need configuration management in any way. I tend to agree with this, there is no need to use Puppet, Chef or Ansible when creating images, but there is still a need to manage configuration settings during operation.

Similar logic is used to end the frequent use of configuration management systems in order to avoid init systems in favor of the docker run .

To compensate for the lack of configuration management tools and robust init systems, Docker users are turning to shell scripts to disguise the flaws of the application around the bootstrap and the startup process.

As soon as you transfer everything to Docker and refuse to use tools that do not have the Docker logo, you will put yourself in an impossible position.

application

We now turn to the sample application to demonstrate some common tasks when starting a typical application. The example performs the following tasks during startup:

• Loads configuration parameters from JSON encoded config. file
• Gives access to the working directory
• Connects to an external MySQL database.

 package main import ( "database/sql" "encoding/json" "fmt" "io/ioutil" "log" "net" "os" _ "github.com/go-sql-driver/mysql" ) var ( config Config db *sql.DB ) type Config struct { DataDir string `json:"datadir"` // Database settings. Host string `json:"host"` Port string `json:"port"` Username string `json:"username"` Password string `json:"password"` Database string `json:"database"` } func main() { log.Println("Starting application...") // Load configuration settings. data, err := ioutil.ReadFile("/etc/config.json") if err != nil { log.Fatal(err) } if err := json.Unmarshal(data, &config); err != nil { log.Fatal(err) } // Use working directory. _, err = os.Stat(config.DataDir) if err != nil { log.Fatal(err) } // Connect to database. hostPort := net.JoinHostPort(config.Host, config.Port) dsn := fmt.Sprintf("%s:%s@tcp(%s)/%s?timeout=30s", config.Username, config.Password, hostPort, config.Database) db, err = sql.Open("mysql", dsn) if err != nil { log.Fatal(err) } if err := db.Ping(); err != nil { log.Fatal(err) } } 

The full source code is available on GitHub .

As you can see, there is nothing special here, but if you look closely, you can see that this application will automatically load only under certain conditions. If the configuration file or working directory is missing, or the database is not available during startup, the above application will not start.

Let's deploy the sample application through Docker and explore it.

Create an application using the docker build :

 $ GOOS=linux go build -o app 

Now, create a container from the app: v1 Docker image using the docker run :

 FROM scratch MAINTAINER Kelsey Hightower <kelsey.hightower@gmail.com> COPY app /app ENTRYPOINT ["/app"] 

All I'm doing here is copying the binary of the application to the right place. This container image will use the base image script, resulting in a minimal container Docker image suitable for deploying our application.

Create an image using the docker build :

 $ docker build -t app:v1 . 

Finally, create a container from the app: v1 image using the docker run :

 $ docker run --rm app:v1 2015/12/13 04:00:34 Starting application... 2015/12/13 04:00:34 open /etc/config.json: no such file or directory 

Let the pain begin! Right, almost at the start, I ran into the first startup problem. Please note that the application does not start due to the missing /etc/config.json configuration file. I can fix this by mounting the configuration file at runtime:

 $ docker run --rm \ -v /etc/config.json:/etc/config.json \ app:v1 2015/12/13 07:36:27 Starting application... 2015/12/13 07:36:27 stat /var/lib/data: no such file or directory 

Another mistake! This time, the application fails to start because the /var/lib/data directory does not exist. I can easily bypass the missing directory by mounting another host directory in the container:

 $ docker run --rm \ -v /etc/config.json:/etc/config.json \ -v /var/lib/data:/var/lib/data \ app:v1 2015/12/13 07:44:18 Starting application... 2015/12/13 07:44:48 dial tcp 203.0.113.10:3306: i/o timeout 

We are making progress, but I forgot to set up database access for this instance.

This is the point where some people start using configuration management tools to ensure that all these dependencies are started before the application starts. Although this works, it is still to some extent overkill and often the wrong approach to solving application-level problems.

I hear silent screams from hipsters “sys-admins”, impatiently waiting to suggest using the Docker user entry point to solve our bootstrap problems.

User entry point to the rescue.

One way to solve our startup problems is to create a shell script and use it as a Docker entry point, instead of the actual application. Here is a short list of things we can do using a shell script as an entry point:

• Generate the required / etc / config.json configuration file
• create the required / var / lib / data directory
• test the database connection while it is available

The following shell script deals with the first two elements, adding the ability to use environment variables along with the /etc/config.json configuration file and creating the missing /var/lib/data directory during the startup process. The script executes the sample application as the final stage, retaining the original behavior when the application is started by default.

 #!/bin/sh set -e datadir=${APP_DATADIR:="/var/lib/data"} host=${APP_HOST:="127.0.0.1"} port=${APP_PORT:="3306"} username=${APP_USERNAME:=""} password=${APP_PASSWORD:=""} database=${APP_DATABASE:=""} cat <<EOF > /etc/config.json { "datadir": "${datadir}", "host": "${host}", "port": "${port}", "username": "${username}", "password": "${password}", "database": "${database}" } EOF mkdir -p ${APP_DATADIR} exec "/app" 

Now the image can be restored using the following Docker file:

 FROM alpine:3.1 MAINTAINER Kelsey Hightower <kelsey.hightower@gmail.com> COPY app /app COPY docker-entrypoint.sh /entrypoint.sh ENTRYPOINT ["/entrypoint.sh"] 

Notice that the custom shell script is copied to the Docker image and used as an entry point instead of the application's binary file.

Create an app: v2 image using the docker build command:

 $ docker build -t app:v2 . 

Now perform the next step:

 $ docker run --rm \ -e "APP_DATADIR=/var/lib/data" \ -e "APP_HOST=203.0.113.10" \ -e "APP_PORT=3306" \ -e "APP_USERNAME=user" \ -e "APP_PASSWORD=password" \ -e "APP_DATABASE=test" \ app:v2 2015/12/13 04:44:29 Starting application... 

User entry point is working. Using only environment variables, we are able to configure and run our application.

But why do we do it?

Why do we have to use such a complex wrapper script? Some will say that it is much easier to write this functionality in a shell than to implement it in an application. But it's not just about managing shell scripts. Notice another difference between v1 and v2 files?

 FROM alpine:3.1 

The v2 file uses alpine - the base image to provide a scripting environment, but it doubles the size of our Docker image:

 $ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE app v2 1b47f1fbc7dd 2 hours ago 10.99 MB app v1 42273e8664d5 2 hours ago 5.952 MB 

Another disadvantage of this approach is the inability to use the configuration file with the image. We can continue to write the script and add support for the configuration file and the environment variable, but all this will simply lose its functionality when the wrapper script is out of sync with the application. But there is another way to solve this problem.

Programming will save all.

Yes, good old programming. Each of the shell script tasks of the Docker entry point can be processed directly by the application.

Don't get me wrong, using the entry point script is good for applications that you do not control. But, when you rely on entry point scripts for applications, you add another level of complexity to the application deployment process without any reason.

Configuration files must be optional

I think that there is absolutely no reason to use configuration files from the end of the 90s. I suggest loading the configuration file if it exists and rolls back to the default settings. The following code fragment does just that.

 // Load configuration settings. data, err := ioutil.ReadFile("/etc/config.json") // Fallback to default values. switch { case os.IsNotExist(err): log.Println("Config file missing using defaults") config = Config{ DataDir: "/var/lib/data", Host: "127.0.0.1", Port: "3306", Database: "test", } case err == nil: if err := json.Unmarshal(data, &config); err != nil { log.Fatal(err) } default: log.Println(err) } 

Use environment variable for configuration.

This is one of the simplest things you can do directly in your application. The following code snippet uses environment variables to override configuration settings.

 log.Println("Overriding configuration from env vars.") if os.Getenv("APP_DATADIR") != "" { config.DataDir = os.Getenv("APP_DATADIR") } if os.Getenv("APP_HOST") != "" { config.Host = os.Getenv("APP_HOST") } if os.Getenv("APP_PORT") != "" { config.Port = os.Getenv("APP_PORT") } if os.Getenv("APP_USERNAME") != "" { config.Username = os.Getenv("APP_USERNAME") } if os.Getenv("APP_PASSWORD") != "" { config.Password = os.Getenv("APP_PASSWORD") } if os.Getenv("APP_DATABASE") != "" { config.Database = os.Getenv("APP_DATABASE") } 

Manage the working directory of the application.

Instead of shifting responsibility for work and connectivity with directories to external tools or to script entry points, your application should manage them directly. If for some reason something does not work, do not forget to set up error logging with details:

 // Use working directory. _, err = os.Stat(config.DataDir) if os.IsNotExist(err) { log.Println("Creating missing data directory", config.DataDir) err = os.MkdirAll(config.DataDir, 0755) } if err != nil { log.Fatal(err) } 

Eliminate the need to start services in a specific order.

Remove the deployment requirement for your application in a specific order. I have seen that in many deployment guides for various applications there is an instruction to launch the application after the database is started, otherwise it will result in a zero result.

You can get rid of this requirement like this:

 $ docker run --rm \ -e "APP_DATADIR=/var/lib/data" \ -e "APP_HOST=203.0.113.10" \ -e "APP_PORT=3306" \ -e "APP_USERNAME=user" \ -e "APP_PASSWORD=password" \ -e "APP_DATABASE=test" \ app:v3 2015/12/13 05:36:10 Starting application... 2015/12/13 05:36:10 Config file missing using defaults 2015/12/13 05:36:10 Overriding configuration from env vars. 2015/12/13 05:36:10 Creating missing data directory /var/lib/data 2015/12/13 05:36:10 Connecting to database at 203.0.113.10:3306 2015/12/13 05:36:40 dial tcp 203.0.113.10:3306: i/o timeout 2015/12/13 05:37:11 dial tcp 203.0.113.10:3306: i/o timeout 

Notice in the above output I am not able to connect to a working target database located at 203.0.113.10 ..

Run the following command to grant access to the MySQL database:

 $ gcloud sql instances patch mysql \ --authorized-networks "203.0.113.20/32" 

The application is able to connect to the database and complete the startup process.

 2015/12/13 05:37:43 dial tcp 203.0.113.10:3306: i/o timeout 2015/12/13 05:37:46 Application started successfully. 

The code to execute looks like this:

 // Connect to database. hostPort := net.JoinHostPort(config.Host, config.Port) log.Println("Connecting to database at", hostPort) dsn := fmt.Sprintf("%s:%s@tcp(%s)/%s?timeout=30s", config.Username, config.Password, hostPort, config.Database) db, err = sql.Open("mysql", dsn) if err != nil { log.Println(err) } var dbError error maxAttempts := 20 for attempts := 1; attempts <= maxAttempts; attempts++ { dbError = db.Ping() if dbError == nil { break } log.Println(dbError) time.Sleep(time.Duration(attempts) * time.Second) } if dbError != nil { log.Fatal(dbError) } 

There is nothing special here. I simply repeat the connection to the database and increase the time between each attempt.

Great, we got a startup process with a friendly message in the log that the application started correctly.

 log.Println("Application started successfully.") 

Believe me, your system administrator will thank you.

You can find the link to the original source here .