Update : this article has a sequel translated by @achekalin . In what order to read - at your discretion: in this article, you can enjoy the author’s extensive popobol, and in the continuation - from his conclusions.

Translator's note : in a previous article about preparing for devops conferences , Gryphon88 asked a reasonable question : how to distinguish between cutting-edge and hyip? The following article is filled with juicy unclouded tantrum, which is so nice to read in the morning, sipping a cup of coffee. The downside is that it was written in November 2016, but the nettle does not age. If after reading you want to add, there are comments on Hacker News . And you, username, the same hell? Write in the comments. So, let's begin.

The first time I met with Docker was in early 2015. We experimented with him in order to understand why we could use it. At that time it was impossible to start the container in the background, there were no commands to see what was running, go under debug or SSH inside the container. The experiment was fast, Docker was considered useless and more like an alpha or a prototype than a release.

We squander our history until 2016. New job, new company, and HYIP around the docker got mad. The developers have already rolled out docker to production, so it will not be possible to escape from it. The good news is that the run command is finally working, we can start and stop containers. It moves!

We have 12 docked applications running on the prod right at the time of this writing, spread on 31 AWS hosts (one application per host, then I’ll explain why).

This article tells how we traveled with Docker - a journey full of dangers and unexpected turns.

Docker: Production problems

Docker: Breakdowns and Regressions in Updates

We used the following versions (or at least tried to do this):
1.6 => 1.7 => 1.8 => 1.9 => 1.10 => 1.11 => 1.12

Every new version has broken something. At the beginning of the docker 1.6, we launched one application.

And they were updated only after 3 months, because a fix was needed, available only in newer versions. Vetka 1.6 was already abandoned.

Versions 1.7 and 1.8 did not start. We switched to 1.9 only to find a critical bug in this version after two weeks, so I had to (again!) Upgrade to 1.10.

Small regressions are constantly occurring between the Docker versions. He constantly breaks in an unpredictable way in unexpected places.

Most of the tricky regressions that we ran into turned out to be connected to the network. Docker completely abstracts the host network. From this happens a big mess with port forwarding, DNS hacks and virtual networks.

Bonus : Docker was removed from the official Debian repositories, so the package was renamed from docker.io to docker-engine. The documentation created before this change is outdated.

Docker: Old images can not be deleted

The most desirable, and painfully missing feature is the command to delete old images (older than X days, or not used X days, it does not matter). Disk space is a critical issue, given that images are often updated and can take up more than 1 Gb.

The only way to clear a place is to run the next hack every day, most likely by crown:

 docker images -q -a | xargs --no-run-if-empty docker rmi  

. , , . , .

. — , .

, . API , 6 . , — "" docker rmi. , . — , , .

: (, )

, , , .

Debian Stable c backports . Debian Jessie 3.16.7-ckt20-1 ( 2015). , - ( , ).

:

. () — AUFS.

AUFS . , .

( ) linux-3.16.x. .

. . AUFS - 2016. , , . ( ), .

, . CI , 2 .

2016 AUFS. , . AUFS linux-3.16.x.

• Debian Stable 3.16. . Debian Testing ( ).
• Ubuntu LTS 3.19. , . , , - .
• RHEL/CentOS-6 2.x, RHEL/CentoS-7 — 3.10 ( RedHat).

Linux 4.x:

, AUFS , . , AFUS .

, , - , . AUFS .

( )

.

.

.

AUFS? , .

( )

.

.

.

, overlay.

"OverlayFS — union , AUFS. AUFS, OverlayFS , Linux 3.18, ". — Docker OverlayFS driver

, . .

, "overlay" — ( Linux), , ( ). — (, ). , .

Overlay:

— , . , Linux ext3 ext4. , , ext4 .

1 — . , -, , — , .

. . .

Overlay .

.

.

.

Overlay2!

" overlay2 overlay, Linux 4.0 , docker 1.12" — "Overlay vs Overlay2 storage drivers"

— . . , ! , .

. , .

: Overlay Overlay2. . . . :) , , , , , , .

:

2 2016, 9 ( ). .

, apt-get update ( ) , , "Error https://apt.dockerproject.org/ Hash Sum mismatch".

. , . Debian Ubuntu, .

, / / — . . , .

. : " . , , , (8 — . ), ."

. CI , , , . — -, .

, .

, 3 . , , .

.

5 ( ).

7 , . — GitHub. . ( ?) , .

Docker Registry

.

CI ===> ( ) ===> docker registry
<=== <=== docker registry

, . , . , ( !). .

3 :

• Registry v1 —
• Registry v2 — Go, 2015
• Trusted Registry — (?) , , — , .

:

v2 — . v1 v2.

(!) . , , , .

v2 . , , .

: API . .

v2 API. , 9 , .

:

. — , . ( , ).

. 50 GB .

. , , S3.

: S3 ( ).

3 . , . ( , CI).

: .

. API . ( v2 API. ).

:

:

• -

, : , , , , API…

, 1 , , , ( ) . , , , ( ) ( ).

" , - , " —

-

-. , . . , .

- — . , , , , , . ( ).

. , , , . - -, . , .

. . , . , .

: . .

, , "core".

. . - , .

, . , , . . ( , , , ).

. . . , .

. , , , 96 . , , .

— , , . .

DBA

stateless. , — , . , . , . .

. , . , - .

. , by design.

. ?

, . , . — IO , — . , , ( ) , , ( ).

: , , , , .

: , .

, , : " ", . .

: . — : , , .

, . , .

, , stateless - . , , .

, , , . , - , . , .

: , 23M (M — ). , , .

— . , .

:

:

, , , , . , , .

ansible '*' -m shell -a "uname -a"

. .

, .

3

, 2 , .

- , 3 . , 2 .

, CI . ( , ). . , 3 .

, , .

, . , .

, , , .

. , .

, . , stateless , - .

. (: .)

, , 5 . . .

Docker

— + + + .

. , , ( , ?). , .

: .

5 , , Docker v1.0 28 , .

— 3 , — 18-36 . , . , , , .

5. .

: , . , .

stateless . Docker Image, AMI (Amazon Machine Image — . .). , .

, Docker AWS, .

-, — , () . ( , , , , 1 ).

. . /, . ( AWS t2.nano, 5 512MB 5% CPU).

-, , (create/stop/start/rolling-update/canary-release/blue-green-deployment). . ( Nomad/Mesos/Kubernetes, ).

AWS , . , .

, AMI (: Packer AMI). AMI AWS, , . . .

: , .

CoreOS

: Docker and CoreOS , .

, . , .

( ?) . , , .

: CoreOS.

, ( CoreOS) ( Debian). , ( ) , .

(, , , , , ). , , .

: CoreOS .

Kubernetes

() , , , , , -, , .

, . . - .

, , .

• Mesos Docker
• Docker Swarm
• Nomad
• Kubernetes

Kubernetes — , . , (, Google , , , ).

Kubernetes , . , . , - , .

, Kubernetes — . — ( , , , () ).

, Kubernetes. , ?

: Kubernetes.

: Kubernetes . . (, Kubernetes , CoreOS).

Google Cloud: Google Container Engine

, : + + , Kubernetes. — .

: Google Container Engine. Kubernetes ( Docker) , Google Cloud.

Google , , , . , , , , . , .

, — . , (, ), Google Container Engine.

: Google Cloud, , AWS. , .. .

Google Container Engine: , Google Cloud — , AWS — ( .. 33% 3 IOPS ).

Disclaimer (, !)

- . — . , , ( — ).

, , (?) .

, "", , , .

, , , 10 . , .

, — . — . , , ( ).