Docker workflow
Switching the hexlet.io infrastructure to Docker required some effort from us. We abandoned many old approaches and tools, rethought the meaning of many familiar things. What we got in the end, we like. Most importantly, this transition has made it possible to greatly simplify, unify and make it much more supported. In this article we will talk about the scheme for the deployment of infrastructure and deployment, which we finally arrived at, as well as describe the pros and cons of this approach.
Initially, we needed the Docker to run untrusted code in an isolated environment. The task is something similar to what hosters do. We directly in the production collect images, which are then used to start the practice. This, by the way, is the rare case when it is impossible to do according to the principle “one container - one service”. We need all services and all the code for a specific task to be in the same environment. Minimally, in each such container, the supervisord rises and our browser idea. Then everything depends on the task itself: the author can add and deploy at least radishes, even Hadup.
And it turned out that the docker allowed me to create a simple way of assembling practical tasks. First, because if the practice has gathered and started working on the local machine from the author, then it is guaranteed (almost) that it will start in production as well. For isolation. And secondly, despite the fact that many consider the docker file to be a “regular bash” with all the consequences, this is not so. Docker is a prime example of using the functional paradigm in the right places. It provides idempotency, but not in the same way as configuration management systems, due to internal verification mechanisms, but due to immutability. Therefore, in the dockerfile, the usual bash, but it rolls as if it always happens on a fresh basic image, and you do not need to take the previous state into account when changing the image. And caching removes (almost) the problem of waiting for reassembly.
At the moment, this subsystem is essentially a continuous delivery for practical tasks. Perhaps we will make a separate article on this topic if the audience has an interest.
')
After that, we thought about translating to Docker and the rest of our system. There were several reasons. It is clear that in this way we would have achieved greater unification of our system, because the docker has already taken a serious (and not quite trivial) part of the infrastructure.
In fact, there is another interesting case. Many years ago I used chef, after that ansible, which is much simpler. At the same time, I have always come across such a story: if you do not have your own admins, and you do not do the infrastructure and playbooks / kukbukami regularly, then there are often unpleasant situations in cases like:
In this regard, we looked at Docker as a miracle that will save us from these problems. So it happened, in general. The server still has to periodically redistribute from scratch, but much less frequently and, most importantly, we have reached a new level of abstraction. Working at the configuration management system level, we think and manage services, and not parts of which they consist. That is, the control unit is a service, not a package.
Also, the key history of painless deployment is a quick, and, importantly, simple rollback. In the case of Docker, it is almost always fixing the previous version and restarting services.
And last but not least. Building a hexlet is a little more complicated than just compiling assets (we are on rails, yes). We have a massive js-infrastructure that is built using a webpack. Naturally, all this farming should be collected on one server and then just scatter. Capistrano does not allow this.
Almost all we need from configuration management systems is the creation of users, the delivery of keys, configs and images. After switching to docker, playbooks became monotonous and simple: they created users, added configs, sometimes a little crown.
Another very important point is the method of launching containers. Despite the fact that Docker out of the box comes with his supervisor, and Ansible comes with a module for running Docker containers, we still decided not to use these approaches (although we tried). The Docker module in Ansible has many problems , some of which are not at all clear how to solve. This is largely due to the separation of the concepts of creation and start of the container, and the configuration is smeared between these stages.
In the end, we stopped at upstart. It is clear that soon you will still have to go to systemd, but it so happened that we use the ubuntu of the version where upstart is running by default. At the same time we solved the issue of universal logging. Well, and upstart allows you to flexibly configure the way to start the restart of the service, in contrast to the Docker’s restart_always: true.
The most interesting thing here is the service start line:
This is done in order to be able to run the container from the server, without the need to manually register all the parameters. For example, so we can enter the rail console:
There is one subtle point. Unfortunately, the history of teams is lost. To restore performance, you need to prokidyvat relevant files, but, frankly, we did not do it.
By the way, one more advantage of the docker is visible here: all external dependencies are indicated clearly and in one place. If you are not familiar with this approach to configuration, I recommend referring to this document from the company heroku.
In the first line you can see that we don’t need to be steamed about installing ruby, we simply indicate the version we want to use (and for which there is an image, of course).
The containers are started with the - read-only flag, which allows you to control the write to disk. Practice shows that trying to write everything, in completely unexpected places. Below you can see that we have created volume / var / folders, there it writes ruby ​​when creating a temporary directory. But we wrap some sections outside, for example / var / tmp, to fumble data between different versions. This is optional, but simply saves us resources.
Also inside we put the docker in order to manage the docker from the docker. It is necessary, just, for managing images with practice.
Further, literally in four lines, we describe everything that capistrano does as a tool for building an application.
You can raise your own docker distribution (ex-registry), but we are quite satisfied with the docker hub, for which we pay $ 7 a month and get 5 private repositories. He, of course, is far from perfect, and in terms of usability and capabilities. And sometimes the assembly of images instead of 20 minutes is delayed for an hour. In general, you can live, although there are alternative cloud solutions.
The way the application is built varies depending on the deployment environment.
At staging, we use an automated build, which is collected as soon as it sees changes in the staging branch.
As soon as the image is assembled, the docker hub via the webhook notifies zapier , which, in turn, sends the information to Slack. Unfortunately, docker hub does not know how to work directly with Slack (and the developers do not plan to support it).
Deploying is done by the command:
Here is how we see it in slack:
Unlike styling, the production image is not automatically compiled. At the time of readiness, it is being built by manual launch on a special build server. We have this server simultaneously serves as a bastion .
Another difference is the active use of tags. If in staging we always have the latest, then here during assembly we clearly indicate the tag (it is the same version).
Build runs like this:
Deploy production is performed by the command:
In general, the deployment itself is loading the necessary images on the servers, performing migrations and restarting services. Suddenly, it turned out that the entire capistran was replaced by a dozen lines of straight-line code. But at the same time a dozen gems of integration with the capistrana, suddenly, turned out to be simply not needed. The tasks that they performed, most often, turn into one task for ansible.
The first thing you have to give up when working with a docker is to develop it on Mac OS. For normal operation, you need a Vagrant. To customize the environment we have written a special playbook vagrant.yml. For example, in it we install and configure the base, although in production we use RDS.
Unfortunately (and perhaps fortunately) we never managed to set up a normal workflow of development through docker. Too many compromises and difficulties. At the same time, services like postgresql, redis and the like, we still run through it, even during development. And all this stuff continues to be managed through upstart.
From interesting things we put google cadvisor, which, in turn, sent the collected data to influxdb. Periodically, cadvisor began to eat some kind of wild memory and had to restart it with his hands. And then it turned out that the influxdb is good, but the alert on top of it simply does not exist. All this led to the fact that we abandoned any samopal. Now we have a datadog with corresponding plugins connected, and we are very pleased.
After moving to the docker, I immediately had to abandon the quick fixes. Building the image can take up to 1 hour. And it pushes you to a more correct flow, to the possibility of quickly and painlessly rolling back to the previous version.
Sometimes we stumble upon bugs in the docker itself (more often than we would like), for example, right now we cannot switch to 1.6.2 from 1.5 because they still have several unclosed tickets with problems that many people stumble upon.
The changing state of the server when deploying software is the pain point of any configuration system. Docker takes over most of this work, which allows servers to be in a very clean state for a long time, and we don’t have to worry about transition periods. Changing the version of the same Ruby was not only a simple task, but also completely independent of the administrator. And the unification of launch, deployment, deployment, assembly and operation allows us to spend much less time on system maintenance. Yes, aws helps us of course, but this does not negate the advantages of docker / ansible ease of use.
The next step we want to introduce continuous delivery and completely abandon the staging. The idea is that the rollout will first be carried out on the production server available only from inside the company.
PS
Well, for those who are not familiar with the ansible, yesterday we released a basic course .
Prehistory
Initially, we needed the Docker to run untrusted code in an isolated environment. The task is something similar to what hosters do. We directly in the production collect images, which are then used to start the practice. This, by the way, is the rare case when it is impossible to do according to the principle “one container - one service”. We need all services and all the code for a specific task to be in the same environment. Minimally, in each such container, the supervisord rises and our browser idea. Then everything depends on the task itself: the author can add and deploy at least radishes, even Hadup.
And it turned out that the docker allowed me to create a simple way of assembling practical tasks. First, because if the practice has gathered and started working on the local machine from the author, then it is guaranteed (almost) that it will start in production as well. For isolation. And secondly, despite the fact that many consider the docker file to be a “regular bash” with all the consequences, this is not so. Docker is a prime example of using the functional paradigm in the right places. It provides idempotency, but not in the same way as configuration management systems, due to internal verification mechanisms, but due to immutability. Therefore, in the dockerfile, the usual bash, but it rolls as if it always happens on a fresh basic image, and you do not need to take the previous state into account when changing the image. And caching removes (almost) the problem of waiting for reassembly.
At the moment, this subsystem is essentially a continuous delivery for practical tasks. Perhaps we will make a separate article on this topic if the audience has an interest.
')
Infrastructure docker
After that, we thought about translating to Docker and the rest of our system. There were several reasons. It is clear that in this way we would have achieved greater unification of our system, because the docker has already taken a serious (and not quite trivial) part of the infrastructure.
In fact, there is another interesting case. Many years ago I used chef, after that ansible, which is much simpler. At the same time, I have always come across such a story: if you do not have your own admins, and you do not do the infrastructure and playbooks / kukbukami regularly, then there are often unpleasant situations in cases like:
- The configuration management system has been updated (especially with the chef), and you spend two days on it to let everyone down.
- You forgot that there was some kind of software on the server, and with the new knurling conflicts start, or everything falls. Looking for transition states. Well, or how do those who cram cones: “every time on a new server”.
- Redistributing services across servers is a pain, everyone influences each other.
- There are still a thousand smaller reasons, mainly due to the lack of isolation.
In this regard, we looked at Docker as a miracle that will save us from these problems. So it happened, in general. The server still has to periodically redistribute from scratch, but much less frequently and, most importantly, we have reached a new level of abstraction. Working at the configuration management system level, we think and manage services, and not parts of which they consist. That is, the control unit is a service, not a package.
Also, the key history of painless deployment is a quick, and, importantly, simple rollback. In the case of Docker, it is almost always fixing the previous version and restarting services.
And last but not least. Building a hexlet is a little more complicated than just compiling assets (we are on rails, yes). We have a massive js-infrastructure that is built using a webpack. Naturally, all this farming should be collected on one server and then just scatter. Capistrano does not allow this.
Deploying infrastructure
Almost all we need from configuration management systems is the creation of users, the delivery of keys, configs and images. After switching to docker, playbooks became monotonous and simple: they created users, added configs, sometimes a little crown.
Another very important point is the method of launching containers. Despite the fact that Docker out of the box comes with his supervisor, and Ansible comes with a module for running Docker containers, we still decided not to use these approaches (although we tried). The Docker module in Ansible has many problems , some of which are not at all clear how to solve. This is largely due to the separation of the concepts of creation and start of the container, and the configuration is smeared between these stages.
In the end, we stopped at upstart. It is clear that soon you will still have to go to systemd, but it so happened that we use the ubuntu of the version where upstart is running by default. At the same time we solved the issue of universal logging. Well, and upstart allows you to flexibly configure the way to start the restart of the service, in contrast to the Docker’s restart_always: true.
upstart.unicorn.conf.j2
description "Unicorn"
start on filesystem or runlevel [2345]
stop on runlevel [! 2345]
env HOME = / home / {{run_user}}
# change to match your deployment user
setuid {{run_user}}
setgid team
respawn
respawn limit 3 30
pre-start script
. / etc / environment
export HEXLET_VERSION
/ usr / bin / docker pull hexlet / hexlet - {{rails_env}}: $ HEXLET_VERSION
/ usr / bin / docker rm -f unicorn || true
end script
pre-stop script
/ usr / bin / docker rm -f unicorn || true
end script
script
. / etc / environment
export HEXLET_VERSION
RUN_ARGS = '- name unicorn' ~ / apprunner.sh bundle exec unicorn_rails -p {{unicorn_port}}
end script
The most interesting thing here is the service start line:
RUN_ARGS='--name unicorn' ~/apprunner.sh bundle exec unicorn_rails -p {{ unicorn_port }} This is done in order to be able to run the container from the server, without the need to manually register all the parameters. For example, so we can enter the rail console:
RUN_ARGS='-it' ~./apprunner.sh bundle exec rails c apprunner.sh.j2
#!/usr/bin/env bash . /etc/environment export HEXLET_VERSION ${RUN_ARGS:=''} COMMAND="/usr/bin/docker run --read-only --rm \ $RUN_ARGS \ -v /tmp:/tmp \ -v /var/tmp:/var/tmp \ -p {{ unicorn_port }}:{{ unicorn_port }} \ -e AWS_REGION={{ aws_region }} \ -e SECRET_KEY_BASE={{ secret_key_base }} \ -e DATABASE_URL={{ database_url }} \ -e RAILS_ENV={{ rails_env }} \ -e SMTP_USER_NAME={{ smtp_user_name }} \ -e SMTP_PASSWORD={{ smtp_password }} \ -e SMTP_ADDRESS={{ smtp_address }} \ -e SMTP_PORT={{ smtp_port }} \ -e SMTP_AUTHENTICATION={{ smtp_authentication }} \ -e DOCKER_IP={{ docker_ip }} \ -e STATSD_PORT={{ statsd_port }} \ -e DOCKER_HUB_USERNAME={{ docker_hub_username }} \ -e DOCKER_HUB_PASSWORD={{ docker_hub_password }} \ -e DOCKER_HUB_EMAIL={{ docker_hub_email }} \ -e DOCKER_EXERCISE_PREFIX={{ docker_exercise_prefix }} \ -e FACEBOOK_CLIENT_ID={{ facebook_client_id }} \ -e FACEBOOK_CLIENT_SECRET={{ facebook_client_secret }} \ -e HEXLET_IDE_VERSION={{ hexlet_ide_image_tag }} \ -e CDN_HOST={{ cdn_host }} \ -e REFILE_CACHE_DIR={{ refile_cache_dir }} \ -e CONTAINER_SERVER={{ container_server }} \ -e CONTAINER_PORT={{ container_port }} \ -e DOCKER_API_VERSION={{ docker_api_version }} \ hexlet/hexlet-{{ rails_env }}:$HEXLET_VERSION $@" eval $COMMAND There is one subtle point. Unfortunately, the history of teams is lost. To restore performance, you need to prokidyvat relevant files, but, frankly, we did not do it.
By the way, one more advantage of the docker is visible here: all external dependencies are indicated clearly and in one place. If you are not familiar with this approach to configuration, I recommend referring to this document from the company heroku.
DECORIZATION
Dockerfile
Dockerfile
FROM ruby: 2.2.1 RUN mkdir -p / usr / src / app WORKDIR / usr / src / app ENV RAILS_ENV production ENV REFILE_CACHE_DIR / var / tmp / uploads RUN curl -sL https://deb.nodesource.com/setup | bash - RUN apt-get update -qq \ && apt-get install -yqq apt-transport-https libxslt-dev libxml2-dev imagejs nodejs RUN echo deb https://get.docker.com/ubuntu docker main> /etc/apt/sources.list.d/docker.list \ && apt-key adv --keyserver hkp: //keyserver.ubuntu.com: 80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9 \ && apt-get update -qq \ && apt-get install -qqy lxc-docker-1.5.0 # bundle config build.rugged - use-system-libraries # bundle config build.nokogiri - use-system-libraries COPY Gemfile / usr / src / app / COPY Gemfile.lock / usr / src / app / COPY package.json / usr / src / app / # without development test RUN npm install RUN bundle install --without development test COPY. / usr / src / app RUN ./node_modules/gulp/bin/gulp.js webpack_production RUN bin / rake assets: precompile VOLUME / usr / src / app / tmp VOLUME / var / folders
In the first line you can see that we don’t need to be steamed about installing ruby, we simply indicate the version we want to use (and for which there is an image, of course).
The containers are started with the - read-only flag, which allows you to control the write to disk. Practice shows that trying to write everything, in completely unexpected places. Below you can see that we have created volume / var / folders, there it writes ruby ​​when creating a temporary directory. But we wrap some sections outside, for example / var / tmp, to fumble data between different versions. This is optional, but simply saves us resources.
Also inside we put the docker in order to manage the docker from the docker. It is necessary, just, for managing images with practice.
Further, literally in four lines, we describe everything that capistrano does as a tool for building an application.
Image hosting
You can raise your own docker distribution (ex-registry), but we are quite satisfied with the docker hub, for which we pay $ 7 a month and get 5 private repositories. He, of course, is far from perfect, and in terms of usability and capabilities. And sometimes the assembly of images instead of 20 minutes is delayed for an hour. In general, you can live, although there are alternative cloud solutions.
Build and Depla
The way the application is built varies depending on the deployment environment.
At staging, we use an automated build, which is collected as soon as it sees changes in the staging branch.
As soon as the image is assembled, the docker hub via the webhook notifies zapier , which, in turn, sends the information to Slack. Unfortunately, docker hub does not know how to work directly with Slack (and the developers do not plan to support it).
Deploying is done by the command:
ansible-playbook deploy.yml -i staging.ini Here is how we see it in slack:
Unlike styling, the production image is not automatically compiled. At the time of readiness, it is being built by manual launch on a special build server. We have this server simultaneously serves as a bastion .
Another difference is the active use of tags. If in staging we always have the latest, then here during assembly we clearly indicate the tag (it is the same version).
Build runs like this:
ansible-playbook build.yml -i production.ini -e 'hexlet_image_tag=v100' build.yml
- hosts: bastions
gather_facts: no
vars:
clone_dir: / var / tmp / hexlet
tasks:
- git:
repo: git@github.com: Hexlet / hexlet.git
dest: '{{clone_dir}}'
accept_hostkey: yes
key_file: / home / {{run_user}} /. ssh / deploy_rsa
become: yes
become_user: '{{run_user}}'
- shell: 'cd {{clone_dir}} && docker build -t hexlet / hexlet-production: {{hexlet_image_tag}}.'
become: yes
become_user: '{{run_user}}'
- shell: 'docker push hexlet / hexlet-production: {{hexlet_image_tag}}'
become: yes
become_user: '{{run_user}}'
Deploy production is performed by the command:
ansible-playbook deploy.yml -i production.ini -e 'hexlet_image_tag=v100' deploy.yml
- hosts: localhost
gather_facts: no
tasks:
- local_action:
module: slack
domain: hexlet.slack.com
token: {{slack_token}}
msg: "deploy started: {{rails_env}}: {{hexlet_image_tag}}"
channel: "#operation"
username: "{{ansible_ssh_user}}"
- hosts: appservers
gather_facts: no
tasks:
- shell: docker pull hexlet / hexlet - {{rails_env}}: {{hexlet_image_tag}}
become: yes
become_user: '{{run_user}}'
- name: update hexlet version
become: yes
lineinfile:
regexp: "HEXLET_VERSION"
line: "HEXLET_VERSION = {{hexlet_image_tag}}"
dest: / etc / environment
backup: yes
state: present
- hosts: jobservers
gather_facts: no
tasks:
- become: yes
become_user: '{{run_user}}'
run_once: yes
delegate_to: '{{migration_server}}'
shell:>
docker run --rm
-e 'SECRET_KEY_BASE = {{secret_key_base}}'
-e 'DATABASE_URL = {{database_url}}'
-e 'RAILS_ENV = {{rails_env}}'
hexlet / hexlet - {{rails_env}}: {{hexlet_image_tag}}
rake db: migrate
- hosts: webservers
gather_facts: no
tasks:
- service: name = nginx state = running
become: yes
tags: nginx
- service: name = unicorn state = restarted
become: yes
tags: [unicorn, app]
- hosts: jobservers
gather_facts: no
tasks:
- service: name = activejob state = restarted
become: yes
tags: [activejob, app]
- hosts: localhost
gather_facts: no
tasks:
- name: "Send deploy to honeybadger"
local_action: shell cd .. && bundle exec honeybadger deploy --environment = {{rails_env}}
- local_action:
module: slack
domain: hexlet.slack.com
token: {{slack_token}}
msg: "deploy completed ({{rails_env}})"
channel: "#operation"
username: "{{ansible_ssh_user}}"
# link_names: 0
# parse: 'none'
In general, the deployment itself is loading the necessary images on the servers, performing migrations and restarting services. Suddenly, it turned out that the entire capistran was replaced by a dozen lines of straight-line code. But at the same time a dozen gems of integration with the capistrana, suddenly, turned out to be simply not needed. The tasks that they performed, most often, turn into one task for ansible.
Development
The first thing you have to give up when working with a docker is to develop it on Mac OS. For normal operation, you need a Vagrant. To customize the environment we have written a special playbook vagrant.yml. For example, in it we install and configure the base, although in production we use RDS.
Unfortunately (and perhaps fortunately) we never managed to set up a normal workflow of development through docker. Too many compromises and difficulties. At the same time, services like postgresql, redis and the like, we still run through it, even during development. And all this stuff continues to be managed through upstart.
Monitoring
From interesting things we put google cadvisor, which, in turn, sent the collected data to influxdb. Periodically, cadvisor began to eat some kind of wild memory and had to restart it with his hands. And then it turned out that the influxdb is good, but the alert on top of it simply does not exist. All this led to the fact that we abandoned any samopal. Now we have a datadog with corresponding plugins connected, and we are very pleased.
Problems
After moving to the docker, I immediately had to abandon the quick fixes. Building the image can take up to 1 hour. And it pushes you to a more correct flow, to the possibility of quickly and painlessly rolling back to the previous version.
Sometimes we stumble upon bugs in the docker itself (more often than we would like), for example, right now we cannot switch to 1.6.2 from 1.5 because they still have several unclosed tickets with problems that many people stumble upon.
Total
The changing state of the server when deploying software is the pain point of any configuration system. Docker takes over most of this work, which allows servers to be in a very clean state for a long time, and we don’t have to worry about transition periods. Changing the version of the same Ruby was not only a simple task, but also completely independent of the administrator. And the unification of launch, deployment, deployment, assembly and operation allows us to spend much less time on system maintenance. Yes, aws helps us of course, but this does not negate the advantages of docker / ansible ease of use.
Plans
The next step we want to introduce continuous delivery and completely abandon the staging. The idea is that the rollout will first be carried out on the production server available only from inside the company.
PS
Well, for those who are not familiar with the ansible, yesterday we released a basic course .
Source: https://habr.com/ru/post/258815/
All Articles