Geekly Articles each Day
📜 ⬆️ ⬇️

Deploy using Salt


Until now, in many deploy companies, it creates big problems and can take days, weeks, and in particularly neglected cases, months. But the situation is not hopeless. There are many tools and practices that can help in this difficult task. But these tools most often cannot be mastered in one or two days, and the terms are burning.

What you usually want:

All this we will achieve on a bunch of Salt + Vagrant on the example of the Django project. But most techniques will be useful to developers not only in Python, but in other languages.


Immediately give a link to the source: bitbucket.org/awnion/salt-django-deploy
')

What is Salt?


If you are familiar with Salt, you can skip this section.

Salt is a powerful enough tool for cluster management (cluster orchestration), but in my personal opinion even use on one machine is fully justified and will not be overkil (roughly speaking, if your team has exactly 1 developer, this does not mean that you should not use version control system).

Salt states are YAML files with the sls extension, which describe what state the machine should be in. For example, this file should be located here, this service should be launched, this user should have such rights, and so on. In Salt, you can maintain the state of not only system utilities (apt, rpm, cron, init scripts and various configs), but also, for example, you can check whether such a user exists in RabbitMQ, is the latest version of the git repository, all packages are in your virtualenv and so on. A complete list of states can be found here docs.saltstack.com/ref/states/all , and in my opinion it is very impressive.

Some facts about Salt



Dev configuration


In my opinion it is difficult to overestimate the importance of a good and convenient development environment. But it can take a couple of days to set up “everything for yourself”, or even a week. Let's save these days to our colleagues in the future and create a configuration that will allow us to raise the current version of the project into one command:
git clone <repo_url> && cd <repo_name> && vagrant up

Okay, these are actually 3 teams, but if you can do it better, I shake your hand.

So, we will build our dev configuration on Vagrant (for those who are not familiar with Vagrant, I strongly recommend to get acquainted):
 mkdir my_app && cd my_app git init vagrant init

The git repository and config for Vagrant appeared in our my_app folder.

Next, replace the Vagrantfile content with the following:
 Vagrant.configure("2") do |config| config.vm.box = "precise64" config.vm.hostname = "dev-my-app" config.vm.network :private_network, ip: '3.3.3.3' config.vm.synced_folder "salt/roots/salt", "/srv/salt/" config.vm.synced_folder "salt/roots/pillar", "/srv/pillar/" config.vm.provision :salt do |salt| salt.minion_config = "salt/minion" salt.run_highstate = true end end


This config will allow you to create a guest machine on Ubuntu, in the config we set the host name and IP, as well as determine which folders to synchronize and indicated that we will use salt to bring our machine to the desired state (by the way, the root of our project will be synchronized with the guest's / vagrant folder).

You can find out more about what is happening here.

So, we are ready to begin to describe the state. Create a salt / minion file with the following content:
 file_client: local


In fact, we say that this machine stores its state itself. By default, salt is configured so that it takes state files from the master server, and locally, they only cache them somewhere in / var / cache / salt. Therefore, if you do not want something custom, then this file on the prod machine will most likely not be needed at all.

Now create two folders:
 mkdir -p salt/roots/pillar mkdir -p salt/roots/salt

The first one will store various variables, and the second one is the state folder for our guest machine.

Create a file salt / roots / salt / top.sls
 base: 'dev-my-app': - nginx - python - supervisor


As you might guess sls is very similar to yaml. But the main difference here is that sls are also a jinja template with all the consequences (then you will see that this really benefits).

base is the name of the configuration of the states of our imaginary cluster. dev-my-app is the hostname of our guest machine. It uses pattern matching, that is, we could specify 'dev- *', and all the states below would apply to all machines like dev-alpha, dev-foobar, etc. The following is a list of states that we will need to describe.

Create the declared states of python, nginx and supervisor:

salt / roots / salt / python.sls
 #    ,   python  python-virtualenv # ,    --      python: pkg: - installed - names: - python - python-virtualenv


salt / roots / salt / nginx.sls
 #    nginx     ,   require # ,   service     pkg nginx: pkg: - installed service: - running - reload: True #   reload - require: - pkg: nginx


salt / roots / salt / supervisor.sls
 supervisor: pkg: - installed service: - running - require: - pkg: supervisor


So, you can already run “ vagrant up ”. This procedure will download the Ubuntu image (if you do not already have it in the image cache), set the salt there and start the state synchronization.
Now we have python, supervisor and nginx on our guest machine.
You can check it by going to the machine via vagrant ssh or by going to 3.3.3.3

So far everything seems to be simple. We continue:

Create pillar variables:

salt / roots / pillar / top.sls
 base: 'dev-my-app': - my-app


salt / roots / pillar / my-app.sls
 my_app: gunicorn_bind: 127.0.0.1:8000 dns_name: dev.my-app.com venv_dir: /home/vagrant/my_app_env work_dir: /vagrant

The first file says that the dev-my-app host has been assigned variables from the my-app config. The second file is the variables themselves.

Now we will create a folder for the states of the configs of our Django application:
 mkdir -p salt/roots/salt/my_app


salt / roots / salt / my_app / init.sls
 {% set my_app = pillar['my_app'] %} my_app.venv: virtualenv.managed: - name: {{ my_app['venv_dir'] }} - system_site_packages: False - require: - pkg: python my_app.pip: pip.installed: - bin_env: {{ my_app['venv_dir'] }} - names: - Django==1.6 - gunicorn==18.0 - require: - virtualenv: my_app.venv my_app.nginx.conf: file.managed: - name: /etc/nginx/sites-enabled/my_app.conf - source: salt://my_app/nginx.my_app.conf - context: #    pillar,        bind: {{ my_app['gunicorn_bind'] }} dns_name: {{ my_app['dns_name'] }} - template: jinja - makedirs: True - watch_in: - service: nginx my_app.supervisor.conf: file.managed: - name: /etc/supervisor/conf.d/my_app.conf - source: salt://my_app/supervisor.my_app.conf - context: app_name: my_app bind: {{ my_app['gunicorn_bind'] }} gunicorn: {{ my_app['venv_dir'] }}/bin/gunicorn directory: {{ my_app['work_dir'] }} workers: {{ grains['num_cpus'] * 2 + 1 }} #     - template: jinja - makedirs: True my_app.supervisor: supervisord.running: - name: my_app - watch: - file: my_app.supervisor.conf - require: - pip: my_app.pip - pkg: supervisor


Hint : when compiling the require, watch, etc., keep in mind that the states will be checked in an arbitrary order. When writing this article, I made this kind of error, and the django and gunicorn packages tried to install themselves into the not yet created virtualenv.

salt / roots / salt / my_app / nginx.my_app.conf
 server { listen 80; server_name {{ dns_name }} _; location / { proxy_pass http://{{ bind }}; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } }


salt / roots / salt / my_app / supervisor.my_app.conf
 [program:{{ app_name }}] command={{ gunicorn }} {{ app_name }}.wsgi:application -b {{ bind }} -w {{ workers }} directory={{ directory }} user=nobody autostart=true autorestart=true redirect_stderr=true


Add the newly created state to salt / roots / salt / top.sls
 base: 'dev-my-app': - nginx - python - supervisor - my_app # <----


We are almost done. But the most important thing is missing - the code of our imaginary Django application. Let's create an empty Django test project as follows:
 vagrant provision


This process will take several minutes (most of the time django and gunicorn will be put into virtualenv).

When provisioning will work go to the guest machine:
 vagrant ssh


And inside it does the following:
 /home/vagrant/my_app_env/bin/django-admin.py startproject my_app /vagrant exit


And on the host machine again doing vagrant provision , and to check that everything works in the hosts file, register temporarily:
 dev.my-app.com 3.3.3.3


Go to dev.my-app.com in the browser, and if everything is fine, then we will see It worked!

Dev configuration is built. You can commit.

Now, if you want to give a friend a play with your project, he will only need to do git clone and vagrant up . Moreover, this imaginary friend will receive not only the source code of the project itself, but will also have an idea of ​​how it will be deployed.

Among other things, by default, our project is spinning independently running gunicorn + supervisor. But what if we want to remotely patch or we want to return our favorite autoreload of code changes? No problem:
 vagrant ssh supervisorctl stop my_app /home/vagrant/my_app_env/bin/python /vagrant/manage.py runserver


Now we can safely edit the code, and all changes will be picked up by the django server automatically.
And if our hosts are still temporarily fixed , then the same django server will respond to dev.my-app.com requests.

Prod configuration


So we got to the most important. We will assume that we will deploy on the prod-my-app.

Next, we consider the deployment option in a situation where we have a separate server for the salt-master (hereafter, just the master).

We copy on configs to the master, we add in /srv/salt/top.sls
  'prod-my-app': - nginx - python - supervisor - my_app


Or in our case, you can do this:
 base: '*-my-app': - nginx - python - supervisor - my_app


Next, do the same with the file /srv/pillar/top.sls
 base: '*-my-app': - my-app


In /srv/pillar/my_app.sls, we change the variables according to our layout map.

On the prod-my-app set salt-minion. We connect salt-minion to salt-master (how to do this, read here ).
Now on the wizard, you can start using configs:
 sudo salt prod-my-app state.highstate


What is left:

Delivery of configs to master

There are a lot of ways from rsync to git. It all depends on your domestic policy.

Delivery of project sources to prod-my-app

Here again, a bunch of options. Personally, I do this: with the help of salt, I support the git repository on a certain commit for which the hash is stored in pillar on the prod-my-app, and if it changes, salt starts the deployment scripts at the end of the work.

Honestly, this is not the best way, but it is the easiest. Ideally, create, for example, native packages, or raise a private pypi.

Links


www.saltstack.com
www.vagrantup.com
hynek.me/talks/python-deployments is a very useful article that contains a set of abstracts on the topic of python projects.

PS
Unfortunately, many things had to be missed, otherwise the article would have swelled to an indecently large size. I tried to omit either the obvious things or something that is easily found in the documentation. However, ask questions in the comments.

Source: https://habr.com/ru/post/208866/

More articles:


All Articles