SaltStack: managing an arbitrary number of configuration files

What is interesting here?

The article will help to understand how to manage an arbitrary number of configuration files of a certain service in SaltStack.

SaltStack (underfloor): what is it and how to deal with it?

Many have already had experience with automated configuration management systems, which include Salt, and talk in detail about how to prepare it and what it may I do not, then you can read what it is, and then read how to quickly and easily prepare it.

SaltStack (lobby): a typical application for a simple example

So, suppose you have a task to automate the installation, the deployment of configuration files and the monitoring of their changes, and the reboot if necessary for such a popular service as nginx. To simplify the system, we assume that the servers being serviced are based on Debian Wheezy. (For everyone else, we read about the grains system, it will help determine on which system we use the state and change its behavior accordingly).
So:

nginx-repo-key-install: cmd.run: - name: 'apt-key adv --keyserver keys.gnupg.net --recv-keys ABF5BD827BD9BF62' - unless: 'apt-key list | grep -q 7BD9BF62' nginx-repo: pkgrepo.managed: - humanname: nginx-repo - name: deb http://nginx.org/packages/mainline/debian/ wheezy nginx - require_in: - pkg: nginx-pkg - required: - cmd: nginx-repo-key-install nginx-pkg: pkg.installed: - name: nginx - refresh: True - require: - pkgrepo: nginx-repo nginx-service: service.running: - name: nginx - full_restart: True - require: - pkg: nginx-pkg - watch: - file: nginx-config-vhost nginx-config-vhost: file.managed: - name: /etc/nginx/conf.d/vhost.conf - source: salt://__CONFIGS/vhost.conf - user: root - group: root - mode: 644 

')
It's all like the “book writes”, but let's take a closer look (very important as a background for the following calculations):

• nginx-repo-key-install: import the official repository key into the system
• nginx-repo: we are adding the official repository to apt
• nginx-pkg: we put a fresh package
• nginx-service: we take control of the service and make monitor the changes to the configuration file
• nginx-config-vhost: actually - the configuration file and the place to get it from

For inquisitive minds, with the naked eye, omissions are visible (for example, control over nginx.conf is not described), but all of them are made in order to reduce the information load not related to the problem area.

SaltStack (second floor): Many configuration files

In the first example, a service was shown in which there is one file where some kind of virtual server nginx is described. If it is changed on the Salt wizard (salt: //__CONFIGS/vhost.conf) and the state is restarted, the file will be updated on the machine and the nginx service will be restarted (full_restart: True) automatically.
Often there will be several virtual hosts on a machine with nginx. Consider how to organize their management.
Suppose there are vhost files {1,2,3,4,5} .conf. In this case, the lower part of the state will look like this:

 ...... nginx-service: service.running: - name: nginx - full_restart: True - require: - pkg: nginx-pkg - watch: {% for cnf in ['vhost1.conf', 'vhost2.conf', 'vhost3.conf', 'vhost4.conf', 'vhost5.conf'] %} - file: nginx-config-{{ cnf }} {% endfor %} {% for cnf in ['vhost1.conf', 'vhost2.conf', 'vhost3.conf', 'vhost4.conf', 'vhost5.conf'] %} nginx-config-{{ cnf }}: file.managed: - name: /etc/nginx/conf.d/{{ cnf }} - source: salt://__CONFIGS/{{ cnf }} - user: root - group: root - mode: 644 {% endfor %} 

Here begins a little Salt magic, namely: Jinja - a template engine for Python (on which SaltStack is written). Everything is obvious in the stack: we make many file objects with different names and tie all of them to the service so that when changes are made, it is restarted. For optimization, you can use a pillar system to store the names of files with configurations. But this is homework.

SaltStack (one floor upper): Many configuration files — but how many and which ones — are unknown.

So, the reason why this article was conceived is a fairly common situation for most virtual hosting systems: a large, previously unknown and constantly changing number of configuration files. How, then, to do with the deployment of infrastructure through Salt? There is a solution! Unfortunately, the obvious is not enough (even the developers had to ask for clarification), but it worked.
So:

 ...... nginx-service: service.running: - name: nginx - full_restart: True - require: - pkg: nginx-pkg - watch: {% for cnf in salt['cp.list_master'](prefix='__CONFIGS/') %} - file: nginx-config-{{ cnf }} {% endfor %} {% for cnf in salt['cp.list_master'](prefix='__CONFIGS/') %} {% set items = cnf.split('/') %} nginx-config-{{ cnf }}: file.managed: - name: /etc/nginx/conf.d/{{ items|last }} - source: salt://{{ cnf }} - user: root - group: root - mode: 644 {% endfor %} 

Actually now about the most interesting places:

 {% for cnf in salt['cp.list_master'](prefix='__CONFIGS/') %} 

call the list_master function from the cp module with the prefix '__CONFIGS /' (the folder name on the Salt wizard where all the configuration files for this service are located), which s lists the files in this directory and iterates over the list with a for loop.

 {% set items = cnf.split('/') %} 

The cp.list_master problem is that it gives the full path to the file (along with the prefix), and we only need the file name in order to specify it in the "- name:" directive. In this regard, we split the full path by slashes and use only the last element of the list - i.e. file name:

 - name: /etc/nginx/conf.d/{{ items|last }} 

As a result, we obtain a revised directory, configuration files included in the turnover that are available in this directory, and immediately we tied it to an automatic restart of the service in case of edits in the files.

Conclusion

Very briefly, but I hope it will help those Habr readers who are just starting to implement SaltStack and want to go a little further than book examples with simple package management. Successes!