Centralized update of Let's Encrypt certificates

Hello!

In this article I will describe how we solved the problem of centralized updating of certificates of Let's Encrypt and infrastructure management using ansible.

In our solution we will use:

• ansible
• rsync, rsyncd
• inotify, incron
• certbot
• nginx

I will offer two options for the architecture, in which our solution can be useful. In turn, you can offer your options in the comments.

Option 1: You have several frontend servers with public ip (for example 3) serving several domains. These domains can be added / deleted. In order not to monitor each of the frontend servers - it is more convenient to do it on one letsencrypt server 'e:

Option 2: You have only one server with public ip, and you need certificates on servers within the network:

Description of roles ansible

The repository with roles available on the link .

There are 4 roles in the repository:

• nginx-simple
  Installs nginx on all hosts and copies base configs. By itself, the role does not start from the playbook. It runs on meta dependencies from other roles.
• letsencrypt-server
  Configures rsyncd on the letsencrypt server host. In meta dependencies, it has the role nginx-simple . Accordingly, nginx will be installed first, and then the role of letsencrypt server will play.
• incron
  Installs the necessary packages for incron and copies the base config. The role also does not run directly as nginx-simple .
• front
  In meta dependencies it has incron and nginx-simple roles. After them, the front role copies the nginx configs needed for example.com, adds a task to cron to pick up new certificates from the letsencrypt server and the task to incron to check file changes and execute the nginx -s reload hook nginx -s reload

Let's go to practice:

Initial configuration

Initially we have:

• 1 server for centralized writing of SSL certificate ( letsencrypt server )
• 1 or more publicly accessible servers with nginx ( front )

Ubuntu 16.04 is installed on all servers.

Install and configure nginx

To begin with, we will install nginx on a common role of nginx-simple on all hosts and scatter nginx configs common for all hosts (nginx.conf, ssl parameters, certificate paths, etc).

For letsencrypt server in the template .../site-available/default.conf .well_known folder will be in /var/www/ :

 {% if letsencrypt_server %} location /.well-known { root /var/www/; } 

For the server / servers of the front group, since the .well_known folder .well_known used not only for obtaining certificates, but also for other software, we import into the config example.conf letsencrypt-proxy.conf and nginx will search the folder locally on the front server using the try_file try_file :

 {% if nginx_proxy_well_known %} try_files $uri $uri/ @letsencrypt; {% endif %} 

Configs for the domain will be filled depending on the variables from inventory. In the repository is the domain example.com

Also, depending on the letsencrypt_server variable, the nginx-simple role will install certbot on the letsencrypt server and add the cron task to update the certificates.

Getting a certificate

Since we solved this problem before the wildcard certificate from Let's Encrypt appeared, we will consider both options for obtaining a certificate.

On the letsencrypt server server we execute:

 certbot certonly --agree-tos -d example.ru --webroot -w /var/www/ 

If there are more than one domains, we add the following ones using the -d key.

To obtain a wildcard certificate, we will need to add records to the DNS TXT. Currently, this is the only way to obtain such a certificate:

 certbot certonly --agree-tos -d example.ru -d *.example.ru --preferred-challenges dns --manual --server https://acme-v02.api.letsencrypt.org/directory 

Certbot will write what TXT records you need to add.

Certificate renewal

We received the certificates, it remains to set up copying them to the front server / server. To do this, we will configure rsyncd on the letsencrypt read-only server for a limited list of ip addresses:

 hosts allow = {{ hosts_allow }} hosts deny = * list = true use chroot = no [cert] path = /etc/letsencrypt/live/ uid = root gid = root read only = true 

Every 5 minutes the cron task from the front servers will check if the certificates have been updated and take them. Since certificates are rotated, /etc/letsencrypt/live/{{ domain }} are located in the /etc/letsencrypt/live/{{ domain }} folder. Add the -L switch to pull out the original files:

 /usr/bin/rsync -zavL --chmod=D0750,F640 --delete rsync://{{ hostvars['letsencrypt-server'].ansible_eth0.ipv4.address }}/cert /etc/letsencrypt/live/ 

Hook for nginx

We set up nginx, received certificates, took them to the front server. It remains to determine that the files in the /etc/letsencrypt/live/{{ domain }} folder have changed and execute the nginx -s reload hook nginx -s reload

The linux inotify kernel subsystem and the incron daemon will help us in this. Read more about them here .

The incron role incron necessary packages, and from the front role template a task will be added to monitor the certificates and the necessary hook:

 /etc/letsencrypt/live/{{ domain }}/ IN_CREATE,IN_DELETE,IN_MODIFY,IN_MOVED_TO nginx -s reload 

Finally

We tried to describe in detail the whole process of installation and configuration, and since everything is described in ansible playbooks - the article was very compact. As they often like to say - "a little more than 100 lines of code." Questions, criticism and comments will be happy to answer in the comments.