Theory and practice of unattended upgrades in Ubuntu

Unattended upgrades is native to Debian / Ubuntu (and other GNU / Linux distributions based on them) automatic updates. By default, it is enabled on the system due to the unattended-upgrades package installed and the /etc/apt/apt.conf.d/50unattended-upgrades configuration file, and is configured to update packages only from the security repository , where, for example, critical fixes for the libssl package, which is released as a result of the next update of the CVE vulnerability base.


Note : here and hereafter, unattended upgrades are considered in the context of Ubuntu server editions, which is most likely applicable “as is” to other distributions, but there may be some peculiarities that remain outside the scope of the article.

So, what additional features do unattended upgrades (apart from the default security updates include ) and what problems can they cause?

The word “unattended” in the name of this mechanism is really important in its literal translation - “without supervision”. Why is that? It is enough to remember that the packages during installation generate the .dpkg-new files that .dpkg-new new config for the package (if the checksum of the config in the installed package differs from the checksum of the config in the system). This circumstance should be taken into account, because otherwise you can get a new version of the software that no longer works with the option you added to the config , and after installing the service / application package it will simply not start. Therefore, collecting or borrowing a package into your repository, remember that installing such a package will not update the configs itself, so, for example, if the version configs are not compatible (which is relevant in the case of a more significant update than a security fix) , it can be very unpleasant a situation when everything is already sleeping for a long time, and your package rolled out on a pile of servers and “Everything is broken, chef!”.
')
Speaking of the response times of unattended upgrades packages: checking for updates and installing them on Ubuntu / Debian systems is defined in /etc/cron.daily/apt . The file is launched from /etc/crontab , in which early morning is set by default (06:25).

Application for repository / ppa

Let's go to practice. We had the following problem: one package was installed on all servers, but not in its native version, but with certain modifications. What to do? Obvious options:

• “Let's curse one of the trainees for a few hours and let them roll out for themselves!” - this may be useful for the trainee, but only at the stage of learning how to work with the system and provided that he doesn’t know how to work apt. Then it really turns into a curse. In addition, the result will be more dependent on the human factor than we would like.
  
  
• “Roll out everywhere with Chef / Puppet / Ansible / ...!” Is a great thought, but not our case at that time and in that situation. For the sake of one package, the dragon would have to be defeated, i.e. set up multiple machines in the selected configuration management system.
  
  
• Since the article is devoted to unattended upgrades, it is easy to guess that it is this mechanism that offers a different way to solve the problem without spending a lot of man-hours ...

Indeed: the unattended upgrades configuration allows you to activate the automatic receipt of all updates of a package for a selected repository - for example, your own or PPA, which you have reason to trust very much . In order for all this automagia to work at the minimum level, it is enough to create a config of /etc/apt/apt.conf.d/51unattended-upgrades-custom on the target machine, which will contain only three lines:

 Unattended-Upgrade::Allowed-Origins { "Origin:Suite"; }; 

If you have your own repository, then the words Origin and Suite should at least cause associations from the category "Somewhere I have already seen it ...". I *_InRelease you that such parameters can be seen at the repository on the machine itself, where the package should be updated, in the *_InRelease file. Illustration for well-known ppa:nginx/stable :

 $ head -10 /var/lib/apt/lists/ppa.launchpad.net_nginx_stable_ubuntu_dists_trusty_InRelease -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Origin: LP-PPA-nginx-stable Label: NGINX Stable Suite: trusty Version: 14.04 Codename: trusty Date: Sat, 11 Feb 2017 21:55:33 UTC Architectures: amd64 arm64 armhf i386 powerpc ppc64el 

We see two parameters:

• Origin - the “origin” of the repository, which may indicate the name of the maintainer or the repository itself;
• Suite - distribution branch; for example, stable, testing for Debian or trusty, xenial for Ubuntu.

Note : For more detailed documentation on the Release / InRelease files and the parameters used in them, see wiki.debian.org .

Thus, if you have your own package repository, you need to add these parameters. As a result, for this PPA example, the nginx/stable configuration that allows unattended upgrades for all packages from it will look like this:

 Unattended-Upgrade::Allowed-Origins { "LP-PPA-nginx-stable:trusty"; }; 

It is logical to automate these settings (by the method that is closest to you and / or is already used), creating all the necessary configuration files when deploying new OS installations. And you can check how the next launch of unattended upgrades behaves as follows:

 $ unattended-upgrade -v --dry-run 

The flags here are simple: -v - to be more verbose, and --dry-run - not to apply changes. During the trial run, we will immediately see that this solution may have reverse sides :

• If package installation requires interactivity, i.e. user intervention (especially important if you are doing a large-scale upgrade of the system, since you have not done it for a long time) - unattended upgrades simply will not do anything.
  
  
• An example with the automatic update of nginx from PPA should not be checked in real-life production, unless you want to be seen as a “dirty Harry” package update. Once again, remember the name of the utility and remember that updating is only possible for something that really does not require any supervision . For example, libraries (although even here there are pitfalls, if we recall at least the recent Ubuntu dances with libc and a broken DNS resolver ) and software that does not require configuration and runs on demand (atop, htop, and the like).

Another “reverse” example, which does not even relate to the addition of specific PPAs to unattended upgrades, but which we have often encountered in practice, is a security update for PostgreSQL. In the standard Ubuntu Server configuration (i.e., installing only security updates through unattended upgrades), automatic updating of critical vulnerabilities in this DBMS led to its restart at a time when not everyone expected it. If this behavior may be unexpected (undesirable) for your case - see the Package-Blacklist option below.

Additional features

In /etc/apt/apt.conf.d/50unattended-upgrades you can see that they can still do unattended upgrades. There are not many settings, but some of them are useful:

• Package-Blacklist - a list of packages that are not allowed to be updated in this way. Here in the example we are immediately offered to do this for libc, and another example is described above - with PostgreSQL. But remember that on the other side of the scale: deferring critical corrections, you risk security .
• AutoFixInterruptedDpkg - if the last installation / upgrade process failed to complete for any reason, you probably had to correct the situation manually. This option does the same, i.e. calls dpkg --force-confold --configure -a . Please note that here the option is --force-confold - it means that old versions of configs will be saved if conflicts arise.
• MinimalSteps - perform updates with the minimum possible parts. Allows you to interrupt the upgrade by sending SIGUSR1 to the unattended-upgrade process.
• InstallOnShutdown - install updates before shutting down the computer. Personally, it seems to me a bad idea, because I would not want to get a corpse after a planned reboot of the server.
• Mail and MailOnlyOnError - to whom to send emails about updates and / or problems with them. Emails are sent via standard sendmail MTA (environment variable SENDMAIL_BINARY ). Unfortunately, only letters, and it is impossible to perform curl to any API here.
• Automatic-Reboot - restart automatically after the installation is completed, if there is a /var/run/reboot-required file. The file itself appears, for example, after installing the Linux kernel package, when the /etc/kernel/postinst.d/update-notifier rule is triggered. In general, another option from the "dirty Harry" set.
• Automatic-Reboot-Time - if you want to do your dark things at night, while no one sees ... sets a specific time for automatic reloading.
• Acquire::http::Dl-Limit is already from the general set of parameters apt. Limits the speed of downloading updates, so as not to clog the channel.

findings

Unattended upgrades is an automated update installation tool built into the distributions based on Debian and Ubuntu. It is usually used to install security updates from the corresponding repository, but it is easy to expand the application to any other repositories. The tool will be useful for supporting easy-to-install and update programs and scripts assembled into packages - all you need to implement is the repository itself and a few lines in the config on the updated machine.

But remember that the simplicity of the tool does not mean that you do not hit them on your fingers: do not set up automatic updates of complex or critical services if you are not 100% sure that this action is safe and this will not affect the length of sleep for you and your colleagues.