Automatic unlocking of the root LUKS container after a hot reboot

Why do people encrypt the disks of their personal computers, and sometimes - the servers? It is clear that no one steals from the disk photos of their favorite pet cats! But that's bad luck: an encrypted disk requires a key phrase to be entered from the keyboard at each boot, and it is long and boring. Remove it to at least sometimes not have to recruit it. So much so that the meaning of the encryption is not lost.

Well, completely remove it will not work. You can instead make a key file on a flash drive, and it will also work. And without a flash drive (and without a second computer on the network) is it possible? If lucky with the BIOS, almost possible! Under the cut there will be a guide on how to configure disk encryption via LUKS with such properties:

1. The key phrase or key file is not stored anywhere in the open form (or in the form equivalent to the open one) when the computer is turned off.
2. When you first turn on the computer, you must enter a passphrase.
3. On subsequent reboots (before shutdown), the key phrase is not required.

The instructions were tested on CentOS 7.6, Ubuntu 19.04 and openSUSE Leap 15.1 in virtual machines and on real hardware (desktop, laptop and two servers). They should work on other distributions that have a workable version of Dracut.

And yes, in an amicable way, this would have to fall into the "abnormal system administration" hub, but there is no such hub.

I propose to use a separate slot of the LUKS container and store the key of it ... in RAM!

The BIOS on most motherboards does not clean the memory when rebooting, or you can configure it so that it does not clean (known exception: "Intel Corporation S1200SP / S1200SP, BIOS S1200SP.86B.03.01.0042.013020190050 01/30/2019"). Therefore, you can store a key there. When you turn off the power, the contents of the RAM itself are erased after some time, along with an unprotected copy of the key.

So let's go.

Step one: install the system on a disk encrypted using LUKS

At the same time, the disk partition (for example, /dev/sda1 ), mounted in /boot , should remain unencrypted, and another partition, which will contain everything else (for example, /dev/sda2 ), should be encrypted. The file system on the encrypted partition can be any, you can still use LVM, so that in one container were the root file system, the volume for the swap, and everything else except /boot . This corresponds to the default disk partitioning in CentOS 7 and in Debian when choosing the encryption option. SUSE does things differently (encrypts /boot ) and therefore requires manual disk partitioning.

As a result, you should get something like this:

 $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 10G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 9G 0 part └─luks-d07a97d7-3258-408c-a17c-e2fb56701c69 253:0 0 9G 0 crypt ├─centos_centos--encrypt2-root 253:1 0 8G 0 lvm / └─centos_centos--encrypt2-swap 253:2 0 1G 0 lvm [SWAP] 

In the case of using UEFI there will be another EFI System Partition section.

For Debian and Ubuntu users: the initramfs-tools package must be replaced with a dracut .

 # apt install --no-install-recommends dracut 

The initramfs-tools implemented incorrect in our case the logic applied to the encrypted sections with the key file. Such sections are either completely ignored, or the contents of the key file are copied to the initramfs (that is, to the disk) in the open form, which we do not need.

Step two: create a key file that will be used to automatically unlock the disk after a hot reboot

128 random bits are enough for us, i.e. 16 bytes The file will be stored on an encrypted disk, so no one who knows the encryption key and does not have root access to the loaded system will not read it.

 # touch -m 0600 /root/key # head -c16 /dev/urandom > /root/key 

In a key file, truly random bits are enough for the slow PBKDF algorithm, which makes a hard-to-choose encryption key out of a potentially weak key phrase, not really needed. Therefore, when adding a key, you can reduce the number of iterations:

 # cryptsetup luksAddKey --key-slot=6 --iter-time=1 /dev/sda2 /root/key Enter any existing passphrase: 

As you can see, the key file is stored on an encrypted disk and therefore does not pose any security threat if the computer is turned off.

Step three: allocate space in physical memory to store the key

Linux has at least three different drivers that allow access to physical memory at a known address. These are linux/drivers/char/mem.c , which is also responsible for the /dev/mem device, as well as the phram modules (emulates an MTD chip, gives the /dev/mtd0 ) and nd_e820 (used when working with NVDIMM, gives /dev/pmem0 ). They all have their unpleasant features:

• /dev/mem not writable when using Secure Boot if the distribution has applied the LOCKDOWN patch set from Matthew Garrett (and this patch set is required if the distribution is going to support Secure Boot with a Microsoft-signed bootloader);
• phram not available on CentOS and Fedora - the maintainer simply did not phram on the appropriate option when building the kernel;
• nd_e820 requires you to reserve at least 128 megabytes of memory - this is how NVDIMM works. But this is the only option that runs on CentOS with Secure Boot.

Since there is no ideal option, then all three are considered.

When using any of the methods, utmost care is needed in order not to accidentally touch devices other than the required range. This is especially true of computers that already have MTD chips or NVDIMM modules. Namely, /dev/mtd0 or /dev/pmem0 may not be the device that corresponds to the memory section reserved for key storage. Also, the numbering of existing devices, on which the configuration files and scripts rely, may be lost. Accordingly, all services relying on existing devices /dev/mtd* and /dev/pmem* are recommended to be temporarily disabled.

Linux memmap physical memory is done by passing the memmap option to the memmap . We are interested in two types of this option:

• memmap=4K$0x10000000 reserves (i.e., marks as reserved so that the kernel itself does not use) 4 kilobytes of memory, starting with the physical address 0x10000000;
• memmap=128M!0x10000000 marks 128 megabytes of physical memory, starting at address 0x10000000, as NVDIMM (obviously fake, but we can do that too).

The version with $ is suitable for use with /dev/mem and phram , the option with ! - for nd_e820 . When using $ starting address of the reserved memory area must be a multiple of 0x1000 (i.e. 4 kilobytes), when used ! - multiples of 0x8000000 (i.e. 128 megabytes).

Important: the dollar sign ( $ ) in the GRUB configuration files is a special character and must be escaped. And double: once - when generating grub.cfg from /etc/default/grub , a second time - when interpreting the resulting configuration file at boot time. Those. In /etc/default/grub following line should appear:

 GRUB_CMDLINE_LINUX="memmap=4K\\\$0x10000000 ...  ..." 

Without double escaping the $ sign, the system will simply not load, as it will think that it has only 4 kilobytes of memory. With an exclamation mark, there are no such difficulties:

 GRUB_CMDLINE_LINUX="memmap=128M!0x10000000 ...  ..." 

A physical memory card (which is needed to find out which addresses to back up) is available to the root in the /proc/iomem pseudo /proc/iomem :

 # cat /proc/iomem ... 000f0000-000fffff : reserved 000f0000-000fffff : System ROM 00100000-7ffddfff : System RAM 2b000000-350fffff : Crash kernel 73a00000-7417c25e : Kernel code 7417c25f-747661ff : Kernel data 74945000-74c50fff : Kernel bss 7ffde000-7fffffff : reserved 80000000-febfffff : PCI Bus 0000:00 fd000000-fdffffff : 0000:00:02.0 ... 

RAM is marked as "System RAM", we just need to reserve one of its page for storing the key. Guess what part of the BIOS memory does not touch when rebooting, it will not work reliably in advance. Is that if there is another computer with the exact same BIOS version and the same memory configuration, on which this manual has already been passed. Therefore, in general, will have to act by trial and error. As a rule, the BIOS changes the data only at the beginning and at the end of each memory range upon reboot. Usually it is enough to retreat 128 megabytes ( 0x8000000 ) from the edges. For KVM virtual machines with 1 GB of memory or more, the suggested options ( memmap=4K$0x10000000 and memmap=128M!0x10000000 ) work.

When using the phram module, another kernel command line parameter is needed, which, in fact, indicates to the module which piece of physical memory to use — ours, reserved. The parameter is called phram.phram and contains three parts: the name (arbitrary up to 63 characters, will be visible in sysfs ), the starting address and the length. The starting address and length should be the same as in memmap , but the K and M suffixes are not supported.

 GRUB_CMDLINE_LINUX="memmap=4K\\\$0x10000000 phram.phram=savedkey,0x10000000,4096 ..." 

After editing /etc/default/grub you need to regenerate the actual configuration file that GRUB reads on boot. The correct command for this depends on the distribution.

 # grub2-mkconfig -o /boot/grub2/grub.cfg # CentOS (Legacy BIOS) # grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg # CentOS (UEFI) # update-grub # Debian, Ubuntu # update-bootloader --reinit # SUSE 

After updating the GRUB configuration, the computer should be restarted, but we will do this later when we update the initramfs.

Step four: configure LUKS to read the key from memory

Disk encryption settings are stored in the /etc/crypttab . Each line consists of four fields:

• a device that should turn out when unlocking,
• encrypted device
• where to get the key file ( none means entering the key phrase from the keyboard),
• optional field for options.

If the key file exists, but does not fit, then Dracut asks for a key phrase. What, in fact, will be required when you first boot.

An example of the /etc/crypttab from the newly installed distribution:

 # cat /etc/crypttab #   luks-d07....69 UUID=d07....69 none 

The key file in our case is a piece of physical memory. Those. /dev/mem , /dev/mtd0 or /dev/pmem0 , depending on the selected memory access technology. Options are needed to indicate which piece of the file is the key.

 # cat /etc/crypttab #   #   /dev/mem: luks-d07....69 UUID=d07....69 /dev/mem keyfile-offset=0x10000000,keyfile-size=16 #   phram: luks-d07....69 UUID=d07....69 /dev/mtd0 keyfile-size=16 #   nd_e820: luks-d07....69 UUID=d07....69 /dev/pmem0 keyfile-size=16 

That's just just because it will not work.

The point is how systemd determines when to unlock a device. Namely, it takes the device from the third column and waits for the corresponding device unit to become active. It seems logical: it makes no sense to try to unlock the LUKS container until a device with a key file appears. But the device unit is not the same as the device itself. Systemd defaults to device units only for kernel devices related to block device subsystems and network interfaces. The devices /dev/mem and /dev/mtd0 are character-based, so they are not monitored by default and will never be declared ready.

We'll have to tell systemd to monitor them by creating udev rules in the /etc/udev/rules.d/99-mem.rules file:

 # /dev/mem KERNEL=="mem", TAG+="systemd" # /dev/mtd* KERNEL=="mtd*", TAG+="systemd" #  /dev/pmem*       

Step Five: regenerate the initramfs

I remind you: the article discusses only distributions using Dracut. Including those where it is not used by default, but is available and workable.

You need to regenerate the initramfs in order to update the /etc/crypttab file there. And also to include additional kernel modules and udev rules. Otherwise, the device /dev/mtd0 or /dev/pmem0 will not be created. The configuration parameter Dracut force_drivers is responsible for including and loading additional kernel modules, and install_items for additional files. Create the file /etc/dracut.conf.d/mem.conf with the following contents (a space after the opening quotation mark is required, this is a separator):

 #   /dev/mem: install_items+=" /etc/udev/rules.d/99-mem.rules" #   phram: install_items+=" /etc/udev/rules.d/99-mem.rules" force_drivers+=" phram" #   nd_e820: force_drivers+=" nd_e820 nd_pmem" 

Actually initramfs regeneration:

 # dracut -f 

Debian and Ubuntu users have a rake for the maintainer: the resulting file is not named correctly. You need to rename it so that it will be called in the same way as specified in the GRUB configuration:

 # mv /boot/initramfs-5.0.0-19-generic.img /boot/initrd.img-5.0.0-19-generic 

When installing new kernels, the automatic creation of initramfs through Dracut is performed correctly, the bug only affects the manual launch of dracut -f .

Step Six: Restart Computer

A reboot is needed for changes in the GRUB and Dracut configuration to take effect.

 # reboot 

At this stage there is no key in the memory, so you will need to enter a key phrase.

After the reboot, you need to check whether the memory reservation worked correctly. At a minimum, in the /proc/iomem pseudo /proc/iomem required memory should be marked as "reserved" (when using /dev/mem or phram ) or as "Persistent Memory (legacy)".

Even when using phram or nd_e820 you need to make sure that the device /dev/mtd0 or /dev/pmem0 really refers to a previously reserved area of ​​memory, and not to something else.

 # cat /sys/class/mtd/mtd0/name #  : "savedkey" # cat /sys/block/pmem0/device/resource #     

If this is not the case, you need to find out which of the /dev/mtd* or /dev/pmem* "ours" devices, then fix / etc / crypttab, regenerate the initramfs and recheck the result after another reboot.

Step Seven: set up copying key file to memory

The key file will be copied to memory before rebooting. One way to run any command at the system shutdown stage is to write it in the ExecStop directive in the systemd-service. In order for systemd to understand that this is not a daemon and did not swear at the absence of the ExecStart directive, you need to specify the type of service as oneshot and also suggest that the service is considered to be running, even if no running process is associated with it. So here’s the /etc/systemd/system/savekey.service file. It is necessary to leave only one of the above options for the ExecStop directive.

 [Unit] Description=Saving LUKS key into RAM Documentation=https://habr.com/ru/post/457396/ [Service] Type=oneshot RemainAfterExit=true #   /dev/mem: ExecStop=/bin/sh -c 'dd if=/root/key of=/dev/mem bs=1 seek=$((0x10000000))' #   /dev/mtd0: ExecStop=/bin/dd if=/root/key of=/dev/mtd0 #   /dev/pmem0: ExecStop=/bin/dd if=/root/key of=/dev/pmem0 [Install] WantedBy=default.target 

The construct with /bin/sh needed because dd does not understand hexadecimal notation.

We activate the service, check:

 # systemctl enable savekey # systemctl start savekey # reboot 

When the next reboot, you do not have to enter a key phrase from the disk. And if it is necessary, it usually means that the address of the beginning of the reserved memory area is selected incorrectly. It's okay to fix and regenerate several files and restart the computer two times.

When using phram or nd_e820 only have to edit the GRUB configuration. When using /dev/mem starting address is also mentioned in /etc/crypttab (so you need to regenerate the initramfs) and in the systemd-service.

But that is not all.

Security questions

Any discussion of security issues is based on the threat model. Those. on the targets and means of the attacker. I am aware that some of the examples below are contrived.

Situations with physical access to a computer that is turned off are no different from those without configured key storage in memory. There are the same types of attacks aimed at obtaining a key phrase, including Evil Maid , and the same defenses . We don’t stop at them, because there is nothing new here.

More interesting are situations when the computer is turned on.

Situation 1 . The attacker does not have physical access to the computer, does not know the key phrase, but has root access via ssh. The goal is the key to decrypt the disk. For example, to access old sector-based backups of a disk image of a virtual machine.

Actually, the key on a saucer is in the file /root/key . The question is how does this relate to what was before the execution of this instruction. Answer: for luks1, the threat is not new. There is a command dmsetup table --target crypt --showkeys , which shows the master key, i.e. also data allowing access to old backups. For luks2, security degradation in this scenario does take place: dm-crypt keys are stored in a keychain at the kernel level, and it’s impossible to look at them from userspace.

Situation 2 . The attacker can use the keyboard and look at the screen, but is not ready to open the case. For example, I used a leaked IPMI password or intercepted a noVNC session in the cloud. The key phrase does not know, no other passwords also know. The goal is root access.

Please: reboot via Ctrl-Alt-Del , add kernel option init=/bin/sh via GRUB, ready. The key phrase was not needed because the key was successfully read from memory. In order to protect against this, it would be necessary to prohibit GRUB from loading something that is not in the menu. Unfortunately, this functionality is implemented in different distributions in different ways.

In CentOS, starting with version 7.2, there is the grub2-setpassword , which actually protects GRUB with a password. Other distributions may have their own utilities for the same task. If not, then you can directly edit the files in the /etc/grub.d directory and regenerate grub.cfg .

In the /etc/grub.d/10_linux file, change the CLASS variable, add the --unrestricted option to the end, if it was not there:

 CLASS="--class gnu-linux --class gnu --class os --unrestricted" 

In the /etc/grub.d/40_custom file, add lines that specify the username and password that are needed to edit the kernel command line:

 set superusers="root" password_pbkdf2 root grub.pbkdf2....... #    grub2-mkpasswd-pbkdf2 

Or, if this functionality needs to be disabled at all, here’s the line:

 set superusers="" 

Situation 3 . The attacker has access to the computer being turned on, allowing it to boot from untrusted media. It can be physical access without opening the case or access via IPMI. The goal is root access.

It can load your GRUB from a flash drive or CD-ROM and add init=/bin/sh to the parameters of your kernel, as in the previous example. Accordingly, it is necessary to prohibit the loading of media from any media in the BIOS. And also protect changing the BIOS settings with a password.

Situation 4 . The attacker has physical access to the computer, including the case can open. The goal is to find out the key or get root access.

In general, this is a losing situation in any case. The attack on the memory modules by cooling them ( Cold boot attack ) has not been canceled. Also theoretically (not tested) you can take advantage of the fact that modern SATA drives support hot reconnection. When you restart your computer, disconnect the disk, change grub.cfg for init=/bin/sh , connect it back, let the system reboot. It will turn out (if I understand correctly) root access.

Approximately the same thing can turn a dishonest employee of cloud hosting by making snapshot of a virtual machine with its subsequent modification.

Other matters

Keeping the key in memory when rebooting is a mockery. Use-after-free in its pure form. A cleaner solution is to use kexec and put the key into dynamically generated initramfs. It also protects against replacing kernel parameters . Yes, this is true if kexec works. Modern distributions have made kexec setup too complicated .

In data centers and even more so in the cloud, the power never disappears. So, the key phrase is no longer needed? Indeed, if you are so sure of it, you can remove it. A working server will turn out, the key from the disk of which no one knows and therefore will not issue, but the system on which it is possible to update the standard means.And if necessary - quickly destroy all data easily remembered command sudo poweroff.

If I haven’t looked in /root/key- but there is still an abracadabra not typed from the keyboard, which can also be changed by cron.

Why is all this necessary? There is an IPMI where I can enter the password from the disk . On older servers, IPMI only works through older versions of Java. I would not want to go there for every reboot.

Why is all this necessary? I can unlock the drive via SSH . Perfectly!It does not interfere. But what if you need to issue rights to a sudo rebootuser who does not deserve knowledge of the key phrase?

, . SSH , . , , .