Thin diskless client based on Ubuntu that does not require mounting the file system over the network

Image from getwallpapers.com

Story

Back in 2013, thin clients based on DisklessUbuntu were used in one bank. There were some problems with them, in my opinion, mounting the root filesystem over the network in large branches with a weak network did not work very well. Then my good friend @deadroot made the first version of the thin client, which was loaded entirely into memory, without requiring something to be mounted on the network for work.

Then I actively finished this client, there were made a lot of useful pieces that are specific to our usage scenario. Then the bank closed (the license was revoked), the remnants of the client’s source code moved to my githab: thunclient . A couple of times I finished it slightly to order.

Recently, I got around to making this pile of scary, unreliable scripts a fairly convenient solution to use:

• Vagrant raises virtualku, which can be configured as a normal workstation.
• One script from it is going to ready for downloading files over the network, the excess is cut.
• Vagrant brings up a virtual PXE server and a network client to check the resulting build.

What can

• Entirely loaded into memory, does not require to install the root file system over the network.
• Built on the basis of Ubuntu, almost any software can be installed from its rich repositories, and connect third-party if something is not enough. It is especially nice that security updates arrive in Ubuntu fairly quickly.
• Able to mount overlay root overlay additional. You can add some software only for some workstations without collecting a new image.
• Able zram - memory compression, useful for old customers with a small amount of RAM. Although for the new usually does not hurt.
• A lightweight desktop (LXDE) with an RDP client is assembled out of the box, the addresses and parameters of the RDP servers are simply transferred from the PXE server via boot parameters.
• You can change one parameter in the config and a minimal console system will be assembled without too much software - the basis for some of your non-standard builds.
• If the download fails due to a server or network problem, it will display an error message for a short time and try to load again. Conveniently, when problems are fixed, the workstations will rise without any extra movements.

In the bank for remote connection to the user's thin client, VNC was used ( x11vnc to connect to the already running Xorg session). Not everyone needs this (usually there is enough connectivity to the RDP session on the terminal server), and everything is very individual in terms of convenience / security requirements. Therefore, I did not publish this part.

Analogs

• Thinstation

If Thinstation is completely satisfied, then it is better to use it, this is an older and more mature project. Plus, it is one and a half times smaller in size, after all, this is a specially crafted assembly for a minimum volume, and not a slightly doped regular Ubuntu.

But the software versions in it are quite ancient and there is little of it. If you need something extra, in addition to RDP / Citrix / ... clients, you will need to collect it by hand, and so with each update.

• Ltsp

kvaps pointed out in the comments that LTSP can copy the squashfs image into memory and work without mounting the FS over the network: this is configured by the LTSP_NBD_TO_RAM variable. Use chroot for configuration, which may be less convenient, especially for setting up the graphical environment and applications. Also a good mature project, can be considered as an alternative.

Vagrant vs chroot

Past versions used chroot, like most similar projects, the same Thinstation for example. This is not difficult, but still a separate program running in chroot does not correspond to what is happening on a real machine: there is no interaction with system init, with other programs and services. Plus Vagrant made it possible to make the process of creating the client as simple as possible: the virtual machine is configured as a regular machine.

Of course, using Vagrant brings some difficulties.

The machine must be running the virtualbox-guest-utils for shared folders to work. In addition, you need a boot manager ( grub ), mandatory for a machine with a disk and useless for a client that is downloaded over the network. I solved these problems, excluding all files of these packages from the assembly. Therefore, they do not affect the size of the resulting image.

In addition, for Vagrant, ssh is required to run on a machine, allowing the user to generate a generated key. I exclude from the assembly the home directory of the vagrant user used for configuration, along with its ssh keys. Keys for the ubuntu user to be used when running can be put in his home directory.

Well, for Vagrant to work, it generates settings for network interfaces that will be erroneous for a real machine. I had to replace the interfaces file at the time of the assembly, and write a script that generates a config on a real machine to configure all available interfaces via DHCP.

Provisioning is done with Ansible. This is a very handy tool for configuring all kinds of software and hardware. But I don’t want to include Ansible in the final image and the required second python with the necessary libraries: useless ballast. I don’t want to put Ansible on a machine where a virtual environment is buried: it will complicate the work.

Vagrant allows you to do the trick: put Ansible on one machine (test PXE server), and from it do the deployment of other machines, within the same playbook. To do this, the machines must have a static IP in order to register it in ansible inventory. Well, the problem with the configuration of the interfaces we solved in the last paragraph.

Naughty zucchini

Squashfs is a compressed read-only file system. Underlies most existing Linux LiveCDs. That is, it allows you to create a fairly compact image of the system that fits in the RAM of a thin client.

From the final image, you need to cut a lot of things: /tmp , /run , /proc , /sys , /usr/share/doc and so on.

The mksquashfs utility supports as many as 3 types of lists for excluding files: by full path, by masks and by regular expressions. It would seem that everything is fine. But the last two options do not support paths starting with / . I did not manage to exclude all the files inside a certain folder structure, not excluding the last folder.

I quickly got tired of struggling with it, I just found all the files and folders that I need to exclude with find , and stuffed it into one big file with exceptions in the full path. Crutches.jpg. But it works. The only artifact of this approach in the final image is the lonely folder /proc/NNN , corresponding to the number of the mksquashfs process, which was not yet in the list of exceptions. Procfs is still mounted above.

Initrd magic

In order not to pull into the kernel all the necessary drivers and mounting logic of the root file system, Linux uses an initial ramdisk. Previously, the initrd format was used, in which this disk was a real image of the file system. In the 2.6 kernel a new format appeared - initramfs, which is a cpio-archive extracted into tmpfs. Both initrd and initramfs can be compressed to save load time. Many utility names and file names still mention the initrd, although it is no longer used.

In Debian / Ubuntu, the initramfs-tools package is used to create initramfs. It gives the following options for customization:

• hooks are special format scripts that allow you to add kernel modules and executable files with all the libraries they need to the image
• scripts in the directories init-bottom , init-premount , init-top , local-block , local-bottom , local-premount , local-top , executed at the appropriate moment of loading. See man initramfs-tools (8)
• the most interesting thing is to add your own boot scripts that are responsible for mounting the root filesystem. They must define the shell mountroot() function, which will be used by the main /init script. The composition already has a local for mounting the root on the local disk and nfs for mounting the root over the network. The script used is selected by the boot parameter boot .

So, to mount the root file system in some very tricky way, you need to create your own boot script, define the mountroot() function in it, transfer the name of this script in the boot parameter to boot and remember to write hooks that pull all the necessary programs and modules into the initramfs kernels.

Fight for overlays

OverlayFS is used to create a single root file system from several. The first versions used AUFS (it is used by most Linux LiveCDs). But it was not accepted into the core, and now everyone is recommended to switch to OverlayFS.

After the real root FS is mounted in the directory inside the initramfs, the run_init program from the klibc-utils will be launched. She will check that the root file system is mounted inside the initramfs, clean the initramfs (why waste memory?) And move the root file mount point to / , and start the system init. Details This program is compiled as a separate executable file, because the script that uses any external utilities will break after cleaning up the initramfs.

If the root file system is assembled from several overlays mounted inside the initramfs, when run_init these mount points disappear and it breaks. This problem can be solved by moving the overlay mount points inside the root file system, where they will not be lost. Recursion :) It is done like this: mount --move olddir newdir .

AppArmor had to be disabled: its profiles are designed for direct mounting of the root file system from one device. When using OverlayFS, it sees that /sbin/dhclient is actually /AUFS/root/sbin/dhclient , and the profile is broken. The only option to use it is to rewrite all profiles for all applications, and update if necessary.

Where do I need to write

Under the idea, Linux can work quietly when all filesystems are mounted only for reading. But many programs are counting on the ability to write to disk, you have to mount tmpfs there:

• /tmp , /var/tmp - clear, needed by very many
• /var/log - write logs
• /run - almost all services will not start without it
• /media - mount mounted media
• /var/lib/system - used by many programs from systemd, in particular systemd-timesyncd
• /var/lib/dhclient - here dhclient records information about leases
• /etc/apparmor.d/cache - if you still beat AppArmor, then it will have to write files in /etc IMHO disgusting, for such things is /var .

Total

If you want to build a Ubuntu build that is downloadable from the network and works only from memory, here is a ready-made convenient constructor: thinclient . If you need help - write in the LAN or in a ticket on a githaba, I will.

PS English version of the article in my blog .