Installing Linux without .ISO and virtualization

Creating a file system, installing and cloning Debian and Ubuntu using radish scripts.

1. Purpose and capabilities of radish scripts

Usually, the installation of the Linux system is done by running any installer program supplied by the distribution developers. This is done either directly on the computer on which the installation is being performed, or in some isolated environment, for example, using virtualization. The procedures described below follow these principles only in the most minimally necessary form. When creating a system image, any installers are reduced to the debootstrap minimal system generator and the apt package manager interface (both on top of the dpkg package manager), and chroot is used instead of virtualization.

The installation of a disk image on a device is performed by a minimal script that does not use the installer or package manager at all, nevertheless creating a configuration completely controlled by the package manager in the disk image - all installed components, including the bootloader and the kernel, can be updated and replaced by the same manager actions packages that would be used on a system installed by the standard installer of the distribution kit.

The scripts are located on the Github server and are available here .

1.1. Restrictions

Scripts have been developed for Debian, Ubuntu, and other distributions based on the Debian package manager. In principle, there are no fundamental restrictions that would prevent the transfer of the same procedures to distributions based on rpm, the Red Hat package manager, or other less common mechanisms. However, the author first needed the support of Debian and Ubuntu, and therefore the development was carried out precisely in this direction.

Another existing limitation relates to the use of the MBR disk loading and partitioning mechanism, and not the more modern GPT. This limits the size of the boot device to 2 terabytes and requires appropriate BIOS configuration on x86 devices. There are no fundamental restrictions that prevent the support of GPT / UEFI, but the author set himself the goal of creating a simple configuration that is not tied to anything outside the boot disk. With all its flaws and limitations, on the x86, MBR architecture has one useful feature - if the BIOS selected the disk with the MBR as a boot device, the entire subsequent boot process is exclusively controlled by the chain of boot loaders on this device and receiving configuration from files on the same device. Apparently, in the future, it makes sense to add support for GPT and UEFI - good, problems with non-standard behavior of UEFI have decreased a lot on the current generation of hardware.

1.2. The procedure for assembling a disk image, its modification and installation on the boot device

The installation procedure consists of two stages - the creation of a disk image and its installation on the device. In addition, each device on which such a disk image is installed becomes bootable on computers with x86 architecture (32-bit or 64-bit, depending on the original build). The procedure for installing a file system image involves creating unique identifiers (UUIDs) of file systems, which helps to eliminate confusion during the boot and update system, which can occur if several devices with identical partitions are connected to the same computer at the time of loading.

The root ( / ) and boot ( /boot ) file systems are identified in the GRUB configuration and the /etc/fstab by their UUID to avoid dependence on the presence or identification of other devices (drives and partitions). GRUB always reads its own configuration (BIOS always installs a boot disk containing the first GRUB stage in the MBR, the “first hard disk” when accessed through its functions), the GRUB configuration contains the UUID of the root file system sent to the kernel via the command line. The boot process uses this identifier to determine the device being mounted as / , and then the file /etc/fstab is read from the same device, from which, also by UUID, the file system is defined, which is mounted as / boot if required (for example, when updating the kernel or bootloader). Also, the root filesystem UUID in /etc/fstab is guaranteed to match the UUID of the file system mounted as / (and containing this file itself). If several connected devices contained the same file system UUIDs, it would be quite possible that after booting from one of the devices, the file systems were mounted from other devices with the same UUID. If the UUIDs are unique, and each physical device in the GRUB configuration and /etc/fstab contains references to the UUIDs of its own partitions, this situation is impossible.

In general, disk images and devices themselves with radish file systems installed on them are designed for maximum compatibility with hardware and maintaining performance in a wide range of possible configurations, provided that the hardware configuration and BIOS does not prevent traditional (via MBR) booting from these devices.

If desired, the user can unpack the disk image, add files and packages, run unpacked disk images under the chroot, and build a new disk image after these changes. The user can also install a disk image on the device, boot from it, use it in the usual way, and then use a simple procedure to create a disk image that creates copies of the device in the state in which the user left it at the time of cloning. At the same time, the installation procedure remains unchanged if the user has not changed any fundamental mechanisms (for example, the boot method or hardware architecture).

1.3. Using a disk image to restore from backups and transfer the running configuration to the new hardware

The latter solves the problem of restoring bootable devices from backups - it is enough to create a disk image from a copy of the device created in this way and start the procedure for installing an image on the device on the device, which should become bootable. Both the file set and the download mechanism after this procedure will be functionally a copy of the device from which the backup was made. Compatibility with various sets of equipment allows you to completely replace equipment when a server fails or when replacing it with new equipment, and to get a workable, bootable system without any manual configuration. In this case, one requirement must be met: in at least one of the workable configurations, the system must be bootable from one disk, recognized by the saved configuration of the operating system.

That is, it may turn out that a disk image from a server booted from a RAID array will not be able to boot to a completely different configuration, which requires additional configuration of hardware and software supported arrays, partitions and logical volumes. To do this, it makes sense to have at least one device with a “simple” configuration containing MBR, partitions and file systems, and maintain a copy of the boot system on it even if it is not a boot device during normal server operation. Then, after changing the hardware, you can start the recovered copy of this device first, and only then, manually or automatically, restore the rest of the configuration.

1.4. Creating a disk image for processor architectures that are different from the architecture of the computer on which the assembly is made

At the moment, radish cannot completely create a file system for a “foreign” architecture, however, it can be used in parts to build the source directory tree to run on the device with the required architecture, and then complete the build procedure on this device (real or emulated) before receiving fully functional file system.

Despite the fact that this is the least developed part of radish, it is quite suitable for inclusion in scripts for creating firmware of various devices from scratch - you only need to add the creation of a minimum file system for running radish (for example, compiling a system based on busybox) bootloader configurations, and procedures for copying files created by radish to a device (for example, switching to the boot file system via ssh / scp, etc.).

2. The principle of operation

radish is implemented as radish-build and radish-install, bash shell scripts that use a small set of utilities that make up the minimal Linux configuration, plus several utilities that are specific to it. In the radish itself there is a list of these utilities. To build the file system are used:

radish checks for their presence at startup, and fails if any of them are missing. This suggests that the utilities themselves should be present even if the shell itself implements them as built-in commands - such an assumption avoids errors when changing versions and implementations of the shell, and corresponds to the typical configuration of modern Linux distributions, even the most minimal ones based on Busybox.

Five of these files perform functions specific to creating file systems with a Debian distribution:

1. debootstrap . This is the main script that configures access to the repository and installs basic system packages. It is well maintained and updated with the release of new versions of various systems compatible with Debian. It is also distributed in the standard repositories of many distributions that are not compatible with Debian, and can be run under them, creating a directory tree containing a workable system compatible with Debian. The only requirement for its operation is the availability of hardware, the kernel and the minimum configuration of Linux for the respective architecture, and access to the repository.
2. mkfs.ext4 and fsck.ext4 . These utilities create and verify the EXT4 file system, typically used for Linux boot / root devices. radish works completely under Linux, so EXT4, traditionally supported by all Linux distributions and configurations, can be used for all operations without any format translation or copying.
3. resize2fs . This utility resizes the collected EXT2, EXT3 or EXT4 file system. During file system assembly, initially the size of the file system image is selected with a margin. At the end of the build, the file system is compressed to a minimum size, and in this form is transferred to the partclone format. When installed on a device, this part of the file system is first installed using the minimum size of the file system, and then the file system is expanded to the size of the partition on the device. This avoids problems when installing on devices of various sizes - the image always corresponds to the minimum supported size determined by the total size of the installed files (plus incomplete blocks, directories and metadata), but after installation the entire device is used.
4. partclone.ext4 . A utility for copying a file system image in a format that allows you to save only blocks occupied by data. Since copying occurs after the file system is compressed, dd could be used instead of this utility, however, dd cannot determine if the copied disk image is complete, and partclone will give an error if for some reason the file is truncated.

For installation are used:

There are some other utilities in this list that are also specific to the operations performed by this script:

1. blockdev . This utility allows you to request kernel operations on a block device, in this case re-reading the partition table.
2. dialog . A utility that implements a simple user interface on a text screen. Used to select a device and enter text - the name of the machine, passwords.
3. parted . A utility that creates and edits a disk partition table. In this case, it is used only in the mode for the format with MBR, although it also supports GPT.
4. tune2fs . A utility that edits file system parameters. Used to create a unique identifier (UUID) for the created file systems. After cloning, the original identifier is preserved, which must be replaced with a new one, in order to exclude the possibility of coincidence of identifiers of several file systems that are simultaneously available on the same computer.
5. blkid A utility that determines the list of block devices in the system and finds their identifiers (tags and UUIDs).

2.1. Creating a file system image

The basis of the radish operation is the creation of a directory tree and files on the file system located on the system image file, which is mounted on the local directory via the block device to which this file is mapped ( /dev/loop n ). That is, the radish ‑ build script creates a file, formats it as a file system, creates a temporary directory, and mounts this file system under it. In this directory, the minimal system is first installed via debootstrap, and then it is supplemented to the minimum working server or embedded system configuration (the set of packages can be changed by the user, but this requires editing the script). In such a workable form, the file system is transferred to the partclone format and compressed with bzip2. After that, the file system is unmounted, the original image file and directory are deleted, and the compressed partclone file remains, ready for installation with the radish-install script.

In developing radish, the challenge was to ensure that software could be installed from randomly selected Debian packages. Debian packages are developed for installation on a computer on which a fully functional system is already installed. The configuration stage, the final stage of the installation of each package, starts after the installation of all the packages that this package depends on, and can rely on their presence. This condition is always fulfilled when installing on a loaded and working system, however, it is impossible to guarantee this behavior during the operation of radish, because the directory under which the image of the file system is mounted is not the full equivalent of a working Debian system. Therefore, additional measures were needed to temporarily create an environment that is fairly close to running Debian during the installation of packages, but “fit” to the mounted directory and after installation does not leave running processes due to which this directory cannot be unmounted. As it turned out, in most cases it is enough to properly install the packages:

1. Run all operations on the mounted file system image under chroot.
2. Mount the special file systems /proc and /sys ( /dev operational as it was created when debootstrap was started).
3. Rename /usr/sbin/invoke-rc.d (from SysV init) and /sbin/initctl (from upstart) if these scripts are installed on the system before starting the installation and rename these files back after the installation is completed. Systems using systemd do not require any changes, because about such installation procedure systemd does not find its own process and its interfaces under chroot - they are started only when the system is “real” loaded.

To fully guarantee that after installing the packages, no processes will be running, the script also includes a procedure for searching for processes running under chroot (the termprocesses() function). Files of the found processes are renamed (to prevent automatic restart), the processes are terminated first with a TERM signal and renamed back. If they are not completed after 5 seconds (it is assumed that the processes may spend some time to restore the state, delete files, etc.,), the procedure is repeated with the KILL signal, which leads to immediate completion.

After completing this installation procedure, the file system is unmounted, compressed by the resize2fs utility, and converted to the partclone format compressed with bzip2. After creating a file in this format, the source file with the image of the file system and the temporary directory are deleted.

As a command-line argument, radish-build takes the name of the distribution version, for example, “artful” for Ubuntu 17.10 or “stretch” for Debian 9. The script creates the root-image.bin file.

2.2. Installing the system from the image on the device

The radish-install script installs the system on the device and configures this device for booting. This script:

1. Interactively requests the device from the list of suitable for installation. A list of devices suitable for installation is created from /sys/class/block/sd* exception of devices containing mounted partitions that are outside the expected size range and also considered as fixed by the operating system. These criteria (which are in radish ‑ install after scanning devices using the above mask), will probably need to be changed in many cases.
2. Creates two partitions: bootloader and root filesystem.
3. Formats the file system loader.
4. Copies the prepared image to the root file system, creates new UUIDs for file systems.
5. Mounts file systems, transfers the / boot directory from the installed root to the loader file system.
6. Creates the configuration files /etc/fstab , /etc/inittab or /etc/init/ttyS0.conf , /etc/default/grub .
7. Installs the GRUB boot loader onto the device using the boot files already installed from under the chroot. Only at this stage of the installation is the system boot record in the MBR and the space before the beginning of partitions, that is, in those parts of the device that are not covered by file systems. This also installs the bootloader files on the file system located in the bootloader section. These procedures are completely equivalent to installing GRUB on subsequent updates, because they are produced by the same programs that come with the GRUB package. The only difference is that in this case they are started from under the chroot.
8. Unmounts filesystems.
9. Extends the root file system to the size of the device partition and mounts it again.
10. Queries the machine name and passwords for the predefined root and user users, creates the /etc/hostname file, and edits user passwords using the chpasswd utility.
11. Unmounts the resulting root filesystem and writes all buffers to devices.

After all these actions, the device can be used to boot a working Linux system. The file system identifiers correspond to the GRUB and /etc/fstab configurations, that is, regardless of the device numbers for Linux or BIOS, the boot procedure correctly recognizes its own file systems. After booting, the system can be updated without any changes by the Debian package manager and use all utilities that support synchronization and updates from remote repositories.

If you wish (and if you have enough free space) you can copy the radish itself and a compressed disk image from which the device was installed to any directory on this device. In this case, you need to additionally install the utilities used by radish scripts. Thus, the resulting system can then create copies of its original state — this can be convenient for distributing a “sample” file system with installed software to users who can then create their own copies.

2.3. Modifying the created file system image

After the file system image file has been created using the radish-build script, it may be necessary to manually install and configure software or any data. The image file itself is not editable, but there are at least two possibilities to create such an edited image:

2.3.1. Installing software on a physical device followed by preparation for cloning

The file system image file is used to install the full bootable system on the device, which after this installation is used to boot. After downloading, the user installs and configures (and, if necessary, tests) the software in the same way as he would install it on a regular computer. Upgrading packages and even upgrading to new versions of the distribution can be done using the usual package manager mechanisms. This results in a file system containing the necessary configuration.

Having achieved the desired configuration, the user reboots the computer and connects the device with the modified file system to the same or another computer. The device becomes available as what will be referred to as $TDEV . , /dev/sdb , :

${TDEV}2 , . boot :

 mount ${TDEV}2 /mnt mount ${TDEV}1 /mnt/tmp cp -a /mnt/tmp/* /mnt/boot/ umount /mnt/tmp umount /mnt 

:

 fsck.ext4 -f ${TDEV}2 resize2fs -M ${TDEV}2 

( – ).

partclone, bzip2:

 partclone.ext4 -c -s ${TDEV}2 -o - | bzip2 -9 -c > root-image.bin 

:

 fsck.ext4 -f ${TDEV}2 resize2fs ${TDEV}2 mount ${TDEV}2 /mnt rm -rf /mnt/boot/* umount /mnt 

, root‑image.bin , .

2.3.2.

, , /dev/loop n , partclone, bzip2. radish-unpack-image radish-pack-image. .

/proc , /sys /dev . radish-unpack-image (, radish‑image‑ bbbbbbbbbb radish‑mount‑ aaaaaaaaaa ):

 ./radish-unpack-image root-image.bin 

( , )

 mount --bind /proc radish-mount-aaaaaaaaaa/proc mount --bind /sys radish-mount-aaaaaaaaaa/sys mount --bind /dev radish-mount-aaaaaaaaaa/dev mount --bind /dev/pts radish-mount-aaaaaaaaaa/dev/pts mount --bind /dev/shm radish-mount-aaaaaaaaaa/dev/shm chroot radish-mount-aaaaaaaaaa /bin/bash 

( chroot)

( - chroot)

 umount radish-mount-aaaaaaaaaa/dev/shm umount radish-mount-aaaaaaaaaa/dev/pts umount radish-mount-aaaaaaaaaa/dev umount radish-mount-aaaaaaaaaa/sys umount radish-mount-aaaaaaaaaa/proc 

, :

 ./radish-pack-image radish-mount-aaaaaaaaaa 

root‑image.bin, . , :

 umount radish‑mount‑aaaaaaaaaa rmdir radish‑mount‑aaaaaaaaaa rm radish‑image‑bbbbbbbbbb 

, , chroot , . , radish‑pack‑image , .

, schroot . , , ,

 cp -a radish‑mount‑aaaaaaaaaa \ /var/lib/chroot‑environments/debian‑system‑1 

/etc/schroot.conf, :

 [debian-system-1] type=directory directory=/var/lib/chroot‑environments/debian‑system‑1 users=user root-users=root 

 schroot -c debian‑system‑1 

, , - , /var/lib/chroot‑environments/debian‑system‑1

- ( /home/user/chroot‑environments/debian‑system‑1/home/user/ /home/user/ ), , $HOME/.config , $HOME/.local , . ., , . , , , , , . .

3. ,

radish , . . , radish-install /etc/default/grub Linux, ( "nomodeset" ) ( "console=ttyS0,115200n8" ). , . ( /etc/inittab /etc/init/ttyS0.conf ) , /etc/default/grub :

 # Uncomment to disable graphical terminal (grub-pc only) #GRUB_TERMINAL=console GRUB_TERMINAL=serial GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1" 

, , GRUB_TERMINAL "console" . , .

«» , /lib/udev/rules.d/75‑persistent‑net‑generator.rules , , radish‑build , . , , , , , , , , .

SSH , SSH , . , . - , , SSH, .

. , radish-install - . ( "x" /etc/passwd , - /etc/shadow ), /home/user/.ssh/authorized_keys , ).

- . .