The method of optimization of the tasks of creating and maintaining the same type of Xen VM

purpose

I, like many inhabitants of Habr, have a home “balcony” server, on which a multitude of services, both personal and public, revolves - from the archive of photos and Gitolite, to several websites. Once I was puzzled by the question of separating the personal from the public in order to restore order and enhance the security of the system. It was decided to transfer public services to separate virtual machines, which, even if they were hacked, the rest of the data will not be affected, and the VM can be easily restored from a backup.
At the same time, I do not like redundancy in any manifestations, and I wanted to start a virtual machine for each service with minimal resources such as my time and disk space. As an additional Wishlist, there was the ability to update the same type of software on all virtual machines at the same time, rather than separately.

Research options

First, I explored the possibility of using OpenVZ, but I didn’t have a relationship with him for several reasons. First, the need to use a special kernel, and my system is Gentoo with a hardened profile and a Grsecurity system, which I didn’t want to refuse. Secondly, OpenVZ caused antipathy in me after I tried to occupy all this memory inside the guest system (container), limited by the amount of maximum available memory, which eventually led to the “killing” of the OOM process (!) system, but not in the container. I was quite surprised by this behavior, since the free memory in the main system was in bulk, and I took OpenVZ out of harm's way.

Fortunately, everything you need for Xen is already included in the standard Linux kernel, and looking ahead, I will say that both Dom0 and DomU work successfully with my hardened kernel (with a slight change).

Implementation

We make an image of the reference virtual machine (AMI)

As I mentioned above, my main system is Gentoo Linux amd64 3.8.6-hardened SMP, so for virtual machines I will use the exact same system and the exact same kernel. I did not rebuild the kernel specifically for the domU, and it contains both the backend and the Xen frontend drivers for me.
')
For the “template” of the virtual machine, on the basis of which “instances” will then be created (you can draw an analogy with AMI and instances in Amazon EC2) I created a separate LVM logical partition and called it vmtemplate . I installed Gentoo for this section (the installation steps are similar to the steps in the standard Gentoo Handbook manual), as well as additional software that I chose based on which services will eventually work inside virtual machines (so that the “difference” between them is minimal). As a result, I got such a list of additional software:

• dev-lang / php
• dev-lang / v8
• dev-lang / erlang
• dev-python / django
• dev-ruby / rails
• www-apache / passenger
• www-apps / wordpress
• www-servers / nginx

To save space, installing packages in virtual machines removes all documentation (noman nodoc noinfo flags in make.conf). To make it convenient to update the software, I wrote a simple script that allows you to quickly enter the environment of our reference virtual machine and exit it:

 #!/bin/sh ROOT=/mnt/gentoo DEV=/dev/xenguests/vm-template-gentoo echo "Mounting filesystems" mount $DEV $ROOT mount -t proc none $ROOT/proc mount --bind /dev $ROOT/dev mount --bind /usr/portage $ROOT/usr/portage cp /etc/resolv.conf $ROOT/etc/resolv.conf chroot $ROOT /bin/bash echo "Unmounting filesystems" umount $ROOT/dev $ROOT/proc $ROOT/usr/portage umount $ROOT 

The installation process of the Xen hypervisor itself is not original, and for this you can use any instructions that are present in abundance on the Internet. However, if you, like me, want to use the hardened-core, then I draw your attention to the fact that my kernel under Xen categorically refused to boot at the earliest stages (when nothing else is displayed on the screen), so I At first I thought that it did not reach the kernel load.

By comparing the exception address that the hypervisor issued before dying with the contents of the System.map file, it was possible to find out that the reason lies in the function that operates on the kernel stack, and it worked when I disabled the CONFIG_PAX_RANDKSTACK function in the kernel (Randomize kernel stack base) related to PAX.

Create a virtual machine based on the reference

First, a small digression: to solve this problem, I tried to use LVM snapshots, but this turned out to be a bad solution, since the free space allocated in the copy for write is filled quite quickly, and that is logical if inside a virtual machine double-record the same file, then the amount of space occupied in the impression will grow by two sizes of the recorded file. Bad optimization. To solve the problem of optimizing the use of disk space, I decided to use aufs, and now we will finally deal with what was actually stated in the title of the article.

• For the virtual machine we create another logical partition LVM - let's call it vm-site1 . The size of this section in my case is 1 GB (the size of the partition from the reference VM is 8 GB). FS on both sections - ext4.
• We sys-fs/aufs3 on the main system - we will need it soon.

In order for the virtual machine to boot with this exotic configuration, you will need a special ramdisk that will mount the partitions correctly in the aufs. First, in the /etc/genkernel.conf file /etc/genkernel.conf uncomment the line: ALLRAMDISKMODULES="1" - this is necessary for the newly installed aufs module to be copied to ramdisk, which we will create using genkernel. I did not find a more elegant way of doing this, and I did not want to edit the system files in /usr/share/genkernel .

In the working directory, create an overlay folder, inside it a file with the name init , which is made executable:

 #!/bin/busybox sh mount -t proc -o noexec,nosuid,nodev proc /proc >/dev/null 2>&1 mount -o remount,rw / >/dev/null 2>&1 /bin/busybox --install -s if [ "$0" = '/init' ] then [ -e /linuxrc ] && rm /linuxrc fi modprobe xen-blkfront RO=/dev/xvda1 RW=/dev/xvda2 mknod /dev/xvda1 b 202 1 mknod /dev/xvda2 b 202 2 modprobe aufs mkdir /aufs mkdir /rw mkdir /ro mount $RO /ro mount $RW /rw mount -t aufs -o dirs=/rw:/ro=ro aufs /aufs [ -d /aufs/ro ] || mkdir /aufs/ro [ -d /aufs/rw ] || mkdir /aufs/rw mount --move /ro /aufs/ro mount --move /rw /aufs/rw cat /aufs/ro/etc/fstab | grep -v ' / ' | grep -v swap >> /aufs/etc/fstab ROTYPE=$(cat /proc/mounts | grep $RO | cut -d' ' -f3) ROOPTIONS=$(cat /proc/mounts | grep $RO | cut -d' ' -f4) RWTYPE=$(cat /proc/mounts | grep $RW | cut -d' ' -f3) RWOPTIONS=$(cat /proc/mounts | grep $RW | cut -d' ' -f4) echo $RO /ro $ROTYPE $ROOPTIONS 0 0 > /aufs/etc/fstab echo $RW /rw $RWTYPE $RWOPTIONS 0 0 >> /aufs/etc/fstab echo "cp /proc/mounts /etc/mtab" > /aufs/etc/local.d/mtab.start chmod a+x /aufs/etc/local.d/mtab.start echo "sysctl -w kernel.grsecurity.grsec_lock=1" > /aufs/etc/local.d/grsec.start chmod a+x /aufs/etc/local.d/grsec.start exec /sbin/switch_root -c "/dev/console" /aufs /sbin/init 

After that we create a modified ramdisk using a script like the one below:

 #!/bin/sh VERSION=`uname -r` MODULE=`modprobe -nv aufs | cut -d' ' -f2` if [ ! -f $MODULE ]; then echo "aufs module not found on your system" fi genkernel initramfs --no-install --no-postclear --initramfs-overlay=/home/xen/overlay cp -v /var/tmp/genkernel/initramfs-${VERSION} /boot/initramfs-domU 

Now the virtual machine will be able to boot from the root filesystem to aufs, while all changes will be written to the /dev/xvda2 , and /dev/xvda1 is our reference image, the files in which we can update if desired, and the updates will “pick up” all virtual machines (at the time of updating the reference image of the machine should be stopped). The partition with the data of a specific virtual machine (in our case, the LVM partition of vm-site1 , contains only differences from the reference filesystem, it can also be freely mounted on the host system, make changes to files, make backup copies, etc.

It remains to create a virtual machine. For this, I use this config:

 kernel = "/boot/vmlinuz" ramdisk = "/boot/initramfs-domU" memory = 128 name = "site1" vcpus = 1 disk = [ "phy:/dev/xenguests/vm-template,xvda1,r", "phy:/dev/xenguests/vm-site1,xvda2,w" ] root = "/dev/xvda1 ro" extra = "xencons=tty" on_poweroff = "destroy" on_reboot = "restart" on_crash = "destroy" vif = [ "mac=0a:11:10:24:14:20,bridge=br1" ] dhcp = "dhcp" 

Different virtual machines based on their unique MAC addresses are given static IP addresses through a DHCP server on the host system, and the DNS records are updated so that connections to nginx can be easily forwarded to the nginx web server inside the VM on the host system, as well as filter traffic. In addition, host-nginx is used to terminate SSL traffic.

Total

Goals achieved:

• The data of individual virtual machines now occupy exactly as much space as the difference between the reference system of what is inside the virtual machine. For a WordPress site, this difference makes me 45 MB .
• Using the login script to the "reference" environment, you can update the same type of software on all VMs at once simultaneously.