ZFS and KVM. @home

This symbolism is already well known, so it makes no sense to explain in detail what kind of animal it is .

There will be no comparison with LVM , for it is possible, but meaningless, to compare a muscle-car with a jet-truck.
Graphics and comics are also not delivered.
It is, let's say, an unfinished success story, because the upgrade by which it was caused can only be stopped, but not completed.

Prerequisites

Practically in every IT family there are several PCs and / or laptops, for sure there is an HTPC or even a NAS. So we had two PCs, then a large wall monitor was added, and in this situation the birth of the HTPC is a matter of time. When he was born, he was immediately named a proud name - Server. Later it turned out that some of the favorite games can do in Linux - and the Server immediately got a Steam account.

So a few years passed, during which the Server received 16GB of memory, virtuals with gitlab and projects appeared on it ...

And then the understanding came that an upgrade is brewing, and upgrading 3 PCs at once is too expensive; At the same time, there was already a successful experience with PCIE-passthrough and a virtual virtual machine.

A suitable solution was the gradual migration of gaming machines to virtuals. While only one is emigrated, in the future, under the guest, I will mean it.
Of the three processors (i3-3220, i5-3470, i5- 3470K), VT-d supported only the second, and it went to the Server instead of i3. A low-profile 1050Ti was bought to replace the 7970, and the old iron was sold in parts.

The server received the latest UEFI-firmware, Ubuntu 16.04 and the largest SSD available - Crucial M4, 256 GB, under the root pool. The installation was carried out according to this manual , from which the installation script was accumulated in the live system. Then, after a successful download, virt-manager , libvirt-bin , ovmf and xubuntu-desktop were installed. Of course, VT-x and VT-d were included.

By the way, the separation of the binary part of the system from logs, caches and other hamsters more than once helped in the process of vivisection of the system of experiments, allowing you to return the working state to a simple zfs rollback , accessible even from initrd (but there is a feature: if you use a separate datasset for /root , use the parameter kernel init=/bin/sh for emergency boot; sh, unlike bash, will not create garbage at the /root mount point).
To avoid guessing what happened, in /etc/default/grub comment out everything related to HIDDEN_TIMEOUT , obviously set a timeout other than 0 (I have 10), uncomment / add GRUB_TERMINAL=console and be sure to remove quiet splash from GRUB_CMDLINE .

Error-43 and all-all

• initial configuration
  UEFI on the host and the guest turned out to be a pledge of painless forwarding and video card operation, but for this the video card must have a fully UEFI-compatible firmware. I was lucky with this, and the old reference 7970 and the new 1050Ti were compatible. To create a UEFI guest, you will need the ovmf package, also (if you will create a guest using virt-manager), you must check the last page of the wizard, select and select UEFI as the microcode and Q35 as the chipset. On my system without performing these actions, the guest system had strange problems with fading and stuttering sound.
• probros devices
  For forwarding devices, you need to tell the kernel to use a special driver - vfio-pci or pci-stub (already out of date) for certain id vfio-pci.ids=10de:1c82,10de:0fb9,8086:1e31 (I have the video card itself thrown, its sound and a four-port USB3 controller) and allow the use of iommu intel_iommu=on (other parameters are intel_iommu=on to AMD, but I will not lie for lack). Also, in order to send a video card, you will need to explicitly prohibit its use by the video driver, since nvidia was being forwarded and the proprietary driver was not installed, then for me it looks like this: modprobe.blacklist=nouveau nouveau.modeset=0 . All these parameters are written in the /etc/default/grub file, in the GRUB_CMDLINE line. Do not forget to run update-grub and restart before adding guest-ready devices for probros.
• nVidia, or "nothing personal, it's just business."
  Unfortunately, in this life, nothing happens simply and immediately, so the notorious Error 43 was waiting for me. The trick with cpu=host although useful, did not help. Fortunately, it was enough to use virsh edit <guestname> , and in the <features>/<hyperv> add the line <vendor_id state='on' value='ASUS'/> . But in turn, libvirt from the repository (including backports) is too rotten and did not understand this parameter. After several unsuccessful attempts to build and install a more recent version, I turned to ppa, and Jacob Zimmermann found a suitable package. I recommend, having found a stable version, not to update the driver on the video card in the guest, due to the constant mutation of the business bug with forwarding.

Using ZFS

• Lxc
  It's all very simple - we create a dataset (for example, like this: zfs create rpool/lxc ), run lxc init , choose the type of ZFS storage, refuse to create a pool, agree to use dataset, enter the name of the dataset created a little earlier. Now they live there gitlab, samba-dc (with the key --usentvfs, a samba over 4.3 will not start), a couple of small projects.
• Deduplication, compression and blocks.
  To provide guests with block devices, I use zvol - it is difficult to accidentally delete it, just to create individual images. But zvol has one tricky feature: if it is not sparse (the -s key when created), the snapshot will take up as much space as the original.
  ')
  When using ZFS, there are two ways to save disk space — compression and deduplication. And if everything is clear with compression, deduplication should be disassembled in more detail.
  
  Overhead for deduplication is from ~ 200 to ~ 700 bytes in memory on real systems (for details, see the output of zdb -D <poolname> ), and DDT (deduplication table), the size of which depends directly on the number of unique blocks, should be stored in memory entirely; and this means that it is simply unprofitable to use deduplication with a small block size (except, perhaps, a virtual farm with a dedup_ratio of a much larger unit and a titanic amount of RAM). In addition, with a small block size (this applies primarily to zvol), it is impossible to use compression other than zle.
  
  Needless to say, the size of the zvol block should coincide with the cluster size of the FS located on it, and it should be aligned with the block size (modern utilities have long been aligned along the 1MB border, which is more than enough)?
  
  It turns out that the use of large blocks favorably affects compression, deduplication and memory overhead, while compression also minimizes the overhead from large blocks in the guest file system.
  
  So, I create all zvol with zfs create -V <size> -s -b 64K <zfs path> , adding -o dedup=on and -o compression=lz4 as necessary.
• 64KB is our choice.
  It remains to teach the guest to boot and work with filesystems with this cluster size. In the case of Linux, using ext4, for example, with such a cluster size is impossible - it simply cannot be mounted. However, by manipulating the mount parameters stride and stripe-width, you can achieve the desired behavior.
  
  In the case of NTFS, everything is trickier and simpler at the same time: Windows creates service partitions whose contents change extremely rarely, so you can ignore the size of their cluster (and the boot partition also cannot be changed - neither bootmgr nor ntldr understands the cluster size anymore 4KB). If, after selecting unallocated space, you click not on the "Install" button, but on "Create", the installer will create and format all the necessary partitions, and you will have the disk number and partition number in which the guest system will be installed.
  
  Next you need a bit of black magic:
  
  • draw a pentagram, not forgetting to set one black candle that once burned for each crosshair of lines
  • call a dremor with a rank of no lower than qwd cmd (press Shift-F10 )
  • we start diskpart , and further in it
  • select disk < >
  • select partition < >
  • format FS=NTFS UNIT=64K QUICK
  • assign LETTER=S
  • we leave diskpart
  • in cmd we execute mkdir s:\windows .

We return to the installation wizard, update the disk configuration, select the largest partition created (with which the magic was done) and continue the installation as usual.

  ,       64,  Windows ,       ,        -  .  ?  ,       Windows-:    ,  ,   ,   . 

Deduplicate Steam, or where it all began

The easiest way was to get smb or nfs balls and create Steam libraries on them. Regular files are easily and naturally deduplicated, including both exported libraries and the Linux client library, without unnecessary gestures. As always, the most obvious solution turned out to be in principle working, but wrong. SMB in this mode moves with the speed of a notebook hard disk, despite the fact that the files are on the SSD. NFS (and Windows 8.1 supports NFS v3, not 4) moved a little faster, it felt like some kind of "green" desktop drive, but it was still too little for a comfortable game. In addition, all the operating systems from M $ love to lose network drives that are automatically connected when logon, and each time pointing Steam to the location of the library quickly get bored.

ISCSI is the solution. The targetcli was installed on the targetcli , configured by this manual , adjusted for blockio usage and by the form /dev/zvols/<zfs path> . For unmap to work properly, you need to use Windows 8+ (starting with 8, unmap support for the iSCSI initiator is implemented), and set attributes emulate_3pc=1,emulate_tpu=1,emulate_caw=1,emulate_tpws=1,emulate_tas=1 for each block device. If done correctly, then deleting files and formatting should free up space and reduce the size of the zvol.

Now identical files in volumes exported for libraries take up space only once, but the Steam library for Linux still occupies a separate place. Let's create a separate dataset for this library with options -o dedup=on -o recordsize=64K . If the sections in zvol are aligned on the 1MB border, and the cluster size is set to 64KB, then it is logical that you need to share files on the host with the same granularity so that deduplication can find the same blocks.

Effects

• The 16GB of memory I have at home is still not enough for a guest with 8GB, firefox on the host, containers and other things - from time to time there are strange slowdowns accompanied by intense disk activity.
• deduplicated storage is better to make a separate pool.
• installing zfs set logbias=throughput <pool> significantly affects performance, especially when using deduplication. Not only does this exclude an additional write-to-log / erase-log loop, it also stops the drive's write cache.
• on a gigabit zvol network on SSD it feels much faster than a hard disk, although it is noticeably slower than a local SSD
• I thought about 960EVO on 1TB

Instead of conclusion

Well, my little experiment ended with undeniable success, although difficulties arose on the way. Further, I would like to upgrade to Zen2, 32-64GB of memory, NVME SSD, transfer a second PC to the Server ... but this will be a completely different witch.