LINBIT recently released its new solution for orchestrating and managing multiple DRBD arrays.

For example, you can have several nodes and each will have its own LVM or ZFS pool in which LINSTOR will automatically create new volumes and replicate them between nodes using the DRBD protocol.

LINSTOR supports thin-provisioning, snapshots, and many other interesting pieces.
This solution is well suited for virtual machines and containers.

LINSTOR Satellites

I believe that you already have a Proxmox cluster, I have three pve1 , pve2 and pve3 .

So the first step is to install drbd-utils and build a kernel module on each node.

Add a linstor repository and install kernel-headers , as well as all the necessary packages from it:

 wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add - echo "deb http://packages.linbit.com/proxmox/ proxmox-5 drbd-9.0" \ > /etc/apt/sources.list.d/linbit.list apt-get update apt-get -y install pve-headers apt-get -y install drbd-dkms drbdtop 

After installation, let's check your version of the kernel module:

 modproble drbd cat /proc/drbd 

If you see version 8, then something went wrong and you have loaded the in-tree kernel module, check the dkms status to learn more about building the dkms module directly.
For LINSTOR, you must use DRBD version 9.

Next, on each node, install the linstor-proxmox and linstor-satellite packages:

 apt-get -y install linstor-proxmox linstor-satellite systemctl start linstor-satellite.service systemctl enable linstor-satellite.service 

LINSTOR Controller

We will deploy the controller inside the LXC container.

Download the debian template:

 wget http://download.proxmox.com/images/system/debian-9.0-standard_9.3-1_amd64.tar.gz -P /var/lib/vz/template/cache/ 

Now we will create a container for the controller:

 pct create 100 local:vztmpl/debian-9.0-standard_9.3-1_amd64.tar.gz \ --hostname=linstor-controller \ --net0=name=eth0,bridge=vmbr0,gw=10.1.0.1,ip=10.1.0.123/16 

Run the container and enter it:

 pct start 100 pct exec 100 bash 

Install updates:

 apt-get update apt-get -y upgrade 

Add a linstor repository and install linstor-controller and linstor-client :

 wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add - echo "deb http://packages.linbit.com/proxmox/ proxmox-5 drbd-9.0" \ > /etc/apt/sources.list.d/linbit.list apt-get update && apt-get install -y linstor-controller linstor-client systemctl start linstor-controller.service systemctl enable linstor-controller.service 

Linstor requires a customized locale. Set up the locale:

 sed -i '/en_US.UTF-8 UTF-8/ s/^# //' /etc/locale.gen locale-gen 

Let's set up the time zone right away:

 dpkg-reconfigure tzdata 

Configuring Storage

Create nodes:

 linstor node create pve1 10.1.0.11 linstor node create pve2 10.1.0.12 linstor node create pve3 10.1.0.13 

For each node, we describe an additional interface that will be used for DRBD replication:

 linstor node interface create pve1 data 10.2.0.11 linstor node interface create pve2 data 10.2.0.12 linstor node interface create pve3 data 10.2.0.13 

Example output of the linstor node list command:

 ╭──────────────────────────────────────────────────────────────╮ ┊ Node ┊ NodeType ┊ Addresses ┊ State ┊ ╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡ ┊ pve1 ┊ SATELLITE ┊ 10.0.1.11,10.0.2.11:3366 (PLAIN) ┊ Online ┊ ┊ pve2 ┊ SATELLITE ┊ 10.0.1.12,10.0.2.12:3366 (PLAIN) ┊ Online ┊ ┊ pve3 ┊ SATELLITE ┊ 10.0.1.13,10.0.2.13:3366 (PLAIN) ┊ Online ┊ ╰──────────────────────────────────────────────────────────────╯ 

Create pools:

On each node you should prepare lvm thin-pool for drbd:

 lvcreate -L 800G --thinpool drbdpool pve 

Now you can add them to linstor:

 linstor storage-pool create lvmthin pve1 drbdpool pve/drbdpool linstor storage-pool create lvmthin pve2 drbdpool pve/drbdpool linstor storage-pool create lvmthin pve3 drbdpool pve/drbdpool 

Set up pools to use the data interface:

 linstor storage-pool set-property pve1 drbdpool PrefNic data linstor storage-pool set-property pve2 drbdpool PrefNic data linstor storage-pool set-property pve3 drbdpool PrefNic data 

Example output from the linstor storage-pool list command:

 ╭─────────────────────────────────────────────────────────────────────────────────────────── ┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ Support ╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ ┊ drbdpool ┊ pve1 ┊ LvmThinDriver ┊ hv1/drbdpool ┊ 800 GiB ┊ 800 GiB ┊ true ┊ drbdpool ┊ pve1 ┊ LvmThinDriver ┊ hv1/drbdpool ┊ 800 GiB ┊ 800 GiB ┊ true ┊ drbdpool ┊ pve3 ┊ LvmThinDriver ┊ hv3/drbdpool ┊ 800 GiB ┊ 800 GiB ┊ true ╰─────────────────────────────────────────────────────────────────────────────────────────── 

Now we can add our new repository to the Proxmox config:

 cat >> /etc/pve/storage.cfg <<EOF drbd: drbdpool content rootdir,images controller 10.1.0.123 controllervm 100 redundancy 3 EOF 

Configuring HA for the controller

Now we will transfer our linstor container to linstor storage.

First we need to create a virtual disk for it:

 pct exec 100 bash linstor resource-definition create vm-100-disk-0 linstor volume-definition create vm-100-disk-0 4G linstor resource create vm-100-disk-0 --auto-place 3 -s pve 

Example output from the linstor resource list command:

 ╭──────────────────────────────────────────╮ ┊ ResourceName ┊ Node ┊ Port ┊ State ┊ ╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡ ┊ vm-100-disk-0 ┊ pve1 ┊ 7000 ┊ UpToDate ┊ ┊ vm-100-disk-0 ┊ pve2 ┊ 7000 ┊ UpToDate ┊ ┊ vm-100-disk-0 ┊ pve3 ┊ 7000 ┊ UpToDate ┊ ╰──────────────────────────────────────────╯ 

By default, linstor-satellite deletes the configs of all resources at the time the service starts, then it expects a new configuration from the linstor-controller , but if we keep the linstor-controller in the same place as the data, we must make an exception for its resource by adding the option --keep-res=vm-100 for linstor-satellite on all nodes:

 echo -e "[Service]\nExecStart=\nExecStart=/usr/share/linstor-server/bin/Satellite --logs=/var/log/linstor-satellite --config-directory=/etc/linstor --keep-res=vm-100" | SYSTEMD_EDITOR=tee systemctl edit linstor-satellite.service 

This action should prevent the removal of the vm-100-disk-0 resource each time it is started.

Now we stop the linstor-controller container and transfer all data from the local disk to the drbd disk.

 pct shutdown 100 dd if=/var/lib/vz/images/100/vm-100-disk-0.raw of=/dev/drbd/by-res/vm-100-disk-0/0 bs=8M status=progress e2fsck -f /dev/drbd/by-res/vm-100-disk-0/0 resize2fs /dev/drbd/by-res/vm-100-disk-0/0 

Update the container config:

 sed -i '/^rootfs:/d' /etc/pve/lxc/100.conf echo 'rootfs: drbdpool:vm-100-disk-0,size=4G' >> /etc/pve/lxc/100.conf 

Run the container, and if everything is OK, remove the old disk:

 pct start 100 rm -f /var/lib/vz/images/100/vm-100-disk-0.raw 

Now we just need to add our container to proxmox ha-manager :

 ha-manager add ct:100 --max_relocate=3 --max_restart=3 

PROFIT

Performance and tuning

At this point, to change the global parameters of the controller, you will need to disable authentication on the controller:

• https://github.com/LINBIT/linbit-documentation/blob/master/UG9/en/administration-linstor.adoc#disable-user-security

To automatically resolve split-brain situations, add the following options for the controller:

 linstor controller drbd-options \ --after-sb-0pri=discard-zero-changes \ --after-sb-1pri=discard-secondary \ --after-sb-2pri=disconnect 

For my 10G network, I found the following settings to be optimal for fast synchronization:

 linstor controller drbd-options \ --max-buffers=36864 \ --rcvbuf-size=2097152 \ --sndbuf-size=1048576 linstor controller drbd-options \ --c-fill-target=10240 \ --c-max-rate=737280 \ --c-min-rate=20480 \ --c-plan-ahead=10