Moving a physical machine in a ProxmoxVE2 cluster (3) from DRBD to a new hardware

Having a cluster of two servers running on Proxmox with DRBD-integrated storage, it may be necessary to update the node in this cluster without stopping the operation of the machines in it. The task is not difficult, but some moments are not always remembered in the process.
Therefore, for myself, the actions were recorded for the future copy-paste into the console. Well, once already written, why not share with people?
PS At the end of a small bonus about the expansion of the block device called DRBD, tuning and useful links.


Let's get started

1. We migrate all virtual machines to one node which remains alive (PM2).
2. Go to the console on PM2 from the privileged user. We allow being in the cluster one node:

pvecm expected 1 

3. We look nodes in a cluster:

 pvecm nodes 

4. Remove the moving node from the cluster:

 pvecm delnode pm1 

5. Stop DRBD synchronization:

 drbdadm disconnect all 

and check the status of StandAlone team

 cat /proc/drbd 

6. Delete the old machine entries in / etc / pve / priv / {autorized_keys, know_hosts}, and also in /root/.ssh
7. Install the ProxmoxVE2 (PM1) on the new machine, specifying the dimensions of the sections during installation by writing in the invitation
boot: linux ext4 maxroot = 8 swapsize = 1 maxvz = 10 (there is also the key minfree = 200)
8. Go to the console fresh PM1, update:

 apt-get update && apt-get dist-upgrade 

9. Add to cluster:

 pvecm add 192.168.0.12 

(192.168.0.12 is the IP address of PM2)
10. Install the utilities for working with DRBD:

 apt-get install drbd8-utils 

but since the old utilities in the Debian repository, we need to update them with similar ones in the kernel:

 dpkg -i drbd8-utils_8.3.13-0_amd64.deb 

(in Proxmox 3 based on Debian 7, the version is relevant, so skip this point)
11. Create / copy a file with PM2 /etc/drbd.d/r0.res for example with the following content:

 resource r0 { protocol C; startup { wfc-timeout 0; degr-wfc-timeout 600; become-primary-on both; } syncer { rate 500M; } net { cram-hmac-alg sha1; shared-secret "my-secret"; allow-two-primaries; after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; sndbuf-size 0; no-tcp-cork; unplug-watermark 16; max-buffers 8000; max-epoch-size 8000; } on pm1 { device /dev/drbd0; disk /dev/pve/drbd; address 192.168.0.11:7788; meta-disk internal; } on pm2 { device /dev/drbd0; disk /dev/pve/drbd; address 192.168.0.12:7788; meta-disk internal; } 

12. Create a logical partition the size of exactly the same as on a working machine:

 lvcreate -L450G -n drbd0 pve 

and see what we have done the team

 lvscan 

13. Start drbd:

 /etc/init.d/drbd start 

14. Create a metadata:

 drbdadm create-md r0 

15. Just in case, we make secondary

 drbdadm secondary r0 

and disconnect from all

 drbdadm disconnect r0 

16. We say that the storage in our machine is not up to date with the command:

 drbdadm -- --discard-my-data connect r0 

and restart the car
17. In the console of the first server, we look at the synchronization status with the command:

 watch cat /proc/drbd 

18. In the web interface in the cluster stack we allow our nodes at the repository with drbd
')
Bonus: DRBD Volume Expansion

This is all true for the external meta disk.
1. We added the amount of disks in Reid. Expand the partition on which drbd lies:

 lvextend -L+250G /dev/pve/drbd0 

2. On an empty machine, we will make the repository secondary:

 drbdadm secondary r0 

3. Expand drbd by running a virtual machine on the machine, because the disk on the second machine temporarily stops:

 drbdadm resize r0 

4. The expansion of the drbd device will increase only this block device, but not pv, in order to increase it:

 pvresize /dev/pve/drbd0 

If the disk is internal, then it will be necessary to temporarily or permanently transfer the metadata to a separate disk. Without stopping it can be done in two stages.
1. On one of the servers:
a) Stop drbd:

 drbdadm down r0 

then we change the lines in the config
meta-disk internal
on
meta-disk / dev / ”your device for metadata” [0]
for which ssd disk is highly recommended, but for a while you can create an lvm partition.
b) Then on another machine

 drbdadm -- --overwrite-data-of-peer primary r0 

and on the current

 drbdadm up r0 

and watch the synchronization process

 watch cat /proc/drbd 

After the process is complete, we have metadata on this server transferred to a separate device.
2. Repeat step 1 on another server.

If there is no ssd, then after the extension, in order not to spoil the speed of the storage, you will have to do the opposite manipulations. But it is better to buy a couple of ssd, and maybe four, for stock or connecting them in RAID1 on each of the servers.

Before any actions it is quite natural to make backup copies, and a line in the /etc/vzdump.conf file will help to make them faster:

 bwlimit: 100000 

which will make the speed limit of 100 megabits, by default it costs only 10, which is quite normal for a 100 megabit network, but for gigabits (a) it is too soft.

Useful links:
Proxmox website
Proxmox wiki
Proxmox forum
Download image for installation

UPD: Congratulations on the release of Proxmox 3.0 based on Debian Wheezy