Remotely reinstalling Linux via ssh without accessing the console
It took me to reinstall the server, which seemed to be hosted by friends of my acquaintances. There was a very outdated Debian, and, most importantly, the system was on regular partitions without lvm and the space was not very optimally distributed. It was almost impossible to get physical access to it, the local admin could be asked to do something, but it could take a week. The server had a virtual KVM, but from the outside it was impossible to get there; as if the hoster had no unnecessary IP-addresses, and it was impossible to get inside its network. It was necessary to reinstall the server from under a running system via ssh. Yeah, let's change the turbine rotor without shutting down, then restart it and it will work with the new rotor!
The first idea was to create a chroot environment on the ramdisk and create lvm from it and flood the system. But it was not there, the system does not allow to change the partition table.
The second idea was to take the sources of the Debian distribution, stitch the server's IP address into them, rebuild the initrd with the Debian installer, ssh server and my IP, substitute this initrd into the grub config with the default block and reboot. After that, I had to get an ssh console with a network installer. I did it on the stand! But in battle everything ended in failure, the server did not rise. The hosts didn’t really need the server, and the case went dead, but I still have a feeling of an unsolved problem.
Once I discussed any destructive actions with the system with my colleagues (such as rm -rf /) and one of my colleagues said that you can disable the scsi disk that contains the root partition and the system does not peep. This gave me idea number three, take one idea, tear off the disk, return the disk and the returned disk will be different, not to those that the system did not give. It turned out that way. And now for the points, how can I reinstall the system without access to the physical console?
')
A warning! It is necessary to understand that everything that we will do is one way, in case of an error we lose access to the system! It is possible that you will have to go 1500 kilometers and climb into the mine to reanimate the server.
We assume that the IP of our system is 192.168.56.102. That was how it was on my stand. Plus access to the Internet through a proxy:
Getting started on the source system.
We come on ssh on the server:
Create a directory and file system for the “System killer”, mount it:
We put a great debootstrap utility that expands the minimal installation of Debian, using it we will create a chroot environment:
There are similar utilities for Fedora and Centos, respectively febootstrap and yumbootstrap, but I did not work with them.
Expand chroot:
The first argument is the version, the second is the installation directory, the third is the repository.
We backup the most necessary:
The most important thing is the settings of network interfaces; without them, it will not be possible to get into the reinstalled system.
Give the name of the chroot-environment:
The word "Killer_system" will appear in the bash prompt. This is an important thing, without it it will not be clear where we are at the moment.
Go to the new environment.
Mounting useful fs:
Once again we put debootstrap:
Next is my troubles: the debian openssh-server package has the xauth package in the recommended packages, and it has all kinds of xbox libraries in its dependencies. I, as a minimalist, do not want to have bits of X on the server where there wasn’t and there’s no graphics. Therefore, we put with the key - no-install-recommends:
Rule configs. We put an alternative port for the ssh daemon so that we can enter the chroot ssh system:
And allow root access:
You can not give root access, but create a user and give him sudo rights, but here I consciously simplify.
Next, you must set the root password, since by default debootstrap does not set any passwords:
Go to the chroot ssh environment:
We do this in order to completely get rid of the old system, from which we will tear off the disks. And so we will have a completely autonomous system in RAM, not connected with the old one.
Such a trick is very well suited if we leave the hosting provider, and we really do not want to leave our files to him (I know, paranoia). At this stage, we simply drive disks with zeros if we want to quickly:
Or random data in several passes if we want well. The advantage of the method is that we can wait for the end of the dd operation and, if necessary, repeat it. If we erase the discs directly from the combat system, then we will not be able to look at the results of the work of dd.
Let's try a simple way, delete volumes and partitions:
But bad luck. At the same time, the partition will be deleted, and the system will break down, but using the simple way without rebooting will fail. And there will be nowhere to be overloaded.
We will go the other way. We check where we have what is:
We assume that the root partition is on the sda ​​disk.
We erase a disk that in any case it did not pick up lvm.
A warning! After this point there is no return, even the next step is not as malicious. Let us think for a minute, check the console at which we sit and justify the name of our chroot:
We tear off the wheels:
We check, the disk has come off:
Connect the drive back:
Checking what is back:
There was sda, became sdb, great.
The important point is that on a boot disk you need to create one primary partition the size of the entire disk and give this partition to lvm in order for it to be able to get up to grub. All other disks can be given to lvm entirely without creating a partition system (pvcreate / dev / sdc). Create a partition table and one primary partition of type 8e, Linux LVM:
In the original version of the script, it was the creation of a single logical volume for the entire system, but when my colleague reinstalled Linux using this script, it turned out that creating several partitions presents some difficulty, especially a separate section for logs. Attention should be paid to the procedure for creating mount points and the actual mount partitions.
We are already deploying the combat system to a new place on the hard disk:
We return to the place backup copies of configs:
Now we are waiting for a new system:
Notice that at the command line prompt is now the name of the new chroot environment.
Mount file systems:
You can also mount these file systems from the parent chroot:
Install and configure openssh:
Install packages without which you can not do:
Yes, I can't live without vim and hate nano:
In principle, grub is prescribed where necessary even during installation, but, nevertheless, to support pants and morale we repeat:
Now we rule configs, at first the most important, without which the system will not rise:
In the interfaces file, everything should be fine, because somehow the network worked for us?
Add the proxy information to the apt configuration:
Change the hostname:
Add a line to / etc / hosts:
Add admin:
Unmount the file systems:
And exit from chroot:
Unmount the file systems:
If you cannot unmount / dev, you will not be able to unmount and / target, but this is not a problem.
If possible, then do so:
If not, then:
These commands will flush the disk caches and remount the root file system to read only. After that, you can reboot.
This is where a systemd surprise awaits us! He knows that we are in chroot and does not allow to reboot! Google gives advice to get out of chroot, but we have nowhere to go. But Magic SysRq comes to the rescue!
Activate SysRq (it is most likely activated, but do we need to make sure?).
And we are overloaded:
Drum roll, anxious waiting, did we really forget something, and the server did not go up?
Hooray! We are in the new system!
Re-create the initrd. This is not necessary, but will save you from some errors when rebooting:
Remove the file named chroot environment:
That's all.
Interesting article ways to overload the server
The first idea was to create a chroot environment on the ramdisk and create lvm from it and flood the system. But it was not there, the system does not allow to change the partition table.
The second idea was to take the sources of the Debian distribution, stitch the server's IP address into them, rebuild the initrd with the Debian installer, ssh server and my IP, substitute this initrd into the grub config with the default block and reboot. After that, I had to get an ssh console with a network installer. I did it on the stand! But in battle everything ended in failure, the server did not rise. The hosts didn’t really need the server, and the case went dead, but I still have a feeling of an unsolved problem.
Once I discussed any destructive actions with the system with my colleagues (such as rm -rf /) and one of my colleagues said that you can disable the scsi disk that contains the root partition and the system does not peep. This gave me idea number three, take one idea, tear off the disk, return the disk and the returned disk will be different, not to those that the system did not give. It turned out that way. And now for the points, how can I reinstall the system without access to the physical console?
')
A warning! It is necessary to understand that everything that we will do is one way, in case of an error we lose access to the system! It is possible that you will have to go 1500 kilometers and climb into the mine to reanimate the server.
We assume that the IP of our system is 192.168.56.102. That was how it was on my stand. Plus access to the Internet through a proxy:
http://proxy:8080 Getting started on the source system.
# System # 0
We come on ssh on the server:
ssh 192.168.56.102 Create a directory and file system for the “System killer”, mount it:
mkdir /target mount none -t tmpfs -o size=1G /target/ We put a great debootstrap utility that expands the minimal installation of Debian, using it we will create a chroot environment:
export http_proxy='http://proxy:8080' apt-get -y install debootstrap There are similar utilities for Fedora and Centos, respectively febootstrap and yumbootstrap, but I did not work with them.
Expand chroot:
debootstrap jessie /target/ http://mirror.mephi.ru/debian/ The first argument is the version, the second is the installation directory, the third is the repository.
We backup the most necessary:
mkdir /target/backup cp /etc/network/interfaces /target/backup The most important thing is the settings of network interfaces; without them, it will not be possible to get into the reinstalled system.
Give the name of the chroot-environment:
echo "Killer_system" > /target/etc/debian_chroot The word "Killer_system" will appear in the bash prompt. This is an important thing, without it it will not be clear where we are at the moment.
Go to the new environment.
# System # 1
chroot /target Mounting useful fs:
mount none -t proc /proc/ mount none -t sysfs /sys/ mount none -t devtmpfs /dev/ mount none -t devpts /dev/pts/ Once again we put debootstrap:
apt-get -y install lvm2 debootstrap Next is my troubles: the debian openssh-server package has the xauth package in the recommended packages, and it has all kinds of xbox libraries in its dependencies. I, as a minimalist, do not want to have bits of X on the server where there wasn’t and there’s no graphics. Therefore, we put with the key - no-install-recommends:
apt-get -y install openssh-server openssh-client openssh-blacklist openssh-blacklist-extra --no-install-recommends Rule configs. We put an alternative port for the ssh daemon so that we can enter the chroot ssh system:
sed -i 's/^Port .*$/Port 11122/' /etc/ssh/sshd_config And allow root access:
sed -i 's/^PermitRootLogin .*$/PermitRootLogin yes/' /etc/ssh/sshd_config /etc/init.d/ssh restart You can not give root access, but create a user and give him sudo rights, but here I consciously simplify.
Next, you must set the root password, since by default debootstrap does not set any passwords:
passwd root Go to the chroot ssh environment:
ssh 192.168.56.102 -l root -p 11122 We do this in order to completely get rid of the old system, from which we will tear off the disks. And so we will have a completely autonomous system in RAM, not connected with the old one.
Such a trick is very well suited if we leave the hosting provider, and we really do not want to leave our files to him (I know, paranoia). At this stage, we simply drive disks with zeros if we want to quickly:
dd if=/dev/zero of=/dev/sda bs=1M Or random data in several passes if we want well. The advantage of the method is that we can wait for the end of the dd operation and, if necessary, repeat it. If we erase the discs directly from the combat system, then we will not be able to look at the results of the work of dd.
Let's try a simple way, delete volumes and partitions:
# lvremove /dev/mapper/vg_old-root Logical volume vg_root/lv_root contains a filesystem in use. # fdisk /dev/sda Command (m for help): d Selected partition 1 Partition 1 has been deleted. Command (m for help): w The partition table has been altered. Calling ioctl() to re-read partition table. Re-reading the partition table failed.: Device or resource busy The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8). But bad luck. At the same time, the partition will be deleted, and the system will break down, but using the simple way without rebooting will fail. And there will be nowhere to be overloaded.
We will go the other way. We check where we have what is:
pvs lsblk We assume that the root partition is on the sda ​​disk.
We erase a disk that in any case it did not pick up lvm.
A warning! After this point there is no return, even the next step is not as malicious. Let us think for a minute, check the console at which we sit and justify the name of our chroot:
dd if=/dev/zero of=/dev/sda bs=1M count=100 We tear off the wheels:
echo 1 > /sys/block/sda/device/delete We check, the disk has come off:
lsblk Connect the drive back:
for i in /sys/class/scsi_host/host?/scan ; do echo "- - -" > $i ; done Checking what is back:
lsblk There was sda, became sdb, great.
The important point is that on a boot disk you need to create one primary partition the size of the entire disk and give this partition to lvm in order for it to be able to get up to grub. All other disks can be given to lvm entirely without creating a partition system (pvcreate / dev / sdc). Create a partition table and one primary partition of type 8e, Linux LVM:
fdisk /dev/sdb n<CR> <CR> <CR> <CR> t<CR> 8e<CR> w<CR> # create new primary partition from start to end; 8e type In the original version of the script, it was the creation of a single logical volume for the entire system, but when my colleague reinstalled Linux using this script, it turned out that creating several partitions presents some difficulty, especially a separate section for logs. Attention should be paid to the procedure for creating mount points and the actual mount partitions.
pvcreate /dev/sdb1 vgcreate vg_root /dev/sdb1 lvcreate -Zn -L500M -n lv_swap0 vg_root lvcreate -Zn -L1G -n lv_root vg_root lvcreate -Zn -L2G -n lv_usr vg_root lvcreate -Zn -L2G -n lv_var vg_root lvcreate -Zn -L1G -n lv_var_log vg_root lvcreate -Zn -L1G -n lv_home vg_root mkswap /dev/vg_root/lv_swap0 mkfs.ext4 /dev/mapper/vg_root-lv_root mkfs.ext4 /dev/mapper/vg_root-lv_usr mkfs.ext4 /dev/mapper/vg_root-lv_var mkfs.ext4 /dev/mapper/vg_root-lv_var_log mkfs.ext4 /dev/mapper/vg_root-lv_home mkdir /target mount /dev/mapper/vg_root-lv_root /target/ mkdir /target/usr /target/var /target/home mount /dev/mapper/vg_root-lv_usr /target/usr mount /dev/mapper/vg_root-lv_var /target/var mkdir /target/var/log mount /dev/mapper/vg_root-lv_var_log /target/var/log mount /dev/mapper/vg_root-lv_home /target/home We are already deploying the combat system to a new place on the hard disk:
export http_proxy='http://proxy:8080' debootstrap jessie /target/ http://mirror.mephi.ru/debian/ echo "NEW_system" > /target/etc/debian_chroot We return to the place backup copies of configs:
cp /backup/interfaces /target/etc/network Now we are waiting for a new system:
# System # 2
chroot /target Notice that at the command line prompt is now the name of the new chroot environment.
Mount file systems:
mount none -t proc /proc/ mount none -t sysfs /sys/ mount none -t devtmpfs /dev/ mount none -t devpts /dev/pts/ You can also mount these file systems from the parent chroot:
mount -o bind /proc/ /target/proc mount -o bind /sys/ /target/sys mount -o bind /dev/ /target/dev mount -o bind /dev/pts /target/dev/pts Install and configure openssh:
apt-get -y install openssh-server openssh-client openssh-blacklist openssh-blacklist-extra --no-install-recommends sed -i 's/^PermitRootLogin .*$/PermitRootLogin yes/' /etc/ssh/sshd_config passwd root Install packages without which you can not do:
apt-get -y install vim sudo linux-image-3.16.0-4-amd64 grub2 lvm2 psmisc vlan Yes, I can't live without vim and hate nano:
update-alternatives --set editor /usr/bin/vim.basic In principle, grub is prescribed where necessary even during installation, but, nevertheless, to support pants and morale we repeat:
update-grub grub-install /dev/sdb Now we rule configs, at first the most important, without which the system will not rise:
cat > /etc/fstab <<EOF # /etc/fstab: static file system information. # # Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices # that works even if disks are added and removed. See fstab(5). # # <file system> <mount point> <type> <options> <dump> <pass> /dev/mapper/vg_root-lv_root / ext4 errors=remount-ro 0 1 /dev/mapper/vg_root-lv_usr /usr ext4 defaults 0 2 /dev/mapper/vg_root-lv_var /var ext4 defaults 0 2 /dev/mapper/vg_root-lv_var_log /var/log ext4 defaults 0 2 /dev/mapper/vg_root-lv_home /home ext4 defaults 0 2 EOF In the interfaces file, everything should be fine, because somehow the network worked for us?
vim /etc/network/interfaces Add the proxy information to the apt configuration:
echo 'Acquire::http::Proxy "http://proxy:8080";' > /etc/apt/apt.conf Change the hostname:
echo new-system > /etc/hostname Add a line to / etc / hosts:
echo "192.168.56.102 new-system.corp new-system" >> /etc/hosts Add admin:
adduser admin usermod -a -G sudo admin visudo Unmount the file systems:
umount /dev/pts umount /dev/ umount /proc/ umount /sys/ And exit from chroot:
exit Unmount the file systems:
umount /target/usr/ /target/var/log/ /target/var/ /target/home/ If you cannot unmount / dev, you will not be able to unmount and / target, but this is not a problem.
If possible, then do so:
umount /target/ If not, then:
sync ; sync ; sync ; mount -o remount,ro /target/ These commands will flush the disk caches and remount the root file system to read only. After that, you can reboot.
This is where a systemd surprise awaits us! He knows that we are in chroot and does not allow to reboot! Google gives advice to get out of chroot, but we have nowhere to go. But Magic SysRq comes to the rescue!
Activate SysRq (it is most likely activated, but do we need to make sure?).
echo 1 > /proc/sys/kernel/sysrq And we are overloaded:
echo b > /proc/sysrq-trigger Drum roll, anxious waiting, did we really forget something, and the server did not go up?
ssh 192.168.56.102 Hooray! We are in the new system!
Re-create the initrd. This is not necessary, but will save you from some errors when rebooting:
update-initramfs -u Remove the file named chroot environment:
rm /etc/debian_chroot That's all.
## The end References:
Interesting article ways to overload the server
Source: https://habr.com/ru/post/321696/
All Articles