Resuscitate Ubuntu servers on Hetzner or some useful commands

This small post-cheat sheet is designed for those who suddenly during a failed experiment or update server stopped pinging, ssh fell off and so on. Simply put, it is about restoring the server after upgrades, hacks, and the like. I feel the latter is much less.

Mention hetzner are extremely fleeting in nature, due to the fact that often have to use his services.
')

Resue, mount partitions, chroot

So let's start with the fact that we have no access to the system, after, for example, another reboot. Therefore, we have two options - to reinstall the system or restore it. In the case of VPS, the Hetzner will simply roll the new system and, of course, nothing will remain on the disk. Therefore, we certainly choose recovery.
Who cares, to reinstall from rescue, use the command:

 installimage 

After that, hetzner will show the generated password. We reboot from the admin server and connect to it, better via IP address via ssh , ssh root@55.22.33.44
Login, of course, root.

After logging in, we are greeted by an invitation of this kind root@rescue chroot color will change to blue root@rescue

The first thing we do is look at the name of our disks:

 ls /dev/[hsv]d[az]*[0-9]* #    : /dev/sda /dev/sda1 /dev/sda2 /dev/sda3 

Then mount the disk with our system:

 mount /dev/sda3 /mnt 

Sections / boot and others do not touch yet. Once mounted, we need to make the contents of /dev /sys /proc visible otherwise, if we are left without a kernel, then it will not be installed.

 mount --bind /dev /mnt/dev/; mount --bind /proc /mnt/proc/; mount --bind /sys /mnt/sys/ 

in debian there is a handy command that replaces this line, it was not found in ubuntu .

 chroot-prepare /mnt 

Thereafter:

 chroot /mnt; 

Now you can dominate everything else: /boot

 mount -a 

At this stage, we have, in principle, a less recently controlled system that is less controlled by us.

Diagnostics

More often, everything has to be faced with upgrade or installation problems. Here, of course, it is difficult to detail something - you can only advise, if the distribution is updated, do it through the screen
For example:

 screen -S upgrade apt-get update apt-get dist-upgrade do-release-upgrade 

The reason for one of the most common problems is a banal lack of disk space.

 df -h 

Sometimes it happens that there is a place and there are no free nodes. Then this will also lead to a failure of the update and further problems.

 df -i 

Security

If there is a suspicion that we have been hacked or broken, the first thing is to look who is now on the server:

 who 

See who entered what commands:

 last 

Look at the story

 history 

Of course, these are all half measures, but nonetheless.
Next you need:

• Check /root/.ssh so that there are no left certificates.
• Look in /etc/passwd so that no one has authority except root .
• nmap so that there are no suspicious open ports, and if there is, then make sure that no one suspicious is listening to them.
• We change just in case the root password using passwd .
• The benefits of studying logs in /var/log can be invaluable.
• Check the system for rootkits

Upgrade and installation:

 apt-get install rkhunter rkhunter --update 

Rootkit scan:

 rkhunter -c -sk 

Warnin-gov he will most likely find a lot, especially in /bin and /usr/bin
There is also an alternative:
Installing chkrootkit :

 apt-get install chkrootkit 

Rootkit scan:

 chkrootkit 

Recovery

Consider the worst option, when the /boot folder is generally empty and is not worth it in the grub system, there is no kernel and most of the packages are broken.

System Restore

Clean the package archive

 apt-get clean 

Remove not deleted dependencies from already deleted packages.

 apt-get autoremove 

We grub2 :

 apt-get install grub2 

Write grub to the MBR

 grub-install /dev/sda 

Install or reinstall the normal kernel.

 apt-get install linux-image-xxx-xx-generic --reinstall 

We update the grub menu

 update-grub 

We execute commands designed to resolve dependency conflicts:

 apt-get install -f dpkg --configure -a 

Reinstall all packages:

 apt-get install --reinstall `dpkg --get-selections | grep -v deinstall | awk '{print $1}'` 

If you know what the problem was, then just reinstall the necessary package:

 apt-get install {_} --reinstall 

At the end you can once again perform:

 apt-get install -f dpkg --configure -a 

Conclusion

Pre-leaving the chroot with Ctrl+D or exit do:

 reboot 

See what happened. If the problem persists, then everything is new and see what we have missed, paying as much attention as possible to /var/log .

All proposed recipes are not a panacea. Perhaps, someone else solved similar problems, but these recipes often helped me.

Useful links:

wiki hetzner rescue system
ubuntu wiki recovery grub
screen