Hetzner may unexpectedly shut down your server
We are a small group of web developers, we write websites to order, we host either from ourselves or from different providers around the world. We have a small department of those. support, we try to respond to the problems that arise as far as possible. The article is written by those who have their own servers on Hetzner, so that they are ready for certain features of support.
There are also such cases as described below. We were not ready for it, both morally and technically. And we are faced with the fact that the server can be turned off, and to understand or eliminate the cause of the shutdown is impossible in operation.
')
The site stopped pinging. Our support team immediately tried to figure out the reason.
The customer of this site is in the US (he sleeps at night), the support team is in Ukraine, Hetzner is in Germany. The site went down at the most opportune time, when it was night in the USA, and in Ukraine and in Germany normal working hours, which, in theory, should give us the opportunity to restore the site’s work. Oh, ...
When trying to enter Rescue Mode, we stumbled upon a strange message:
"The Ip is locked" with reference to: " wiki.hetzner.de/index.php/Leitfaden_bei_Serversperrung/en "
We read ... Tips like: "First of all, please check the log files of the server" I was somewhat puzzled.
Q: How to check log files if access is disabled?
Further, more interesting:
Q: There is no access, the server is disabled, you can turn it on only if you fix it, but to fix it, you need to go in? ..
It turned out that this type of problem should (according to the Hetzner rules) be resolved through KVM. Requested KVM, but it is not clear what to repair. In the admin there is no mention of specifics, there is only a link to a general document about possible problems.
Well, it is logical, if nothing is clear, you need to ask for support:
The support works, but the situation reminds me of an anecdote about the programmers in the balloon, when the answer turned out to be correct and completely useless, and time goes on. Each question / answer is about an hour of delay.
I urgently call our client in the USA, asking them to check and forward all mail from hetzner. Finally, I get a less clear explanation:
KVM (LARA) gave (+1 h), 3 hours later after downtime, finally we start to do something and there is hope to deal with the problem. We get access to the server through Lara.
Considering that there is no network, it is not possible to install the root kit check. We check what we can check, but even tcpdump cannot be started - the port is down and the packets are refused to be sent.
Our system architecture:
The host system with KVM, kernel 3.5.2, regular updates GLSA, port only SSH is on guard of borders (I lie, there was still nrpe, checked, it seems there are no references to the fact that nagios-nrpe broke).
Understanding that on the Web part of the project a “hodgepodge” of different technologies is installed, including PHP code, therefore the virtual machines are physically separated from the outside world, and they certainly cannot capture a foreign IP address in any way.
A study of the basic host of the Gentoo system did not show any changes. In the messages, dmesg, last file, everything is fine too.
In general, after a series of studies, they wrote in support of them that we cannot find anything ourselves and we need help from them to identify the problem. We also asked to make sure that incomprehensible traffic comes specifically from our VLAN.
Received the answer:
Hetzner> Please follow us through fax or email:
Hetzner> www.hetzner.de/pdf/en/Comment_Serversuspension.pdf
Hmm ... Then I get a little type. It took 4 hours, we did not move a jot, soon morning in the US, adequate (in my opinion) help from Hetzner did not follow, and instead of helping, they ask to send them a fax. Che is sad ...
Q: what to subscribe in the form of "on the elimination of the problem," if we could not find the problem, and accordingly did not fix anything (and said about it)? .. Asked a question. He received the answer that they have no right to turn on the server for any proceedings until they receive a fax / scan with a signature.
Considering that we have already decided to recover from backups at this point (it took a lot of time to ask questions / answers and attempts to understand what the problem was), there was no more rush, we calmly filled out the form and began to expect results.
After some time, we get the answer:
Ok ... Considering that the server was turned on for about a minute, we didn’t have time to see anything.
After that, we asked what can now be done with the server, without having network access to it, we were advised to format the server and install a new system by mounting the remote ISO image.
We tried to do it, but we were not technically ready (there wasn’t a small ISO on hand, all were large for the desktop, tried to load X, and were not very suitable for installation via KVM). The story ended around 10 pm, when KVM was chopped off during the installation of the system (the rule in Hetzner - free KVM is given for no more than 2 hours).
In a relaxed atmosphere, on Monday morning, they made an application for KVM, got access, installed a small system on sda1 (raid collapsed) via Lara / ISO mount image, sent a fax scan about problem solving, and received an answer that the server is activated. But for some reason he did not ping ...
After re-applying for activation, the site finally stopped, we logged off the network, and the first thing I did was a full tar cjvpf backup of the old main system, download it to myself for experiments and deployed locally.
Locally also could not find the problem. I picked up a separate machine as the default router, turned it on to the one on which the copy was running, looked at the tcpdump traffic at both ends, raised the NAT from the gw address for the grid. No strange packages found.
We checked on the root kit, checked all the packages and files - there is not a single modified MD5, there are no extra processes, and so on.
What was the problem? One can only guess. Perhaps they have the same MAC addresses on the network, maybe something else ... Who else can tell?
In general, even if there were problems with us (which I doubt), it is impossible to find out. And they, for their part, are hardly recognized or will be able to provide a specialist for help.
1) Always have a small ISO image of your system at hand, so that at the request of hetzner you can quickly format your server. It is desirable that the main section and services / data are separated.
2) It is highly desirable to stream netflow outside Hetzner, so that you can check the charges later.
3) Always be prepared for the fact that the server can disappear for good (which, incidentally, is the right strategy).
PS Admins tell me that this is normal: first we chop off the problem, then we understand what it is. But if the former occurs promptly, then the proceedings must follow.
There are also such cases as described below. We were not ready for it, both morally and technically. And we are faced with the fact that the server can be turned off, and to understand or eliminate the cause of the shutdown is impossible in operation.
10.36 GMT +2
')
The site stopped pinging. Our support team immediately tried to figure out the reason.
The customer of this site is in the US (he sleeps at night), the support team is in Ukraine, Hetzner is in Germany. The site went down at the most opportune time, when it was night in the USA, and in Ukraine and in Germany normal working hours, which, in theory, should give us the opportunity to restore the site’s work. Oh, ...
When trying to enter Rescue Mode, we stumbled upon a strange message:
"The Ip is locked" with reference to: " wiki.hetzner.de/index.php/Leitfaden_bei_Serversperrung/en "
We read ... Tips like: "First of all, please check the log files of the server" I was somewhat puzzled.
Q: How to check log files if access is disabled?
Further, more interesting:
"Before the server can go back online, We require a signed statement
from the problem, explaining how you have solved the problem
"
Q: There is no access, the server is disabled, you can turn it on only if you fix it, but to fix it, you need to go in? ..
It turned out that this type of problem should (according to the Hetzner rules) be resolved through KVM. Requested KVM, but it is not clear what to repair. In the admin there is no mention of specifics, there is only a link to a general document about possible problems.
Well, it is logical, if nothing is clear, you need to ask for support:
Question to Hetzner 11:36> I can not ping and login to the server
Reply from Hetzner 12:25> please check your mails, you should have received a mail.
The support works, but the situation reminds me of an anecdote about the programmers in the balloon, when the answer turned out to be correct and completely useless, and time goes on. Each question / answer is about an hour of delay.
I urgently call our client in the USA, asking them to check and forward all mail from hetzner. Finally, I get a less clear explanation:
Dear Sir or Madam IPs from the same sub net in addition to the main IP mentioned in the above subject line. This is not the case. deactivated. May be found at http://wiki.hetzner.de/index.php/Leitfaden_bei_Serversperrung/en. Yours faithfully Your Hetzner Support Team 09: 29: 55.027863 a8: be: dd: 56: e7: 15> cf: 40: 04: 22: 32: 1f, ethertype IPv4 (0x0800), length 66: 188.40.25.34.42709> 5.9.xx.xx.80: Flags [.], ack16154, win 661, options [nop, nop, TS val 1003012 ecr 2687744519], length 0
13:21
KVM (LARA) gave (+1 h), 3 hours later after downtime, finally we start to do something and there is hope to deal with the problem. We get access to the server through Lara.
Considering that there is no network, it is not possible to install the root kit check. We check what we can check, but even tcpdump cannot be started - the port is down and the packets are refused to be sent.
Our system architecture:
The host system with KVM, kernel 3.5.2, regular updates GLSA, port only SSH is on guard of borders (I lie, there was still nrpe, checked, it seems there are no references to the fact that nagios-nrpe broke).
Understanding that on the Web part of the project a “hodgepodge” of different technologies is installed, including PHP code, therefore the virtual machines are physically separated from the outside world, and they certainly cannot capture a foreign IP address in any way.
A study of the basic host of the Gentoo system did not show any changes. In the messages, dmesg, last file, everything is fine too.
In general, after a series of studies, they wrote in support of them that we cannot find anything ourselves and we need help from them to identify the problem. We also asked to make sure that incomprehensible traffic comes specifically from our VLAN.
15:40
Received the answer:
Hetzner> Please follow us through fax or email:
Hetzner> www.hetzner.de/pdf/en/Comment_Serversuspension.pdf
Hmm ... Then I get a little type. It took 4 hours, we did not move a jot, soon morning in the US, adequate (in my opinion) help from Hetzner did not follow, and instead of helping, they ask to send them a fax. Che is sad ...
Q: what to subscribe in the form of "on the elimination of the problem," if we could not find the problem, and accordingly did not fix anything (and said about it)? .. Asked a question. He received the answer that they have no right to turn on the server for any proceedings until they receive a fax / scan with a signature.
Considering that we have already decided to recover from backups at this point (it took a lot of time to ask questions / answers and attempts to understand what the problem was), there was no more rush, we calmly filled out the form and began to expect results.
After some time, we get the answer:
> Dear Client,
>
> it seems
> that you’re answering MAC’s. So please check
> your server again and solve this issue.
Ok ... Considering that the server was turned on for about a minute, we didn’t have time to see anything.
After that, we asked what can now be done with the server, without having network access to it, we were advised to format the server and install a new system by mounting the remote ISO image.
We tried to do it, but we were not technically ready (there wasn’t a small ISO on hand, all were large for the desktop, tried to load X, and were not very suitable for installation via KVM). The story ended around 10 pm, when KVM was chopped off during the installation of the system (the rule in Hetzner - free KVM is given for no more than 2 hours).
In a few days
In a relaxed atmosphere, on Monday morning, they made an application for KVM, got access, installed a small system on sda1 (raid collapsed) via Lara / ISO mount image, sent a fax scan about problem solving, and received an answer that the server is activated. But for some reason he did not ping ...
After re-applying for activation, the site finally stopped, we logged off the network, and the first thing I did was a full tar cjvpf backup of the old main system, download it to myself for experiments and deployed locally.
Locally also could not find the problem. I picked up a separate machine as the default router, turned it on to the one on which the copy was running, looked at the tcpdump traffic at both ends, raised the NAT from the gw address for the grid. No strange packages found.
We checked on the root kit, checked all the packages and files - there is not a single modified MD5, there are no extra processes, and so on.
What was the problem? One can only guess. Perhaps they have the same MAC addresses on the network, maybe something else ... Who else can tell?
In general, even if there were problems with us (which I doubt), it is impossible to find out. And they, for their part, are hardly recognized or will be able to provide a specialist for help.
Conclusions (and by the way, we have many other clients already / still being hosted on Hetzner).
1) Always have a small ISO image of your system at hand, so that at the request of hetzner you can quickly format your server. It is desirable that the main section and services / data are separated.
2) It is highly desirable to stream netflow outside Hetzner, so that you can check the charges later.
3) Always be prepared for the fact that the server can disappear for good (which, incidentally, is the right strategy).
PS Admins tell me that this is normal: first we chop off the problem, then we understand what it is. But if the former occurs promptly, then the proceedings must follow.
Source: https://habr.com/ru/post/160759/
All Articles