Leap second caused some servers to hang on Linux
User Bron Gondwana on ServerFault notes that, starting in the morning of June 30th, some of his servers on Debian Squeeze began to hang without giving any signs of life.
From one of the servers we managed to pull out such a dump:
The rest of the car just silently went into deep thought and did not return.
')
The solution was to temporarily disable ntpd on all machines.
It's a shame that the bug has been known since at least March 15th .
Leap second day has already passed, so the details on the fix, I do not cite, see them in the thread on the link above. However, this information may be useful to someone when investigating yesterday's machine behavior. The thread also contains a very detailed description of the causes of the incident.
PS In the same thread they say that similar problems were observed in Java applications.
From one of the servers we managed to pull out such a dump:
[3161000.864001] BUG: spinlock lockup on CPU#1, ntpd/3358
[3161000.864001] lock: ffff88083fc0d740, .magic: dead4ead, .owner: imapd/24737, .owner_cpu: 0The rest of the car just silently went into deep thought and did not return.
')
The solution was to temporarily disable ntpd on all machines.
It's a shame that the bug has been known since at least March 15th .
Leap second day has already passed, so the details on the fix, I do not cite, see them in the thread on the link above. However, this information may be useful to someone when investigating yesterday's machine behavior. The thread also contains a very detailed description of the causes of the incident.
PS In the same thread they say that similar problems were observed in Java applications.
Source: https://habr.com/ru/post/146863/
All Articles