HP iLO 2 is vulnerable to Heartbleed - defend and upgrade!
Heartbleed is not terrible for iLO 2 in terms of penetration into the system, however, some scanners of this vulnerability (or an attempt to exploit it) tightly cut down the iLO 2 interface from Hewlett Packard servers.
On Friday, April 11, I thus lost access to three hundred servers with 2nd generation iLO modules (iLO interfaces are publicly available, since these are hosting servers).
The bulk of the servers are blades, however, several DL servers also fell into the hands of the hand.
Neither iLO 3, nor SuperMicro IPMI, nor anything else on the neighboring IP addresses were affected.
')
Symptoms:
1) the latest version of iLO 2 v.2.23 is vulnerable;
2) even PING to iLO 2 disappears;
3) the cart (Onboard Administrator) stops seeing the server, does not know its type, does not know its current state, allocates power for it with a margin (attention! This moment may entail shutting down a number of servers if Dynamic Power Capping is on);
4) oddly enough, iLO 2 on DL servers continued to respond to PING, but this is the maximum that could be achieved from them;
5) restarting the server using OS means will not lead it to boot, because (apparently) it gets stuck on the start diagnosis of components;
6) a regular shutdown does not help - the server is of course muffled, but iLO 2 does not come to life;
7) it is impossible to reset or communicate (via hponcfg for example) from the inside of a working server (from OS), the iLO 2 module is not possible, the module is not responding;
8) the situation is treated ONLY by completely disconnecting the power from the DL servers (via controlled PDUs or by hands), or slightly more conveniently (but in fact with the same effect) for the blade servers using the reset server XX command through the Onboard Administrator;
9) Cold Restart does not help either.
The most unpleasant thing is that in the case of high-loaded blade baskets, the OA modules, having lost contact with the servers, will not know their nutritional needs, and will begin to allocate to them obviously large values (2-3 times higher than the actual consumption). This will lead to the fact that OA considers that there is not enough power and can cut down some servers (or prevent it from starting). At least, this is true if you have dynamic power distribution control enabled.
HP did not immediately recognize the problem, and initially they reported only the fact that only the Onboard Administrator (ver.> = 4.11) was vulnerable to Heartbleed, and it was said in c04239413 that iLO was OK . Even used the expression NOT Impacted by "HeartBleed" .
Yes, in terms of penetration into the iLO system is not vulnerable, but, unfortunately, iLO 2 turned out to be just Impacted, and how!
There is no official HP position on cutting iLO 2 as a result of a Heartbleed scan.
Fortunately, by Monday, an employee of HP Oscar A. Perez made us happy with the beta version of firmware 2.25 and laid it out for tests .
You can download from here .
After hard reset affected servers and firmware upgrades to 2.25 beta, while the flight is normal.
If your iLO 2 is publicly accessible from the Internet, then do not wait until you lose connection to the servers and cannot manage it in a critical situation (or simply after a planned restart) - upgrade to at least 2.25 beta.
As an option - to temporarily restrict public access to iLO 2 before the release and application of the official release of the patched firmware.
On Friday, April 11, I thus lost access to three hundred servers with 2nd generation iLO modules (iLO interfaces are publicly available, since these are hosting servers).
The bulk of the servers are blades, however, several DL servers also fell into the hands of the hand.
Neither iLO 3, nor SuperMicro IPMI, nor anything else on the neighboring IP addresses were affected.
')
Symptoms:
1) the latest version of iLO 2 v.2.23 is vulnerable;
2) even PING to iLO 2 disappears;
3) the cart (Onboard Administrator) stops seeing the server, does not know its type, does not know its current state, allocates power for it with a margin (attention! This moment may entail shutting down a number of servers if Dynamic Power Capping is on);
4) oddly enough, iLO 2 on DL servers continued to respond to PING, but this is the maximum that could be achieved from them;
5) restarting the server using OS means will not lead it to boot, because (apparently) it gets stuck on the start diagnosis of components;
6) a regular shutdown does not help - the server is of course muffled, but iLO 2 does not come to life;
7) it is impossible to reset or communicate (via hponcfg for example) from the inside of a working server (from OS), the iLO 2 module is not possible, the module is not responding;
8) the situation is treated ONLY by completely disconnecting the power from the DL servers (via controlled PDUs or by hands), or slightly more conveniently (but in fact with the same effect) for the blade servers using the reset server XX command through the Onboard Administrator;
9) Cold Restart does not help either.
The most unpleasant thing is that in the case of high-loaded blade baskets, the OA modules, having lost contact with the servers, will not know their nutritional needs, and will begin to allocate to them obviously large values (2-3 times higher than the actual consumption). This will lead to the fact that OA considers that there is not enough power and can cut down some servers (or prevent it from starting). At least, this is true if you have dynamic power distribution control enabled.
HP did not immediately recognize the problem, and initially they reported only the fact that only the Onboard Administrator (ver.> = 4.11) was vulnerable to Heartbleed, and it was said in c04239413 that iLO was OK . Even used the expression NOT Impacted by "HeartBleed" .
Yes, in terms of penetration into the iLO system is not vulnerable, but, unfortunately, iLO 2 turned out to be just Impacted, and how!
There is no official HP position on cutting iLO 2 as a result of a Heartbleed scan.
Fortunately, by Monday, an employee of HP Oscar A. Perez made us happy with the beta version of firmware 2.25 and laid it out for tests .
You can download from here .
After hard reset affected servers and firmware upgrades to 2.25 beta, while the flight is normal.
If your iLO 2 is publicly accessible from the Internet, then do not wait until you lose connection to the servers and cannot manage it in a critical situation (or simply after a planned restart) - upgrade to at least 2.25 beta.
As an option - to temporarily restrict public access to iLO 2 before the release and application of the official release of the patched firmware.
Source: https://habr.com/ru/post/219453/
All Articles