"Memory Component Issue", or large-scale defective network equipment
The existence of a problem that many guessed was confirmed.
Cisco announced that an unnamed memory maker for five years (from 2005 to 2010) supplied them with a marriage. The nature of the marriage: the equipment with this memory can save uptime for years without causing any complaints about its work, but it is worth reloading it (for power or even simple reload) - the memory stops working correctly, the device either does not load, or loads and periodically drops. This is due to the degradation of memory chips. According to the vendor, the main problems begin after two years of operation.
Before the rotten tomatoes fly to Cisco, I’m in a hurry to warn you: the memory is standard, many vendors bought it, so a great number of pieces of equipment may be affected. There is evidence of similar problems with Juniper. But only Cisco confessed, despite the inevitable reputational damage. Their financial losses due to this disaster amount to about $ 655 million .
We take seat, we get validol and we look at the list of the affected equipment.
')
Particular partners and a detailed description of the symptoms can be found in Field Notices or directly on the links.
Again, in the risk zone, equipment produced 5–10 years ago and still well-functioning, and failure occurs precisely during a reboot in any way, and not during normal operation.
Replacing the standard, on RMA, the entire piece of iron or memory strip, as soon as it breaks. Apparently, defective memory is far from 100% of the above-mentioned equipment, and even if it is in your piece of hardware it is worth it - it can die not from today's reboot, but after 10 years.
Check by serial number, who is in danger, it is impossible. No I've tried.
Colleagues. I think at this stage everyone understood that many times the approach I had seen “I once bought a Cisco router for big money, it worked for years and will last for many years, the reserve is not needed” is criminal. And even a hot reserve can no longer help. Imagine that the light blinked in the data center, and all your network equipment broke down and needs to be replaced by the very fact of a short-term de-energizing and rebooting. Even a simple scheduled night reboot of an unreserved piece of iron can turn into a convulsive search for a replacement and a long downtime. Evaluate risks, execute service contracts with fast delivery, find in advance or purchase replacement memory, change the hardware itself to a newer one. Assume that after the next reboot, any piece of hardware from the list above (and not only) may not rise, plan the paths of retreat.
Finally, with a minute of silence, one of the many untimely dead memory plates that had previously been faithfully serving as part of 2811 routers.
Cisco announced that an unnamed memory maker for five years (from 2005 to 2010) supplied them with a marriage. The nature of the marriage: the equipment with this memory can save uptime for years without causing any complaints about its work, but it is worth reloading it (for power or even simple reload) - the memory stops working correctly, the device either does not load, or loads and periodically drops. This is due to the degradation of memory chips. According to the vendor, the main problems begin after two years of operation.
Before the rotten tomatoes fly to Cisco, I’m in a hurry to warn you: the memory is standard, many vendors bought it, so a great number of pieces of equipment may be affected. There is evidence of similar problems with Juniper. But only Cisco confessed, despite the inevitable reputational damage. Their financial losses due to this disaster amount to about $ 655 million .
We take seat, we get validol and we look at the list of the affected equipment.
')
Particular partners and a detailed description of the symptoms can be found in Field Notices or directly on the links.
- ACE10, ACE20, and ACE30 Modules
- Fwsm
- ADM & AGM
- SAMI
- Various HWIC, EHWIC, EVM, NME, SM modules
- Different SPA
- Some phones
- Some VCS Codecs
- ONS 15310 and 15454
- ASR 1000
- Routers 7200/7300
- CRS
- Much under 7600 and 6500
- 800 , 1800, 2800, and 3800 Series Routers
- ESR10k
- Sensors IPS-4240 and 4255
- ASA 5505 and 5510-5550 / ASA-SSM (models 5500-X are not mentioned)
- MDS 9000
- Catalyst Express 500
- Nexus 7000
- Catalyst 4500/4900
- Catalyst 3k
- Catalyst 2k
- ME3400 and ME2400
- MGX
- IE3000
- AS5400XM and AS5350
- UC520 / 540
Again, in the risk zone, equipment produced 5–10 years ago and still well-functioning, and failure occurs precisely during a reboot in any way, and not during normal operation.
Replacing the standard, on RMA, the entire piece of iron or memory strip, as soon as it breaks. Apparently, defective memory is far from 100% of the above-mentioned equipment, and even if it is in your piece of hardware it is worth it - it can die not from today's reboot, but after 10 years.
Check by serial number, who is in danger, it is impossible. No I've tried.
Colleagues. I think at this stage everyone understood that many times the approach I had seen “I once bought a Cisco router for big money, it worked for years and will last for many years, the reserve is not needed” is criminal. And even a hot reserve can no longer help. Imagine that the light blinked in the data center, and all your network equipment broke down and needs to be replaced by the very fact of a short-term de-energizing and rebooting. Even a simple scheduled night reboot of an unreserved piece of iron can turn into a convulsive search for a replacement and a long downtime. Evaluate risks, execute service contracts with fast delivery, find in advance or purchase replacement memory, change the hardware itself to a newer one. Assume that after the next reboot, any piece of hardware from the list above (and not only) may not rise, plan the paths of retreat.
Finally, with a minute of silence, one of the many untimely dead memory plates that had previously been faithfully serving as part of 2811 routers.
Hidden text
Source: https://habr.com/ru/post/216287/
All Articles