The “perfect storm” and how it is treated
Hi, Habr!
Broadcast storm is a nightmare for networkers when in a matter of seconds the transmission of useful traffic is paralyzed in the entire data center network. How this happens and what should have been thought before - in our today's post.
Where did it go
')
Initially, there was Ethernet, and there was no routing in it, and now it doesn’t exist either, and therefore the switch has to send data packets to all ports - well, for sure. From the addressee the answer to a certain port comes, the switch remembers the correspondence [port - addressee] and the next “parcel” is sent not anyhow somewhere, but by memorable, so to speak, places.
If there is no answer, and at the moment of sending unknown unicast, a couple of switches are locked into a ring, the package is returned to the “sender”, which, of course, throws it back to all ports, because what else can it do? The "unattended" data packet is again returned and flies away again. Every next round of “looped” broadcast flood is accompanied by an exponential increase in the number of packets in a network segment. Very soon, this avalanche "clogs" the bandwidth of the ports, in the background the overloaded switch processors "boil" beautifully - and your network becomes a monument to itself.
The cause of a storm can be either a hacker attack, or a misfire of your engineer when setting up equipment - or a protocol failure. In other words, no one is insured.
There was such a case
In 2009, a Broadcast storm occurred in the network of one of our clients, which instantly spread to us, paralyzing the work of the entire data transmission network of the company. Mail, telephony and the Internet fell off, the monitoring system "went crazy" - and it became impossible to even locate the "original source". A full reset of the switch did not help. There was nothing left for us to do but to consistently turn off ALL ports, client and own ports ... Such is Black Monday.
As it turned out later, one of the client’s employees confused the equipment ports - and looped his topology at the level of the brodcast domain. It happens.
It is clear that in the risk group here, first of all, commercial data centers: a redundant connection for each client in itself already means redundancy of connections between switches. However, I would not recommend networkers to corporate data centers to relax either: it’s possible to close a pair of switches over each other by absent-mindedness in the server room.
The good news is that with proper preparation you can get off with a little fright where you would otherwise get a simple service.
Where to begin
Start with network segmentation by VLAN : when the network is divided into small isolated segments (fault domain), the storm that covers one of the sections is limited to this area. Well, ideally. In fact, a powerful storm, alas, is capable of throwing packets into so-called independent virtual networks, too.
In an amicable way, you need to abandon the "looped-in" VLANs both within your own infrastructure and when clients connect to the network (if we are talking about a commercial data center). For example, use the FHRP protocols and the U-shaped topology at the access level.
U-shaped topology avoids loopback
In some cases, however, with all the desire, there is no way out of the "looping". Let's say you need to deploy a fault-tolerant (2N) infrastructure for a customer with a connection to our cloud. Client VLANs here have to be “forwarded” within the cloud infrastructure between all the ESXi virtualization cluster hosts — that is, the solution itself implies the complete duplication of all network elements.
We look at the picture - we see the ring.
Ring topology occurs with full redundancy of communication channels and customer infrastructure
What to do if you are “Lord of the Rings"
You can do different things, and each option, as usual, has its own advantages and costs.
Here, for example, the RSTP protocol (modification SpanningTreeProtocol) is able to quickly - within 6 seconds - find and “break” broadcasted loops. He finds them through the exchange of BPDU (Bridge Protocol Data Unit) messages between the switches, and “breaks” the blocking of backup links. In case of problems with the primary channel, the RTSP rebuilds the topology using the backup port.
Good in general, but there is a nuance. One RSTP process is allocated for each VLAN, while the number of processes, unlike VLANs, is very limited, and with a sharp increase in the number of VLANs within one process grid, RSTP may simply not be enough. That is, for the corporate data center will go, and for the commercial - with an ever-growing number of clients (VLAN) - is not very.
In this case, there is MSTP - an improved and updated edition of RSTP. It is able to combine several VLANs into one STP process (instance), which in a good sense of the word affects the scalability of the network: the “ceiling” here is 4,096 clients (the maximum number of VLANs). MSTP also allows you to manage traffic by distributing MST processes between the main link and the backup link, and allows you to unload the “plugged” switches if necessary. However, you need to be able to work with MST, that is, this is plus at least one expensive nerd in the state (which is not available to everyone).
The RSTP and MST protocols "break" the loop, blocking traffic on one of the channels
From the proven MSTP alternatives, we can advise Cisco FlexLinks , which we use when there is one switch or stack on the client side under a single control. FlexLinks is able to reserve switch links without using STP, “assigning” in each pair of ports the primary and backup. It is used at the access level when connecting equipment from different companies on the principle of Looped Triangle in a commercial data center. A very simple tool in terms of settings, which is nice in itself, and allows you to count on greater stability of the service (as compared with, for example, STP). You will love FlexLinks for instantly switching to backup links and load balancing on VLANs - and then, perhaps, fall out of love for the opportunity to apply it exclusively in the Looped Triangle topology.
In the Looped Triangle topology, you can instantly switch traffic between channels in the event of a failure.
Now we digress from the technology. Do you love economic efficiency just as I love her? Then you will be interested to know that both MST \ RSTP and FlexLinks, blocking backup links, virtually exclude half of the ports from the traffic cycle in nature.
But solutions that do not do this: Cisco VSS (Virtual Switch System), Nexus vPC (virtual port-channel), Juniper virtual router and other mLAG-like (multichassis link aggregation) technologies. They are good because they use all available links, combining them into one logical channel EtherChannel . It turns out a kind of switching cluster in which the control module of one of the switches (Control-plane) “drives” all the links of the cluster (Data-Plane). In the event of failure of the current Control-plane, its authority is automatically transferred to the “survivor”. We use Cisco Nexus vPC for load balancing between customer links that have one switch or stack. If, on the client side, two separate switches connected by a common VLAN are added to the STP scheme.
Combining links into one logical channel solves the problem with storms and does not affect performance
If you have clouds
Virtualization, disaster-tolerant cloud services distributed between data centers, and other cluster solutions - all this requires a slightly different approach to the organization of a Layer 2 network. We remove STP on the mezzanine - we get TRILL .
TRILL uses a routing mechanism at the Ethernet level and builds a loop-free path for broadband traffic itself, thereby preventing the occurrence of storms. Well, isn't it a miracle? :) Another TRILL allows you to evenly distribute the load between the links (up to 16 links), combine distributed data centers into a single L2 network and manage traffic flexibly. TRILL is a generally accepted standard that vendor options quickly emerged: FabricPath from Cisco (which we use) and VCS from Brocade. Juniper developed its own Qfabric technology, which allows to create a single Ethernet factory.
Control shot
What protocol will give you 100% storm protection? That's right, no. Therefore, you may be interested in the following two tools:
• Storm-control
Allows you to set per second “quota” on the number of broadcasted packets passing through one port. Anything beyond the “quota” is discarded, and thus the load is controlled. Some nuance is that Storm-control does not distinguish the useful traffic from garbage.
• Control — plane policing (CoPP)
A sort of storm-control for a switch processor. With a broadcast storm, among other things, the number of ARP requests increases dramatically. When this number rolls over, the processor is loaded 100% - and the network, of course, tells you “goodbye”. CoPP can “dose” the number of ARP requests and thus control the load on the processor. It copes well with Broadcast storms from traffic exchange points, and with various DDoS attacks - checked.
How to build a death proof network
So, which of the possible options we have tested on ourselves and use, depending on the input:
1. U, V and U-shaped topology + RSTP (MST) + storm control + CoPP.
The basic set, first of all, for a commercial data center, in which you have to connect a large number of external (uncontrollable) networks to your own network - and therefore it is highly desirable to prevent the emergence of "loops" in general.
If U, V, and U-shaped topologies are not your case, the “abbreviated” version of RSTP (MST) + storm control + CoPP will also work.
2. If there is a task to maximize the capabilities of the equipment and channels, take a closer look at the option mLAG (VSS, vPC) + storm control + CoPP .
3. If you already have Cisco or Juniper equipment and there are no topological contraindications, try the Flex Links / RTG + storm control + CoPP combination.
4. If you have a complex case with distributed platforms and other virtualization and fault tolerance refinements, your version is TRILL + storm control + CoPP.
5. If you do not know what your case is, we can talk about it :).
The main thing is to start doing at least something now, even if you sincerely think that a broadcasted storm is what happens to others. In reality, storms "cover" the network of very different scales, and ridiculous mistakes are made even by people who, as they say, twenty years in the arts. "And let it inspire you to the feat" (c).
Broadcast storm is a nightmare for networkers when in a matter of seconds the transmission of useful traffic is paralyzed in the entire data center network. How this happens and what should have been thought before - in our today's post.
Where did it go
')
Initially, there was Ethernet, and there was no routing in it, and now it doesn’t exist either, and therefore the switch has to send data packets to all ports - well, for sure. From the addressee the answer to a certain port comes, the switch remembers the correspondence [port - addressee] and the next “parcel” is sent not anyhow somewhere, but by memorable, so to speak, places.
If there is no answer, and at the moment of sending unknown unicast, a couple of switches are locked into a ring, the package is returned to the “sender”, which, of course, throws it back to all ports, because what else can it do? The "unattended" data packet is again returned and flies away again. Every next round of “looped” broadcast flood is accompanied by an exponential increase in the number of packets in a network segment. Very soon, this avalanche "clogs" the bandwidth of the ports, in the background the overloaded switch processors "boil" beautifully - and your network becomes a monument to itself.
The cause of a storm can be either a hacker attack, or a misfire of your engineer when setting up equipment - or a protocol failure. In other words, no one is insured.
There was such a case
In 2009, a Broadcast storm occurred in the network of one of our clients, which instantly spread to us, paralyzing the work of the entire data transmission network of the company. Mail, telephony and the Internet fell off, the monitoring system "went crazy" - and it became impossible to even locate the "original source". A full reset of the switch did not help. There was nothing left for us to do but to consistently turn off ALL ports, client and own ports ... Such is Black Monday.
As it turned out later, one of the client’s employees confused the equipment ports - and looped his topology at the level of the brodcast domain. It happens.
It is clear that in the risk group here, first of all, commercial data centers: a redundant connection for each client in itself already means redundancy of connections between switches. However, I would not recommend networkers to corporate data centers to relax either: it’s possible to close a pair of switches over each other by absent-mindedness in the server room.
The good news is that with proper preparation you can get off with a little fright where you would otherwise get a simple service.
Where to begin
Start with network segmentation by VLAN : when the network is divided into small isolated segments (fault domain), the storm that covers one of the sections is limited to this area. Well, ideally. In fact, a powerful storm, alas, is capable of throwing packets into so-called independent virtual networks, too.
In an amicable way, you need to abandon the "looped-in" VLANs both within your own infrastructure and when clients connect to the network (if we are talking about a commercial data center). For example, use the FHRP protocols and the U-shaped topology at the access level.
U-shaped topology avoids loopback
In some cases, however, with all the desire, there is no way out of the "looping". Let's say you need to deploy a fault-tolerant (2N) infrastructure for a customer with a connection to our cloud. Client VLANs here have to be “forwarded” within the cloud infrastructure between all the ESXi virtualization cluster hosts — that is, the solution itself implies the complete duplication of all network elements.
We look at the picture - we see the ring.
Ring topology occurs with full redundancy of communication channels and customer infrastructure
What to do if you are “Lord of the Rings"
You can do different things, and each option, as usual, has its own advantages and costs.
Here, for example, the RSTP protocol (modification SpanningTreeProtocol) is able to quickly - within 6 seconds - find and “break” broadcasted loops. He finds them through the exchange of BPDU (Bridge Protocol Data Unit) messages between the switches, and “breaks” the blocking of backup links. In case of problems with the primary channel, the RTSP rebuilds the topology using the backup port.
Good in general, but there is a nuance. One RSTP process is allocated for each VLAN, while the number of processes, unlike VLANs, is very limited, and with a sharp increase in the number of VLANs within one process grid, RSTP may simply not be enough. That is, for the corporate data center will go, and for the commercial - with an ever-growing number of clients (VLAN) - is not very.
In this case, there is MSTP - an improved and updated edition of RSTP. It is able to combine several VLANs into one STP process (instance), which in a good sense of the word affects the scalability of the network: the “ceiling” here is 4,096 clients (the maximum number of VLANs). MSTP also allows you to manage traffic by distributing MST processes between the main link and the backup link, and allows you to unload the “plugged” switches if necessary. However, you need to be able to work with MST, that is, this is plus at least one expensive nerd in the state (which is not available to everyone).
The RSTP and MST protocols "break" the loop, blocking traffic on one of the channels
From the proven MSTP alternatives, we can advise Cisco FlexLinks , which we use when there is one switch or stack on the client side under a single control. FlexLinks is able to reserve switch links without using STP, “assigning” in each pair of ports the primary and backup. It is used at the access level when connecting equipment from different companies on the principle of Looped Triangle in a commercial data center. A very simple tool in terms of settings, which is nice in itself, and allows you to count on greater stability of the service (as compared with, for example, STP). You will love FlexLinks for instantly switching to backup links and load balancing on VLANs - and then, perhaps, fall out of love for the opportunity to apply it exclusively in the Looped Triangle topology.
In the Looped Triangle topology, you can instantly switch traffic between channels in the event of a failure.
Now we digress from the technology. Do you love economic efficiency just as I love her? Then you will be interested to know that both MST \ RSTP and FlexLinks, blocking backup links, virtually exclude half of the ports from the traffic cycle in nature.
But solutions that do not do this: Cisco VSS (Virtual Switch System), Nexus vPC (virtual port-channel), Juniper virtual router and other mLAG-like (multichassis link aggregation) technologies. They are good because they use all available links, combining them into one logical channel EtherChannel . It turns out a kind of switching cluster in which the control module of one of the switches (Control-plane) “drives” all the links of the cluster (Data-Plane). In the event of failure of the current Control-plane, its authority is automatically transferred to the “survivor”. We use Cisco Nexus vPC for load balancing between customer links that have one switch or stack. If, on the client side, two separate switches connected by a common VLAN are added to the STP scheme.
Combining links into one logical channel solves the problem with storms and does not affect performance
If you have clouds
Virtualization, disaster-tolerant cloud services distributed between data centers, and other cluster solutions - all this requires a slightly different approach to the organization of a Layer 2 network. We remove STP on the mezzanine - we get TRILL .
TRILL uses a routing mechanism at the Ethernet level and builds a loop-free path for broadband traffic itself, thereby preventing the occurrence of storms. Well, isn't it a miracle? :) Another TRILL allows you to evenly distribute the load between the links (up to 16 links), combine distributed data centers into a single L2 network and manage traffic flexibly. TRILL is a generally accepted standard that vendor options quickly emerged: FabricPath from Cisco (which we use) and VCS from Brocade. Juniper developed its own Qfabric technology, which allows to create a single Ethernet factory.
Control shot
What protocol will give you 100% storm protection? That's right, no. Therefore, you may be interested in the following two tools:
• Storm-control
Allows you to set per second “quota” on the number of broadcasted packets passing through one port. Anything beyond the “quota” is discarded, and thus the load is controlled. Some nuance is that Storm-control does not distinguish the useful traffic from garbage.
• Control — plane policing (CoPP)
A sort of storm-control for a switch processor. With a broadcast storm, among other things, the number of ARP requests increases dramatically. When this number rolls over, the processor is loaded 100% - and the network, of course, tells you “goodbye”. CoPP can “dose” the number of ARP requests and thus control the load on the processor. It copes well with Broadcast storms from traffic exchange points, and with various DDoS attacks - checked.
How to build a death proof network
So, which of the possible options we have tested on ourselves and use, depending on the input:
1. U, V and U-shaped topology + RSTP (MST) + storm control + CoPP.
The basic set, first of all, for a commercial data center, in which you have to connect a large number of external (uncontrollable) networks to your own network - and therefore it is highly desirable to prevent the emergence of "loops" in general.
If U, V, and U-shaped topologies are not your case, the “abbreviated” version of RSTP (MST) + storm control + CoPP will also work.
2. If there is a task to maximize the capabilities of the equipment and channels, take a closer look at the option mLAG (VSS, vPC) + storm control + CoPP .
3. If you already have Cisco or Juniper equipment and there are no topological contraindications, try the Flex Links / RTG + storm control + CoPP combination.
4. If you have a complex case with distributed platforms and other virtualization and fault tolerance refinements, your version is TRILL + storm control + CoPP.
5. If you do not know what your case is, we can talk about it :).
The main thing is to start doing at least something now, even if you sincerely think that a broadcasted storm is what happens to others. In reality, storms "cover" the network of very different scales, and ridiculous mistakes are made even by people who, as they say, twenty years in the arts. "And let it inspire you to the feat" (c).
Source: https://habr.com/ru/post/253609/
All Articles