A bit about iptables, iproute2 and emulation of network problems
Once I had to do monitoring of packet loss between master and replicas in Zabbix (replication feels bad if the channel is not very good). For this, Zabbix has a built-in icmppingloss parameter, a series of ICMP packets is sent to a remote host and the result is recorded in the monitoring system. And now the parameter is added, the trigger is configured. It would seem that the task was completed, but as they say, "trust, but verify." It remains to verify that the trigger will work when the losses really are. So how to emulate packet loss? This, and not only, will be discussed under the cut.
The first thought that occurred to me is to use iptables. Indeed, a brief search led me to the statistic module - in short, this module processes packets introducing some statistical probability into the process.
In general, the problem was solved using the rule:
Or:
')
As you can see there are 2 options --mode random --probability 0.1 - means that the package will be selected randomly, with a probability of 10%. And --mode nth --every 10 - process every tenth packet. In this way, I achieved packet loss of about 10%, the trigger worked, everything is fine.
It seems that you can stop at this, but quite by chance I found out about such functionality as Network Emulator from the nuclear subsystem Network QoS. Moreover, NetEm's capabilities are much wider than those of the statistic module:
Now about all this in more detail.
Network emulator is supported from 2.6 kernels and at the moment it is present in all modern distributions (we are talking about Linux of course). To check for Network Emulator support in the kernel, check the kernel configuration in / boot (or /proc/config.gz):
As can be seen from the output, support is in the form of a module, so we load it. We also need the tc utility from the iproute package:
If there is nothing, you need to rebuild the kernel. Anyone familiar with the kernel build is a small hint, the Network emulator is here:
If you are unfamiliar with building a kernel, look for articles on building a kernel in the documentation for your distribution.
When everything is ready, you can proceed. I recommend to open a second session in which to launch ping to any node in the local network. Thus, the introduced emulations will be observed quite clearly.
For experiments, we need the tc utility from the iproute2 package. The full syntax is as follows:
1) Packet Delay.
Add 100ms delay for sending packets:
Here we specify the jitter and thus to the already existing delay of 100ms and add some deviation of ± 10ms.
Now we add the correlation, so the delay in sending the next packet will depend on the delay of the previous packet.
2) Distribution of delay.
In the previous examples, we received a more or less uniform distribution of delays over the entire number of sent packets. However, in the real world, network latency is completely uneven. For a more realistic picture, the distribution is used (by default, if you do not explicitly indicate the distribution, then normal is used).
In the example below, we indicate the distribution of pareto, normal and paretonormal are also available - the delay will be calculated using mathematical formulas. In addition, you can create your own distribution tables. In my opinion, this is a rather specific application case, but suddenly someone will be interested.
3) Loss of packets.
That's where it all started, yes ...
Indicates a packet loss of 20%.
Additionally, you can specify the correlation, in this case, the random number generator will be less random and it will be possible to observe bursts in losses:
4) Damage to packages.
Intentional damage to packages, how is it done? With a specified probability, an incorrect bit is written to a random place inside a randomly selected packet. As a result, the checksum does not converge - the packet is discarded. As in the case of losses, you can specify a correlation for the formation of bursts.
5) Duplication of packages.
Duplication of packages is determined in the same way as the loss or corruption of packages. And of course, you can specify the correlation.
6) Reorder packages
In the following example, 30% of the packets will be sent immediately, the rest will be delayed by 100ms.
In the example below, the first 4 packets (gap - 1) will be delayed by 100ms, subsequent packets will be sent immediately with a probability of 25% (+ 50% correlation) or vice versa delayed with a probability of 75%. As soon as the packet is reordered, the iteration is repeated and the next 4 packets are delayed, the rest are sent immediately or delayed with the indicated probability.
Who is too lazy to bother with this case, there is a small demo video .
That's it. Thank you all for your attention, experiment on health.
The first thought that occurred to me is to use iptables. Indeed, a brief search led me to the statistic module - in short, this module processes packets introducing some statistical probability into the process.
In general, the problem was solved using the rule:
iptables -A INPUT -p icmp -s zabbix_ip -m statistic --mode random --probability 0.1 -j DROP Or:
')
iptables -A INPUT -p icmp -s zabbix_ip -m statistic --mode nth --every 10 -j DROP As you can see there are 2 options --mode random --probability 0.1 - means that the package will be selected randomly, with a probability of 10%. And --mode nth --every 10 - process every tenth packet. In this way, I achieved packet loss of about 10%, the trigger worked, everything is fine.
It seems that you can stop at this, but quite by chance I found out about such functionality as Network Emulator from the nuclear subsystem Network QoS. Moreover, NetEm's capabilities are much wider than those of the statistic module:
- delay — add the specified delay to the sent packets on the specified interface;
- Distribution — allows you to specify a delay distribution option;
- reset (drop) - allows you to specify the packet loss;
- corrupt - determines the possibility of packet damage;
- duplicate (duplicate) - allows you to define duplicate packets before they are queued;
- reorder (reorder) - defines the reordering of packets (used in conjunction with the delay);
- Limiting (limit) - limits the effect of the above options on the specified number of packets.
Now about all this in more detail.
Network emulator is supported from 2.6 kernels and at the moment it is present in all modern distributions (we are talking about Linux of course). To check for Network Emulator support in the kernel, check the kernel configuration in / boot (or /proc/config.gz):
# grep NETEM /boot/config-$(uname -r) CONFIG_NET_SCH_NETEM=m As can be seen from the output, support is in the form of a module, so we load it. We also need the tc utility from the iproute package:
# modprobe sch_netem If there is nothing, you need to rebuild the kernel. Anyone familiar with the kernel build is a small hint, the Network emulator is here:
Networking --> Networking Options --> QoS and/or fair queuing --> Network emulator If you are unfamiliar with building a kernel, look for articles on building a kernel in the documentation for your distribution.
When everything is ready, you can proceed. I recommend to open a second session in which to launch ping to any node in the local network. Thus, the introduced emulations will be observed quite clearly.
For experiments, we need the tc utility from the iproute2 package. The full syntax is as follows:
tc qdisc ACTION dev DEVICE add netem OPTIONS ACTION := [ add | change | delete ] OPTIONS := [ LIMIT ] [ DELAY ] [ DROP ] [ CORRUPT ] [ DUPLICATION ] [ REORDER ] LIMIT := limit packets DELAY := delay TIME [ JITTER [ CORRELATION ]]] [ distribution { uniform | normal | pareto | paretonormal } ] DROP := drop PERCENT [ CORRELATION ] CORRUPT := corrupt PERCENT [ CORRELATION ]] DUPLICATION := duplicate PERCENT [ CORRELATION ]] REORDER := reorder PERCENT [ CORRELATION ] [ gap DISTANCE ] 1) Packet Delay.
Add 100ms delay for sending packets:
# tc qdisc add dev eth0 root netem delay 100ms Here we specify the jitter and thus to the already existing delay of 100ms and add some deviation of ± 10ms.
# tc qdisc add dev eth0 root netem delay 100ms 10ms Now we add the correlation, so the delay in sending the next packet will depend on the delay of the previous packet.
# tc qdisc add dev eth0 root netem delay 100ms 10ms 50% 2) Distribution of delay.
In the previous examples, we received a more or less uniform distribution of delays over the entire number of sent packets. However, in the real world, network latency is completely uneven. For a more realistic picture, the distribution is used (by default, if you do not explicitly indicate the distribution, then normal is used).
In the example below, we indicate the distribution of pareto, normal and paretonormal are also available - the delay will be calculated using mathematical formulas. In addition, you can create your own distribution tables. In my opinion, this is a rather specific application case, but suddenly someone will be interested.
# tc qdisc add dev eth0 root netem delay 100ms 10ms distribution pareto 3) Loss of packets.
That's where it all started, yes ...
Indicates a packet loss of 20%.
# tc qdisc add dev eth0 root netem drop 20% Additionally, you can specify the correlation, in this case, the random number generator will be less random and it will be possible to observe bursts in losses:
# tc qdisc add dev eth0 root netem drop 20% 10% 4) Damage to packages.
Intentional damage to packages, how is it done? With a specified probability, an incorrect bit is written to a random place inside a randomly selected packet. As a result, the checksum does not converge - the packet is discarded. As in the case of losses, you can specify a correlation for the formation of bursts.
# tc qdisc add dev eth0 root netem corrupt 20% 5) Duplication of packages.
Duplication of packages is determined in the same way as the loss or corruption of packages. And of course, you can specify the correlation.
# tc qdisc add dev eth0 root netem duplicate 20% 6) Reorder packages
In the following example, 30% of the packets will be sent immediately, the rest will be delayed by 100ms.
# tc qdisc add dev eth0 root netem delay 100ms reorder 30% In the example below, the first 4 packets (gap - 1) will be delayed by 100ms, subsequent packets will be sent immediately with a probability of 25% (+ 50% correlation) or vice versa delayed with a probability of 75%. As soon as the packet is reordered, the iteration is repeated and the next 4 packets are delayed, the rest are sent immediately or delayed with the indicated probability.
# tc qdisc add dev eth0 root netem delay 100ms reorder 25% 50% gap 5 Who is too lazy to bother with this case, there is a small demo video .
That's it. Thank you all for your attention, experiment on health.
Source: https://habr.com/ru/post/237217/
All Articles