Linux is under load. High performance shaper
In the last article, we looked at some aspects of tuning a Linux router that is designed to work under high load conditions: Linux under load. Router, NAT-server Now it’s about the shapers.
Limiting the speed of passing traffic (shaping) is a fairly resource-intensive task. Therefore, in the presence of at least several hundred subscribers in the network, and in the presence of large volumes of passing traffic, the question of optimizing the shapers is particularly acute.
Consider the shaper running Linux: this OS has shown the best performance results under high load conditions.
')
Shaper is the discipline of serving packet queues. Discipline can be with classes and without them. With classes - it means that traffic can be “stitched” in accordance with a particular class.
What traffic is shaped by which class - determines the filter.
Simply put, we have two trees: a tree of filters and a tree of classes. Filters spread traffic to classes according to certain criteria. In classes, traffic is prioritized or shaped according to the parameters specified in the classes.
Like any tree, the filter tree becomes too resource-consuming when it reaches a certain threshold.
When a packet from some IP address falls on the filter tree, it begins to be compared with the criteria of each filter. If it matches, the packet is sent to the appropriate class. Those. For each incoming packet, a sequential check is performed to see if each filter in the tree meets the criteria until a match occurs.
For example, for the network on the 24th mask, we will have an average of 128 steps for each packet when searching for the required class.
This is insignificant with small volumes of traffic and with a small number of subscribers. When there are tens of thousands of subscribers, and gigabits go to the Internet, this approach becomes simply impossible - the shaping server will simply not cope with the load.
If the entire tree is a sequence of checks for an IP address, it will be much more efficient to use hashes. A hash is a table of correspondences of some “values” to certain “keys”. In our case, the key is the IP address, and the value is the filter that sends the packet to its class.
Thus, by the key (IP-address) we quickly find the filter for the packet, in 1 step.
Actually, a lot has already been written about the use of hashes of articles - this is not “something military”. You can refer to the source .
To make life easier when building shapers, the Fast U32 hashing filter generator can be such a C program written by a Romanian (?) Sysadmin.
At the input it is given the following parameters:
Detailed examples can be found on the project page above.
Strictly speaking, you do not need to particularly delve into the essence of the work of this program - in this case you will not have to deal with it.
For convenience, I wrote a small script that builds tables of prefixes and classes on the basis of a given configuration, runs the above-mentioned program for building filters with the necessary parameters, builds a ready-made configuration of the shaper from all the available stuff and runs it through tc –b.
All that is required is to specify some configuration parameters in the file.
The folder of the script has the following folders and files:
All script configuration is in the config file. To configure, it is necessary to change the following parameters for yourself:
The rest is not necessary to change.
In the networks file, you can describe the networks for which you want to build shapers. If the file is empty, the shapers will be built for all subscribers.
Subscriber IP addresses are selected from the database in MySQL.
Shapers table:
Table cl_status:
Most likely, it will be easier to adapt the request to select the IP addresses of subscribers for your specific database, rather than vice versa. To do this, correct the request in the 70th line of the shaper.php file.
For the script to work, it is necessary to compile the attached prefixtree.c file.
This is done by the command: gcc prefixtree.c –o prefixtree
I want to note that along with this script a slightly revised version of the prefixtree is attached, adapted for such use.
After you have made the appropriate changes in the config, sorted out the selects in shaper.php, compiled prefixtree.c, made (if necessary) the necessary networks in networks — you can run the script.
Starting should be done as root, and it simply consists in:
/ usr / bin / php –q shaper.php
After this script:
As a result, a shaper will be built and launched on the specified interface according to the specified parameters for the specified IP addresses.
If the subscriber has several IP addresses, one class will be built with the specified speed parameter and all subscriber IP addresses will be sent to this class. Thus, if the subscriber has several IPs, one “channel” of a given speed will be created, which will be divided between his IP.
The resulting configuration of the shaper will be in the data / _classes - this is a ready-made config file that you can feed tc from the batch (-b) options.
All stages of the script are reflected in the logs - the log folder.
Such shaper successfully serves more than 5 thousand subscribers.
Specific bandwidth values ​​and htb parameters in the shaper.php file will need to be corrected in each specific case.
Since there is too much code to bring it all here, you can download the script on its home page.
Limiting the speed of passing traffic (shaping) is a fairly resource-intensive task. Therefore, in the presence of at least several hundred subscribers in the network, and in the presence of large volumes of passing traffic, the question of optimizing the shapers is particularly acute.
Consider the shaper running Linux: this OS has shown the best performance results under high load conditions.
Short introduction
')
Shaper is the discipline of serving packet queues. Discipline can be with classes and without them. With classes - it means that traffic can be “stitched” in accordance with a particular class.
What traffic is shaped by which class - determines the filter.
Simply put, we have two trees: a tree of filters and a tree of classes. Filters spread traffic to classes according to certain criteria. In classes, traffic is prioritized or shaped according to the parameters specified in the classes.
With and without hashes
Like any tree, the filter tree becomes too resource-consuming when it reaches a certain threshold.
When a packet from some IP address falls on the filter tree, it begins to be compared with the criteria of each filter. If it matches, the packet is sent to the appropriate class. Those. For each incoming packet, a sequential check is performed to see if each filter in the tree meets the criteria until a match occurs.
For example, for the network on the 24th mask, we will have an average of 128 steps for each packet when searching for the required class.
This is insignificant with small volumes of traffic and with a small number of subscribers. When there are tens of thousands of subscribers, and gigabits go to the Internet, this approach becomes simply impossible - the shaping server will simply not cope with the load.
If the entire tree is a sequence of checks for an IP address, it will be much more efficient to use hashes. A hash is a table of correspondences of some “values” to certain “keys”. In our case, the key is the IP address, and the value is the filter that sends the packet to its class.
Thus, by the key (IP-address) we quickly find the filter for the packet, in 1 step.
Actually, a lot has already been written about the use of hashes of articles - this is not “something military”. You can refer to the source .
The script to build the shaper
To make life easier when building shapers, the Fast U32 hashing filter generator can be such a C program written by a Romanian (?) Sysadmin.
At the input it is given the following parameters:
- prefix.in - a list of prefixes and corresponding classes in the format <prefix> <class>
- u32filters.out - output file, filters will be saved here
- interface - the name of the interface on which the shaper will be built
- src / dst - flow direction (inbound or outbound)
- batch - if this parameter is specified, the output file will be generated suitable for running tc –b
Detailed examples can be found on the project page above.
Strictly speaking, you do not need to particularly delve into the essence of the work of this program - in this case you will not have to deal with it.
For convenience, I wrote a small script that builds tables of prefixes and classes on the basis of a given configuration, runs the above-mentioned program for building filters with the necessary parameters, builds a ready-made configuration of the shaper from all the available stuff and runs it through tc –b.
All that is required is to specify some configuration parameters in the file.
Script configuration
The folder of the script has the following folders and files:
- data - the intermediate results of the script will be located here, as well as the final configuration of the shaper: _classes - ready configuration of the shaper, _filters - filters with hashes, _prefixes - table of correspondences of prefixes to classes, _speeds - correspondence of classes to speeds
- lib - necessary for the library to work
- log - event logging during script operation
- pid - here lies the pid of the process to prevent simultaneous running of multiple copies of the script
- config - the main script configuration file
- networks - the list of networks to which it is necessary to build shapers.
- prefixtree.c - source filter builder with hashes
- shaper.php - the shaper script itself
All script configuration is in the config file. To configure, it is necessary to change the following parameters for yourself:
- At the beginning of the config, the parameters for connecting to the database for retrieving IP addresses and speed parameters
- DEV - the interface on which to build the shaper
- DIR - traffic flow direction in relation to subscribers: can be in or out (incoming and outgoing respectively)
The rest is not necessary to change.
In the networks file, you can describe the networks for which you want to build shapers. If the file is empty, the shapers will be built for all subscribers.
Subscriber IP addresses are selected from the database in MySQL.
Shapers table:
- shaper_id - unique caller ID
- id - the shaper ID
Table cl_status:
- ip - subscriber IP address
- shaper_id - unique caller ID
- status - subscriber status (3 - on)
- sin - incoming speed
- sout - outgoing speed
Most likely, it will be easier to adapt the request to select the IP addresses of subscribers for your specific database, rather than vice versa. To do this, correct the request in the 70th line of the shaper.php file.
Compiling prefixrtee.c
For the script to work, it is necessary to compile the attached prefixtree.c file.
This is done by the command: gcc prefixtree.c –o prefixtree
I want to note that along with this script a slightly revised version of the prefixtree is attached, adapted for such use.
Running script
After you have made the appropriate changes in the config, sorted out the selects in shaper.php, compiled prefixtree.c, made (if necessary) the necessary networks in networks — you can run the script.
Starting should be done as root, and it simply consists in:
/ usr / bin / php –q shaper.php
After this script:
- read config
- read networks
- make the necessary sample from the database for each IP (based on networks)
- will create a data / _prefixes file
- will create a data / _speeds file
- runs the prefixtree for execution, which will create the data / _filters file
- will create a data / _classes file
- will launch tc –b data / _classes
As a result, a shaper will be built and launched on the specified interface according to the specified parameters for the specified IP addresses.
If the subscriber has several IP addresses, one class will be built with the specified speed parameter and all subscriber IP addresses will be sent to this class. Thus, if the subscriber has several IPs, one “channel” of a given speed will be created, which will be divided between his IP.
The resulting configuration of the shaper will be in the data / _classes - this is a ready-made config file that you can feed tc from the batch (-b) options.
Logs
All stages of the script are reflected in the logs - the log folder.
Statistics
Such shaper successfully serves more than 5 thousand subscribers.
Notes
Specific bandwidth values ​​and htb parameters in the shaper.php file will need to be corrected in each specific case.
Download script
Since there is too much code to bring it all here, you can download the script on its home page.
Source: https://habr.com/ru/post/110616/
All Articles