Limit inbound and outbound traffic to Linux

In this article I want to tell you how I built a system for restricting incoming and outgoing traffic in Linux.

As well as traffic accounting, limiting network bandwidth is a very important task, although the first one fades into the background every year, shaping traffic remains a necessary task for each system / network administrator.



What are the ways to limit traffic?



In order to answer this question, you need to decide why this traffic should be restricted at all.

Taking as a basis my network from about 50 workstations that access the Internet through a gateway, running Ubuntu OS and some of the users use local resources on this server via the SMB protocol.

My goal is to limit users to the speed of data transfer to the Internet with a fair division of bandwidth between them.



Based on my tasks, the following methods can be used to limit bandwidth:

1. Restriction using Squid proxy.

This method allows quite flexible control over all www, ftp user traffic with the possibility of flexible transmission rate limiting.

2. Using traffic control from iproute2.

Very flexible and optimal method of restricting traffic, but does not provide control over WWW traffic, as in the previous method.

3. Of course, it is possible to limit the speed by using the –m limit module for iptables - but I consider this unacceptable.

')

In general, I decided to dwell on the method of limiting traffic using the iproute2 package.



What, where and how?



As already mentioned, I use the server: OS Ubuntu 10.04, kernel 2.6.32-30. In the server 3 interfaces: eth0 - internal network, eth1 - provider 1, eth2 - provider 2.



The task: to limit the speed of incoming and outgoing traffic of users with the prioritization of traffic by classes, based on certain conditions. Local traffic is not limited.



A bit of theory



Imagine a situation when a user has established a connection with youtube.com and is watching some video in HD quality. The bulk of the traffic goes from the server, in this case, youtube.com to the user. Given that all traffic passes through our gateway, we can affect the speed of transmission of this traffic by installing a traffic shaper on the internal network interface.

A similar situation occurs when a user uploads a photo report about a vacation, consisting of 300 photos in the resolution of 5000x3500 pixels to some photo storage service on the Internet.



Naturally, in the absence of a traffic restriction system, this user will occupy the entire channel and other users will not be provided with the normal speed of work with the Internet. But we can not limit the speed of sending data by the user on the external interface of the server, because NAT is used for user access to the Internet, and considering that traffic shaping is performed after address translation, then there will be no more packets with internal network addresses on the external interface of the server.



To solve the problem of restricting outgoing traffic from the client, I used the IFB device, to which I redirected all outgoing traffic from the client.



In traffic management theory, we can limit only outgoing traffic. Consequently, the traffic that is directed to the internal network user will be outgoing relative to the eth0 internal interface, and the traffic sent from the internal network user is outgoing relative to the external eth1 interface.



Based on the above, I restricted the incoming traffic to the user on the internal network interface - eth0, and outgoing traffic from the user - on the virtual ifb0 interface.



In order for the user to capture all the bandwidth limited to him at the gateway, to download any large amount of data and at the same time he could use ssh normally and ping for him - I used traffic prioritization.



I set the following traffic priorities:

1. icmp
2. udp, ssh
3. tcp sport 80
4. other unclassified traffic

The lower the parameter, the higher the priority of traffic.



Disciplines, classes, filters



As I have already noted, incoming traffic to users will be limited on the interface eth0, and outgoing from users - on the virtual interface ifb0.



To initialize the ifb0 interface, you must first load the interface control module:

/sbin/modprobe ifb

After successful loading of the module, you need to enable the interface:

/sbin/ip link set dev ifb0 up

Then, after the interface is raised, you need to organize the redirection of all outgoing traffic from users to this interface:

/sbin/tc qdisc add dev eth0 ingress

/sbin/tc filter add dev eth0 parent ffff: protocol ip u32 match u32 0 0 action mirred egress redirect dev ifb0

Now you can safely start building classes and filters for incoming traffic to users on the eth0 interface, and outgoing traffic - on the ifb0 interface.



The following principle is used to limit traffic:

1. A so-called root queue handler is created on the interface.
2. Attached to this discipline is a class that will gain information about the maximum data throughput that will fall into this class.
3. A filter is added that, by using certain parameters, assigns each package to a particular class.

Classes can be nested. That is, if class 1: describes the maximum bandwidth of 1Mbit, then class 1: 1, which is its subclass, cannot exceed the speed limit of its parent.



We limit the traffic entering to users



All traffic manipulations will be carried out on the interface eth0.



First, create a root queue handler on the interface:

/sbin/tc qdisc add dev eth0 root handle 1: htb default 900



Thus, we tied the root queue handler to interface eth0, assigned it the number 1: and indicated the use of the HTB scheduler with sending all unclassified traffic to the class with number 900.



Then we create a child class 1: 1 with a channel width equal to the interface speed:

/sbin/tc class add dev eth0 parent 1: classid 1:1 htb rate 100Mbit burst 15k

All subsequent classes will be subclasses of the class we just created. This gives us a more accurate prioritization and processing of data rate.



Create a class for local traffic, the destination address or source address of which will be the internal server address. This is necessary for the convenience of using server resources such as SSH, SMB, FTP, WWW and so on. The speed described by the class is 50Mbit, but in case the flow rate of the parent class is not less than 100Mbit, then we allow using 80Mbit as the maximum data transfer rate.

/sbin/tc class add dev eth0 parent 1:1 classid 1:10 htb rate 50Mbit ceil 80Mbit burst 15k



Next, create a class whose speed will be equal to the bandwidth provided by the provider. In my case, it is 15Mbit.

/sbin/tc class add dev eth0 parent 1:1 classid 1:100 htb rate 15Mbit burst 15k



Even if the provider provides greater speed, for example 18Mbit, I recommend reducing this speed for the shaper by 1-2 Mbit for a more “soft" traffic restriction.

Next, create a class in which all data packets will be sent that do not fall into any of the previously created classes.

/sbin/tc class add dev eth0 parent 1:1 classid 1:900 htb rate 56Kbit ceil 128Kbit



For each user, I created a separate subclass, with dedicated bandwidth, and then created subclasses of this class to prioritize traffic:

/sbin/tc class add dev eth0 parent 1:100 classid 1:101 htb rate 4Mbit ceil 6Mbit



With this command, we indicated the creation of a class with the number 1: 101, which is a subclass of the class with the number 1: 100, and indicated the bandwidth of the class at 4Mbit, and in case of free bandwidth of the parent class, allow the maximum data flow through the class at a speed of 6Mbit.



Next, create subclasses to prioritize traffic:

# PRIO 1 -> icmp traffic -

/sbin/tc class add dev eth0 parent 1:101 classid 1:102 htb rate 33kbit ceil 6Mbit prio 1

# PRIO 2 -> udp, ssh

/sbin/tc class add dev eth0 parent 1:101 classid 1:103 htb rate 33kbit ceil 6Mbit prio 2

# PRIO 3 -> tcp sport 80 – WWW

/sbin/tc class add dev eth0 parent 1:101 classid 1:104 htb rate 33kbit ceil 6Mbit prio 3

# PRIO 4 -> unclassified traffic – , ,

/sbin/tc class add dev eth0 parent 1:101 classid 1:105 htb rate 33kbit ceil 6Mbit prio 4



After creating classes, it's time to create filters that will classify traffic according to certain criteria.



There are several ways to classify traffic.

The most convenient ones are the u32 classifiers , which allow classifying packets based on the destination or sender address, the protocol used, the port number, and so on, and classifiers based on iptables tags. To use the latter, you must first mark the packets with iptables in the PREROUTING chain, based on any conditions, and then use tc to send packets with the appropriate label to the required classes.



I preferred to use the u32 classifier.



We assign icmp traffic the lowest priority and send it to class 1: 102

/sbin/tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip dst 192.168.10.78 \

match ip protocol 1 0xff flowid 1:102



UDP and SSH traffic is sent to class 1: 103

/sbin/tc filter add dev eth0 protocol ip parent 1:0 prio 2 u32 match ip dst 192.168.10.78 \

match ip protocol 17 0xff flowid 1:103

/sbin/tc filter add dev eth0 protocol ip parent 1:0 prio 2 u32 match ip dst 192.168.10.78 \

match ip protocol 6 0xff match ip sport 22 0xffff flowid 1:103



WWW traffic coming from tcp port 80 is sent to class 1: 104

/sbin/tc filter add dev eth0 protocol ip parent 1:0 prio 3 u32 match ip dst 192.168.10.78 \

match ip protocol 6 0xff match ip sport 80 0xffff flowid 1:104



Traffic that does not meet any of the conditions is sent to class 1: 105

/sbin/tc filter add dev eth0 protocol ip parent 1:0 prio 4 u32 match ip dst 192.168.10.78 flowid 1:105



Prioritization works on the principle that each class is allocated the minimum bandwidth with the possibility of borrowing the maximum bandwidth from the parent class, depending on the traffic priority, so if the class is blocked by WWW traffic from the tcp port 80, lower priority than WWW traffic, it will be skipped out of turn, given its priority.



Restrict outgoing traffic



To limit the traffic coming from users, the same actions are performed as for incoming traffic, only the virtual interface ifb0 is used. You also need to change the destination of the following traffic: instead of dst 192.168.10.78 - you need to specify src 192.168.10.78, respectively.



Automation and Scripting



To begin with, to automate the speed limit process, you need to create a file that lists the addresses of users for whom you want to set limits with these restrictions.

 root@mixer:/etc/rc.d/shape# cat ./users #CLIENT IP DOWN CEIL UP CEIL PROVIDER ID user1 192.168.10.78 1Mbit 2Mbit 1Mbit 4Mbit mageal 10 user2 192.168.10.44 1Mbit 1Mbit 2Mbit 3Mbit ukrtel 11 

The file is a field, separated by a tab or a space with the following values:

CLIENT - Username. It is necessary for the convenience of providing data

IP address of the user on the network

DOWN - data rate to user

CEIL - the maximum speed of incoming traffic to the user when this band is available from the parent class

UP - user data rate

CEIL - the same as CEIL for incoming traffic to the user

PROVIDER - which of the providers is used to service user requests (if there are several)

ID is the class number for the user. Read more about the class numbers below.

I also use several bash scripts.



root@steel:/etc/rc.d/shape# cat ./rc.shape



#!/bin/bash

. /etc/init.d/functions

/sbin/modprobe ifb

/sbin/ip link set dev ifb0 up



TC="/sbin/tc"

DEV_P1_DOWN="eth0"

DEV_P1_UP="ifb0"



stop(){

$TC qdisc del dev $DEV_P1_DOWN root

$TC qdisc del dev $DEV_P1_UP root

$TC qdisc del dev $DEV_P1_DOWN ingress

}

start(){

#

$TC qdisc del dev $DEV_P1_DOWN root

$TC qdisc del dev $DEV_P1_UP root

$TC qdisc del dev $DEV_P1_DOWN ingress



## ifb0

$TC qdisc add dev $DEV_P1_DOWN ingress

$TC filter add dev $DEV_P1_DOWN parent ffff: protocol ip u32 match u32 0 0 action mirred egress redirect dev $DEV_P1_UP



#



. /etc/rc.d/shape/rc.shape.down.classes

. /etc/rc.d/shape/rc.shape.up.classes



# , 50 80.

$TC filter add dev $DEV_P1_UP protocol ip parent 1:0 prio 1 u32 match ip dst 10.0.0.1 flowid 1:10

$TC filter add dev $DEV_P1_DOWN protocol ip parent 1:0 prio 1 u32 match ip src 10.0.0.1 flowid 1:10



#

. /etc/rc.d/shape/rc.shape.filters

}



case "$1" in

start)

start

;;

stop)

stop

;;

restart|reload)

start

;;

*)

msg_usage "${0##*/} {restart|start|stop}"

RETVAL=1

esac



Further the code which is loaded:



Classes



root@steel:/etc/rc.d/shape# cat ./rc.shape.down.classes



#!/bin/bash

## DOWNLOAD CLASSES

##########################################################

#

$TC qdisc add dev $DEV_P1_DOWN root handle 1: htb default 900



#

$TC class add dev $DEV_P1_DOWN parent 1: classid 1:1 htb rate 100Mbit burst 15k



# (SERVER -> CLIENTS)

$TC class add dev $DEV_P1_DOWN parent 1:1 classid 1:10 htb rate 50Mbit ceil 80Mbit burst 15k



# (SERVER -> CLIENTS)

$TC class add dev $DEV_P1_DOWN parent 1:1 classid 1:100 htb rate 15Mbit burst 15k



# (SERVER -> CLIENTS)

$TC class add dev $DEV_P1_DOWN parent 1:1 classid 1:900 htb rate 128Kbit ceil 128Kbit



root@steel:/etc/rc.d/shape# cat ./rc.shape.up.classes

#!/bin/bash



## UPLOAD CLASSES

#############################################################

#

$TC qdisc add dev ifb0 root handle 1: htb default 900



#

$TC class add dev ifb0 parent 1: classid 1:1 htb rate 100Mbit burst 15k

# (CLIENTS -> SERVER)

$TC class add dev $DEV_P1_UP parent 1:1 classid 1:10 htb rate 50Mbit ceil 80Mbit burst 15k



# (CLIENTS -> SERVER)

$TC class add dev $DEV_P1_UP parent 1:1 classid 1:100 htb rate 5Mbit burst 15k

# (CLIENTS -> SERVER)

$TC class add dev $DEV_P1_UP parent 1:1 classid 1:900 htb rate 128Kbit ceil 128Kbit





And filters:



root@steel:/etc/rc.d/shape# cat ./rc.shape.filters

#!/bin/bash

# “users”

while read LINE

do

set -- $LINE

if [[ $1 =~ '^\#' ]]

then

continue

fi

################################################################

CLIENT_IP=$2

CLIENT_DOWN_RATE=$3

CLIENT_DOWN_CEIL=$4

################################################################

# DOWNSTREAM

#####################



#

$TC class add dev $DEV_P1_DOWN parent 1:100 classid 1:${8}1 htb rate $CLIENT_DOWN_RATE ceil $CLIENT_DOWN_CEIL



# PRIO 1 -> icmp traffic

$TC class add dev $DEV_P1_DOWN parent 1:${8}1 classid 1:${8}2 htb rate 33kbit ceil $CLIENT_DOWN_CEIL prio 1



# PRIO 2 -> udp, ssh

$TC class add dev $DEV_P1_DOWN parent 1:${8}1 classid 1:${8}3 htb rate 33kbit ceil $CLIENT_DOWN_CEIL prio 2



# PRIO 3 -> tcp sport 80

$TC class add dev $DEV_P1_DOWN parent 1:${8}1 classid 1:${8}4 htb rate 33kbit ceil $CLIENT_DOWN_CEIL prio 3



# PRIO 4 -> unclassified traffic

$TC class add dev $DEV_P1_DOWN parent 1:${8}1 classid 1:${8}5 htb rate 33kbit ceil $CLIENT_DOWN_CEIL prio 4



# icmp- icmp- 1

$TC filter add dev $DEV_P1_DOWN protocol ip parent 1:0 prio 1 u32 match ip dst $CLIENT_IP \

match ip protocol 1 0xff flowid 1:${8}2



# udp

$TC filter add dev $DEV_P1_DOWN protocol ip parent 1:0 prio 2 u32 match ip dst $CLIENT_IP \

match ip protocol 17 0xff flowid 1:${8}3

# ssh

$TC filter add dev $DEV_P1_DOWN protocol ip parent 1:0 prio 2 u32 match ip dst $CLIENT_IP \

match ip protocol 6 0xff match ip sport 22 0xffff flowid 1:${8}3

# WWW, sport 80

$TC filter add dev $DEV_P1_DOWN protocol ip parent 1:0 prio 3 u32 match ip dst $CLIENT_IP \

match ip protocol 6 0xff match ip sport 80 0xffff flowid 1:${8}4

# – ,

$TC filter add dev $DEV_P1_DOWN protocol ip parent 1:0 prio 4 u32 match ip dst $CLIENT_IP flowid 1:${8}5



# UPSTREAM

#####################



CLIENT_UP_RATE=$5

CLIENT_UP_CEIL=$6



###



$TC class add dev $DEV_P1_UP parent 1:100 classid 1:${8}1 htb rate $CLIENT_UP_RATE ceil $CLIENT_UP_CEIL

# PRIO 1 -> icmp traffic

$TC class add dev $DEV_P1_UP parent 1:${8}1 classid 1:${8}2 htb rate 1kbit ceil $CLIENT_UP_CEIL prio 1

# PRIO 2 -> udp, ssh

$TC class add dev $DEV_P1_UP parent 1:${8}1 classid 1:${8}3 htb rate 1kbit ceil $CLIENT_UP_CEIL prio 2

# PRIO 3 -> unclassified traffic

$TC class add dev $DEV_P1_UP parent 1:${8}1 classid 1:${8}4 htb rate 1kbit ceil $CLIENT_UP_CEIL prio 3

$TC filter add dev $DEV_P1_UP protocol ip parent 1:0 prio 1 u32 match ip src $CLIENT_IP \

match ip protocol 1 0xff flowid 1:${8}2

$TC filter add dev $DEV_P1_UP protocol ip parent 1:0 prio 2 u32 match ip src $CLIENT_IP \

match ip protocol 17 0xff flowid 1:${8}3

$TC filter add dev $DEV_P1_UP protocol ip parent 1:0 prio 2 u32 match ip src $CLIENT_IP \

match ip protocol 6 0xff match ip dport 22 0xffff flowid 1:${8}3

$TC filter add dev $DEV_P1_UP protocol ip parent 1:0 prio 3 u32 match ip src $CLIENT_IP flowid 1:${8}4



done < ./users



These scripts need to be put in one directory, Run:



chmod +x ./rc.shape



and then



./rc.shape start



I described one of the methods to limit traffic. The tc utility is very powerful in terms of traffic restrictions. I recommend to familiarize with the document: LARTC-HOWTO for more in-depth study of this issue.