Principles of IP traffic accounting
Any administrator sooner or later receives instructions from the management: “calculate who goes to the network, and how many shakes”. For providers, it is complemented by the tasks “to let someone go, take payment, restrict access”. What to count? How? Where? There are many fragmentary information, they are not structured. Let's get rid of the novice admin from tedious searches, providing him with general knowledge, and useful links to the materiel.
In this article I will try to describe the principles of organizing the collection, accounting and control of traffic on the network. We will consider the problem of the issue, and list the possible ways of retrieving information from network devices.
This is the first theoretical article from a series of articles devoted to the collection, accounting, management and billing of traffic and IT resources.
In general, the structure of network access is as follows:
')
In this structure, network traffic passes from external resources to internal, and vice versa, through the access device. It sends traffic information to the management server. The management server processes this information, stores it in the database, displays it, and issues commands for blocking. However, not all combinations of access devices (methods), and methods of collection and control, are compatible. On the various options and will be discussed below.
First you need to determine what is meant by “network traffic” and what useful statistical information can be extracted from the user data stream.
The dominant interworking protocol is still IP version 4 . The IP protocol corresponds to the 3rd level of the OSI model (L3). Information (data) between the sender and the recipient is packaged in packages - having a header, and "payload". The header determines where the packet goes from and to (sender's and recipient's IP addresses), packet size, type of payload. The main part of network traffic consists of packets with UDP and TCP payloads - these are Layer 4 (L4) protocols. In addition to addresses, the header of these two protocols contains port numbers that define the type of service (application) that transmits data.
To transmit an IP packet over wires (or radio), network devices are forced to "wrap" (encapsulate) it into a packet of the 2nd level protocol (L2). The most common protocol of this type is Ethernet . The actual transfer "to the wire" is at the 1st level. Usually, the access device (router) does not analyze the packet headers at a level higher than 4th (the exception is intelligent firewalls).
Information from the fields of addresses, ports, protocols, and length counters from the L3 and L4 headers of data packets constitutes the “source material” that is used to account for and control traffic. The actual amount of information transmitted is in the Length field of the IP header (including the length of the header itself). By the way, due to packet fragmentation due to the MTU mechanism, the total amount of transmitted data is always larger than the size of the payload.
The total length of IP and TCP / UDP fields of a packet that we are interested in in this context is 2 ... 10% of the total packet length. If you process and store all this information in a package, there will not be enough resources. Fortunately, the vast majority of traffic is structured in such a way that it consists of a set of “conversations” between external and internal network devices, the so-called “flows”. For example, within a single e-mail transfer operation (SMTP protocol), a TCP session is opened between the client and the server. It is characterized by a constant set of parameters {source IP address, source TCP port, destination IP address receiver TCP port} . Instead of processing and storing information per packet, it is much more convenient to store the flow parameters (addresses and ports), as well as additional information - the number and sum of the lengths of the transmitted packets in each direction, optionally the session duration, interface indexes of the router, the value of the ToS field, and so on. This approach is beneficial for connection-oriented protocols (TCP), where you can clearly intercept the time when a session ends. However, for non-session-oriented protocols, it is possible to carry out aggregation and logical completion of a stream record by, for example, timeout. The following is an excerpt from the SQL database of a proprietary billing system that logs traffic flow information:
It is necessary to note the case when the access device translates addresses ( NAT , masquerading) to provide Internet access to computers on a local network using a single, external, public IP address. In this case, a special mechanism substitutes IP addresses and TCP / UDP ports of traffic packets, replacing the internal (not routed on the Internet) addresses according to its dynamic translation table. In this configuration, it must be remembered that in order to correctly record data on internal network hosts, the statistics should be removed in a way and in a place where the result of the broadcast does not “de-personalize” the internal addresses.
You can remove and process information about passing traffic directly on the access device itself (PC router, VPN server), transferring it from this device to a separate server (NetFlow, SNMP), or from the wire (tap, SPAN). We analyze all the options in order.
Consider the simplest case - an access device (router) based on a PC with Linux OS.
Much has been written about how to set up such a server, address translation and routing. We are also interested in the next logical step - information on how to get information about traffic passing through such a server. There are three common ways:
In the first case, a copy of the packet passing through the interface after passing through the filter ( man pcap-filter ) can be requested by the client program on the server written using this library. The packet comes with a Layer 2 (Ethernet) header. You can limit the length of the captured information (if we are only interested in the information from its header). Examples of such programs are tcpdump and Wireshark . There is an implementation of libpcap under Windows . If address translation is used on a PC router, such interception can only be performed on its internal interface connected to local users. On the external interface, after the broadcast, the IP packets do not contain information about the internal hosts of the network. However, with this method it is impossible to take into account the traffic generated by the server itself on the Internet (which is important if it is running a web or mail service).
The work of libpcap requires support from the operating system, which currently boils down to installing a single library. At the same time, an application (user) program that collects packages must:
When a packet is transmitted through the selected interface, after passing the filter, this function receives a buffer containing Ethernet, (VLAN), IP, etc. headers, total size up to snaplen. Since the libcap library copies packages, it is impossible to block their passage with its help. In this case, the program for collecting and processing traffic will have to use alternative methods, for example, invoking a script to place the specified IP address in the traffic blocking rule.
Capturing data passing through the firewall allows you to take into account both the traffic of the server itself and the traffic of network users, even when address translation is running. The main thing in this case is to correctly formulate the rule of capture, and put it in the right place. This rule activates the transfer of the packet towards the system library, from where the traffic accounting and control application can receive it. For Linux OS iptables is used as a firewall, and interception tools are ipq, netfliter_queue or ulog . For FreeBSD OC, ipfw with rules like tee or divert . In any case, the firewall mechanism is complemented by the ability to work with the user program in the following way:
Since the IP packet is not copied, but sent to the analysis software, it becomes possible to “eject” it and, therefore, completely or partially restrict traffic of a certain type (for example, to the selected local network subscriber). However, if the application stops responding to the kernel about its decision (hung, for example), traffic through the server is simply blocked.
It should be noted that the described mechanisms with significant amounts of transmitted traffic create excessive load on the server, which is associated with the constant copying of data from the kernel to the user program. This disadvantage is deprived of the method of collecting statistics at the kernel level of the OS, with issuing aggregated statistics to the application program using the NetFlow protocol.
This protocol was developed by Cisco Systems to export traffic information from routers for the purpose of traffic accounting and analysis. The most popular now version 5 provides the recipient with a stream of structured data in the form of UDP packets containing information about past traffic in the form of so-called flow records:
The amount of information about traffic is less than the traffic itself by several orders of magnitude, which is especially important in large and distributed networks. Of course, it is impossible to block the transfer of information when collecting statistics on netflow (unless additional mechanisms are used).
Currently, further development of this protocol is becoming popular - version 9, based on the flow record template structure, implementation for devices from other manufacturers ( sFlow ). Recently, the IPFIX standard was adopted, which allows the transfer of statistics through deeper protocol layers (for example, by application type).
The implementation of netflow-sources (agents, probe) is available for PC routers, both in the form of utilities (flowprobe, fprobe , softflowd ) working according to the mechanisms described above, and directly embedded in the OS kernel (FreeBSD: ng_netgraph , Linux: ipt_neflow ). For software routers, the flow of netflow statistics can be received and processed locally on the router itself, or sent over the network (transmission protocol - over UDP) to the receiving device (collector).
A collector program can collect information from many sources at once, having the ability to distinguish their traffic even in overlapping address spaces. With the help of additional tools, such as nprobe, it is also possible to carry out additional data aggregation, split flows or protocol conversion, which is important when managing a large and distributed network with dozens of routers.
Netflow export features support Cisco Systems, Mikrotik, and some other routers. Similar functionality (with other export protocols) is supported by all major network equipment manufacturers.
Let's complicate the task a bit. What if your access device is a hardware router from another manufacturer? For example, D-Link, ASUS, Trendnet, etc. On it, most likely, it is impossible to put additional software tool for data acquisition. As an option, you have an intelligent access device, but it is not possible to configure it (no rights, or it is controlled by your provider). In this case, you can collect traffic information directly at the access point of the access device with the internal network, using “hardware” packet copying tools. In this case, you will certainly need a standalone server with a dedicated network card to receive copies of Ethernet packets.
The server must use the packet collection mechanism of the libpcap method described above, and our task is to send a data stream identical to the output from the access server to the input of the dedicated network card. To do this, you can use:
Naturally, you can configure the SPAN port on the access device itself (router), if it allows it - Cisco Catalyst 6500, Cisco ASA. Here is an example of such a configuration for a Cisco switch:
What if there is no router under our control, there is no desire to communicate with netflow, we are not interested in the traffic details of our users. They are simply connected to the network through a managed switch, and we just have to roughly estimate the amount of traffic per each of its ports. As you know, network devices with remote control support, and can display packet counters (bytes) passing through network interfaces. For their survey, it will be correct to use a standardized remote management protocol SNMP . Using it, you can simply get not only the values of the specified counters, but also other parameters, such as the name and description of the interface, the MAC addresses visible through it, and other useful information. This is done both with command line utilities ( snmpwalk ), graphical SNMP browsers, and more sophisticated network monitoring programs ( rrdtools , cacti , zabbix , whats up gold , etc.). However, this method has two significant drawbacks:
We should also consider the case of user access to the network by explicitly establishing a connection to the access server. A classic example is the good old dial-up, the analogue of which in the modern world are VPN- remote access services (PPTP, PPPoE, L2TP, OpenVPN, IPSEC)
The access device not only routes the IP traffic of users, but also represents a specialized VPN server, and terminates logical tunnels (often encrypted), within which user traffic is transmitted.
To account for such traffic, you can use all the tools described above (and for in-depth analysis by ports / protocols they are well suited), as well as additional mechanisms that provide VPN access control. First of all, we will focus on the RADIUS protocol. His work is a rather complicated topic. We briefly mention that the control (authorization) of access to the VPN server (RADIUS client) is controlled by a special application (RADIUS server), which has a base (text file, SQL, Active Directory) of valid users with their attributes (restrictions on connection speeds assigned IP addresses). In addition to the authorization process, the client periodically sends the account messages to the server, information about the status of each currently running VPN session, including counters of transmitted bytes and packets.
Let's summarize all the above methods of collecting information about traffic together:
Let's summarize a little. In practice, there are a large number of methods for connecting a network managed by you (with clients or office subscribers) to an external network infrastructure using a variety of access tools — software and hardware routers, switches, VPN servers. However, in almost any case, you can come up with a scheme when information about the traffic transmitted over the network can be sent to a software or hardware tool for analyzing and managing it. It is also possible that this tool will allow feedback from the access device, applying intelligent access restriction algorithms for individual clients, protocols and other things.
On it I will finish analysis of materiel. Of the unsolved topics left:
In this article I will try to describe the principles of organizing the collection, accounting and control of traffic on the network. We will consider the problem of the issue, and list the possible ways of retrieving information from network devices.
This is the first theoretical article from a series of articles devoted to the collection, accounting, management and billing of traffic and IT resources.
The structure of access to the Internet
In general, the structure of network access is as follows:
')
- External resources - the Internet, with all sites, servers, addresses and other things that do not belong to the network that you control.
- The access device is a router (hardware, or PC-based), a switch, a VPN server or a hub.
- Internal resources - a set of computers, subnets, subscribers whose work in the network must be taken into account or monitored.
- A management or accounting server is a device running specialized software. It can be functionally combined with a software router.
In this structure, network traffic passes from external resources to internal, and vice versa, through the access device. It sends traffic information to the management server. The management server processes this information, stores it in the database, displays it, and issues commands for blocking. However, not all combinations of access devices (methods), and methods of collection and control, are compatible. On the various options and will be discussed below.
Network traffic
First you need to determine what is meant by “network traffic” and what useful statistical information can be extracted from the user data stream.
The dominant interworking protocol is still IP version 4 . The IP protocol corresponds to the 3rd level of the OSI model (L3). Information (data) between the sender and the recipient is packaged in packages - having a header, and "payload". The header determines where the packet goes from and to (sender's and recipient's IP addresses), packet size, type of payload. The main part of network traffic consists of packets with UDP and TCP payloads - these are Layer 4 (L4) protocols. In addition to addresses, the header of these two protocols contains port numbers that define the type of service (application) that transmits data.
To transmit an IP packet over wires (or radio), network devices are forced to "wrap" (encapsulate) it into a packet of the 2nd level protocol (L2). The most common protocol of this type is Ethernet . The actual transfer "to the wire" is at the 1st level. Usually, the access device (router) does not analyze the packet headers at a level higher than 4th (the exception is intelligent firewalls).
Information from the fields of addresses, ports, protocols, and length counters from the L3 and L4 headers of data packets constitutes the “source material” that is used to account for and control traffic. The actual amount of information transmitted is in the Length field of the IP header (including the length of the header itself). By the way, due to packet fragmentation due to the MTU mechanism, the total amount of transmitted data is always larger than the size of the payload.
The total length of IP and TCP / UDP fields of a packet that we are interested in in this context is 2 ... 10% of the total packet length. If you process and store all this information in a package, there will not be enough resources. Fortunately, the vast majority of traffic is structured in such a way that it consists of a set of “conversations” between external and internal network devices, the so-called “flows”. For example, within a single e-mail transfer operation (SMTP protocol), a TCP session is opened between the client and the server. It is characterized by a constant set of parameters {source IP address, source TCP port, destination IP address receiver TCP port} . Instead of processing and storing information per packet, it is much more convenient to store the flow parameters (addresses and ports), as well as additional information - the number and sum of the lengths of the transmitted packets in each direction, optionally the session duration, interface indexes of the router, the value of the ToS field, and so on. This approach is beneficial for connection-oriented protocols (TCP), where you can clearly intercept the time when a session ends. However, for non-session-oriented protocols, it is possible to carry out aggregation and logical completion of a stream record by, for example, timeout. The following is an excerpt from the SQL database of a proprietary billing system that logs traffic flow information:
It is necessary to note the case when the access device translates addresses ( NAT , masquerading) to provide Internet access to computers on a local network using a single, external, public IP address. In this case, a special mechanism substitutes IP addresses and TCP / UDP ports of traffic packets, replacing the internal (not routed on the Internet) addresses according to its dynamic translation table. In this configuration, it must be remembered that in order to correctly record data on internal network hosts, the statistics should be removed in a way and in a place where the result of the broadcast does not “de-personalize” the internal addresses.
Traffic / Statistics Information Collection Methods
You can remove and process information about passing traffic directly on the access device itself (PC router, VPN server), transferring it from this device to a separate server (NetFlow, SNMP), or from the wire (tap, SPAN). We analyze all the options in order.
PC router
Consider the simplest case - an access device (router) based on a PC with Linux OS.
Much has been written about how to set up such a server, address translation and routing. We are also interested in the next logical step - information on how to get information about traffic passing through such a server. There are three common ways:
- interception (copying) of packets passing through the server's network card using the libpcap library
- interception of packets passing through the built-in firewall
- use of third-party tools for converting per-packet statistics (obtained by one of the two previous methods) into the netflow aggregated information flow
Libpcap
In the first case, a copy of the packet passing through the interface after passing through the filter ( man pcap-filter ) can be requested by the client program on the server written using this library. The packet comes with a Layer 2 (Ethernet) header. You can limit the length of the captured information (if we are only interested in the information from its header). Examples of such programs are tcpdump and Wireshark . There is an implementation of libpcap under Windows . If address translation is used on a PC router, such interception can only be performed on its internal interface connected to local users. On the external interface, after the broadcast, the IP packets do not contain information about the internal hosts of the network. However, with this method it is impossible to take into account the traffic generated by the server itself on the Internet (which is important if it is running a web or mail service).
The work of libpcap requires support from the operating system, which currently boils down to installing a single library. At the same time, an application (user) program that collects packages must:
- open the required interface
- specify the filter through which to pass the received packets, the size of the captured part (snaplen), the size of the buffer,
- set the promisc parameter, which translates the network interface into the capture mode in general of all packets passing by, and not just addressed to the MAC address of this interface
- set the function (callback) called on each received packet.
When a packet is transmitted through the selected interface, after passing the filter, this function receives a buffer containing Ethernet, (VLAN), IP, etc. headers, total size up to snaplen. Since the libcap library copies packages, it is impossible to block their passage with its help. In this case, the program for collecting and processing traffic will have to use alternative methods, for example, invoking a script to place the specified IP address in the traffic blocking rule.
Firewall
Capturing data passing through the firewall allows you to take into account both the traffic of the server itself and the traffic of network users, even when address translation is running. The main thing in this case is to correctly formulate the rule of capture, and put it in the right place. This rule activates the transfer of the packet towards the system library, from where the traffic accounting and control application can receive it. For Linux OS iptables is used as a firewall, and interception tools are ipq, netfliter_queue or ulog . For FreeBSD OC, ipfw with rules like tee or divert . In any case, the firewall mechanism is complemented by the ability to work with the user program in the following way:
- A user program - a traffic handler registers itself in the system using a system call, or a library.
- A user program or an external script sets a rule in the firewall that “wraps” the selected traffic (according to the rule) inside the handler.
- For each passing packet, the handler receives its contents as a memory buffer (with IP headers, etc. After processing (accounting), the program must also be told to the operating system kernel what to do next with such a packet — discard or transfer further. As an option, it is possible pass a modified packet to the kernel.
Since the IP packet is not copied, but sent to the analysis software, it becomes possible to “eject” it and, therefore, completely or partially restrict traffic of a certain type (for example, to the selected local network subscriber). However, if the application stops responding to the kernel about its decision (hung, for example), traffic through the server is simply blocked.
It should be noted that the described mechanisms with significant amounts of transmitted traffic create excessive load on the server, which is associated with the constant copying of data from the kernel to the user program. This disadvantage is deprived of the method of collecting statistics at the kernel level of the OS, with issuing aggregated statistics to the application program using the NetFlow protocol.
Netflow
This protocol was developed by Cisco Systems to export traffic information from routers for the purpose of traffic accounting and analysis. The most popular now version 5 provides the recipient with a stream of structured data in the form of UDP packets containing information about past traffic in the form of so-called flow records:
The amount of information about traffic is less than the traffic itself by several orders of magnitude, which is especially important in large and distributed networks. Of course, it is impossible to block the transfer of information when collecting statistics on netflow (unless additional mechanisms are used).
Currently, further development of this protocol is becoming popular - version 9, based on the flow record template structure, implementation for devices from other manufacturers ( sFlow ). Recently, the IPFIX standard was adopted, which allows the transfer of statistics through deeper protocol layers (for example, by application type).
The implementation of netflow-sources (agents, probe) is available for PC routers, both in the form of utilities (flowprobe, fprobe , softflowd ) working according to the mechanisms described above, and directly embedded in the OS kernel (FreeBSD: ng_netgraph , Linux: ipt_neflow ). For software routers, the flow of netflow statistics can be received and processed locally on the router itself, or sent over the network (transmission protocol - over UDP) to the receiving device (collector).
A collector program can collect information from many sources at once, having the ability to distinguish their traffic even in overlapping address spaces. With the help of additional tools, such as nprobe, it is also possible to carry out additional data aggregation, split flows or protocol conversion, which is important when managing a large and distributed network with dozens of routers.
Netflow export features support Cisco Systems, Mikrotik, and some other routers. Similar functionality (with other export protocols) is supported by all major network equipment manufacturers.
Libpcap “outside”
Let's complicate the task a bit. What if your access device is a hardware router from another manufacturer? For example, D-Link, ASUS, Trendnet, etc. On it, most likely, it is impossible to put additional software tool for data acquisition. As an option, you have an intelligent access device, but it is not possible to configure it (no rights, or it is controlled by your provider). In this case, you can collect traffic information directly at the access point of the access device with the internal network, using “hardware” packet copying tools. In this case, you will certainly need a standalone server with a dedicated network card to receive copies of Ethernet packets.
The server must use the packet collection mechanism of the libpcap method described above, and our task is to send a data stream identical to the output from the access server to the input of the dedicated network card. To do this, you can use:
- Ethernet - hub (hub): a device that simply forwards packets between all its ports without parsing. In modern reality, it can be found somewhere in a dusty warehouse, and this method is not recommended: unreliable, low speed (hubs at a speed of 1 Gbit / s does not happen)
- Ethernet switch with the possibility of mirroring (mirroring, SPAN ports . Modern intelligent (and expensive) switches allow copying to the specified port all traffic (incoming, outgoing, both) of another physical interface, VLAN, including remote (RSPAN)
- Hardware splitter , which may require installation to collect two network cards instead of one - and this is in addition to the main system one.
Naturally, you can configure the SPAN port on the access device itself (router), if it allows it - Cisco Catalyst 6500, Cisco ASA. Here is an example of such a configuration for a Cisco switch:
monitor session 1 source vlan 100 !
monitor session 1 destination interface Gi6/3!
SNMP
What if there is no router under our control, there is no desire to communicate with netflow, we are not interested in the traffic details of our users. They are simply connected to the network through a managed switch, and we just have to roughly estimate the amount of traffic per each of its ports. As you know, network devices with remote control support, and can display packet counters (bytes) passing through network interfaces. For their survey, it will be correct to use a standardized remote management protocol SNMP . Using it, you can simply get not only the values of the specified counters, but also other parameters, such as the name and description of the interface, the MAC addresses visible through it, and other useful information. This is done both with command line utilities ( snmpwalk ), graphical SNMP browsers, and more sophisticated network monitoring programs ( rrdtools , cacti , zabbix , whats up gold , etc.). However, this method has two significant drawbacks:
- Traffic blocking can only be done by completely disabling the interface using the same SNMP
- Traffic counters, taken via SNMP, refer to the sum of Ethernet packet lengths (unicast, broadcast and multicast separately), while the rest of the previously described means give values relative to IP packets. This creates a noticeable discrepancy (especially on short packets) due to the overhead caused by the length of the Ethernet header (however, this can be roughly dealt with: L3_byte = L2_byte - L2_packages * 38).
VPN
We should also consider the case of user access to the network by explicitly establishing a connection to the access server. A classic example is the good old dial-up, the analogue of which in the modern world are VPN- remote access services (PPTP, PPPoE, L2TP, OpenVPN, IPSEC)
The access device not only routes the IP traffic of users, but also represents a specialized VPN server, and terminates logical tunnels (often encrypted), within which user traffic is transmitted.
To account for such traffic, you can use all the tools described above (and for in-depth analysis by ports / protocols they are well suited), as well as additional mechanisms that provide VPN access control. First of all, we will focus on the RADIUS protocol. His work is a rather complicated topic. We briefly mention that the control (authorization) of access to the VPN server (RADIUS client) is controlled by a special application (RADIUS server), which has a base (text file, SQL, Active Directory) of valid users with their attributes (restrictions on connection speeds assigned IP addresses). In addition to the authorization process, the client periodically sends the account messages to the server, information about the status of each currently running VPN session, including counters of transmitted bytes and packets.
Conclusion
Let's summarize all the above methods of collecting information about traffic together:
Let's summarize a little. In practice, there are a large number of methods for connecting a network managed by you (with clients or office subscribers) to an external network infrastructure using a variety of access tools — software and hardware routers, switches, VPN servers. However, in almost any case, you can come up with a scheme when information about the traffic transmitted over the network can be sent to a software or hardware tool for analyzing and managing it. It is also possible that this tool will allow feedback from the access device, applying intelligent access restriction algorithms for individual clients, protocols and other things.
On it I will finish analysis of materiel. Of the unsolved topics left:
- how and where do the collected traffic data go
- traffic accounting software
- How does billing differ from simple “counting”?
- how can I impose a restriction on traffic
- accounting and restriction of websites visited
Source: https://habr.com/ru/post/136844/
All Articles