We count network traffic using netflow and lightsquid

Once in our small office it was necessary to set up a traffic accounting system for linux. Skimming through the ready-made solutions, I decided to use something simpler. Googling, I liked the approach to use a samopisny script that would convert netflow statistics into a squid-compatible log, which you can then analyze with anything. I didn’t use a ready-made script walking around Google, because it uses ipcad, which is absent in debian and ubuntu. I wrote my own version using any netflow sensor and flow-tools as a collector. The script gradually became more complicated and, as a result, grew into a small traffic accounting system. The archive of this pleasure weighs about 50kb, while:

• installed using deb for debian-based distributions or using your own installer for the rest
• can count Internet traffic on the router for all ip v4 protocols and ports
• can limit access to unregistered users
• instead of bare statistics of ip-addresses, netflow can detail reports using dig and squid
• can selectively block traffic
• has a web interface to manage users and user groups
• configured a little harder than lightsquid, which uses in work
• does not use the access.log intermediate log; own parser converts netflow statistics directly into lightsquid report

It looks like this: user management and access to statistics is carried out through a web interface ( demo ). Each user can be authorized via ip, mac or ip + mac. Unauthorized users see the “access denied” error. Each user can be assigned a “squid” flag to redirect web traffic to transparent squid. Each user can be assigned an “ipblock” flag to block any traffic through ipset. Manage lock lists and other fine-tuning settings are done through the console. Installing and configuring everything is not necessary, you can configure only the necessary components.
The name was chosen as the name lightwrapper (derived from the lightsquid wrapper) and a project at sourceforge was created . How to try:

1. install recommended dependencies: apache2 softflowd ipset lightsquid perl iptables flow-tools dnsutils conntrack iptables-persistent
2. download archive , install deb or run installer.

Actually, there are quite a lot of dependencies, and many of them require tuning - this is a minus. But with this approach it is unlikely that this is possible. Briefly, the setup is as follows:

1. configure netflow sensor and collector (one line in the config of both)

   Setting up flow-tools

   Enter in the configuration file (in debian - /etc/flow-tools/flow-capture.conf)
   -N 0 -w / tmp -R / usr / sbin / lw-export -n 95 0 / 127.0.0.1 / 2055

   
   

   Softflowd setup

   Enter in the configuration file (in debian - / etc / default / softflowd):
   INTERFACE = "name of the local network interface, for example, eth0"
   OPTIONS = "- n 127.0.0.1:2055"

   Softflowd is easy to configure, but can lose packets. If you are not satisfied with its performance, try ipt-netflow .
2. configure lightsquid

   Apache2 setup

   The archive contains examples: lightwrapper.conf - virtualhost configuration for apache2 and access_denied.html - stub for displaying to non-users. Examples are in / usr / share / doc / lightwrapper / examples. You must turn off the virtualhost settings for lightsquid (because they give access without a password) and enable the settings for lightwrapper. In debian, you can replace the /etc/apache2/sites-enabled/000-default.conf file with the one given. You must also enable the cgi module. In debian, you need to make a link with /etc/apache2/mods-available/cgi.load in / etc / apache2 / mods-enabled. After that, you need to enable access with password protection. Install the apache2-utils package, if you do not have the htpasswd utility, create an etc directory in the site root and set the administrator password: htpasswd -c / www / etc / passwd administrator
   

   
   

   Configuring lightsquid

   It is necessary to delete cron jobs for lightparser.pl, if they are present in the system.
   The configuration file is configured automatically by installation scripts. You can configure lightsquid to your taste, except for the following parameters, which can not be changed:
   $ graphmaxuser = 1
   $ graphmaxall = 1
   $ showgroupid = 0
   $ showputpost = 1
   $ templatename = 'lightwrapper'
   $ logpath = '/ var / log / lightwrapper'
   $ lang = 'lw-eng'
   or another language (lw-ru)
   
3. edit config file. In the simplest version, you only need to enter the names of 2x network interfaces

   / etc / lightwrapper / cfg:
   in_if and out_if
   
4. run the iptables configuration script (it does everything automatically, you only need it once)

   Run lw-geniprules. Make sure that ip v4 forwarding is enabled (add net.ipv4.ip_forward = 1 in /etc/sysctl.conf and apply the changes to sysctl --system).
5. install and configure additional dependencies, if any

   Installation of transparent squid is not described; ipset setting for blocking traffic is not described. Detailed instructions can be found in man or on the wiki.
6. Restart all services to apply the settings or reboot.

Statistics is updated by default every 15 minutes. The sensor can give out streams with some delay.
Details are described step by step in broken English in the wiki and in man lightwrapper. There are examples of configuration files of everything you can, in most cases, you can simply copy them to your system.
')
A little bit about the detail reports. Netflow gives the following information: ip addresses, source and destination ports, protocol.



If for a pair (protocol: destination port) there is a service name in / etc / services, then lightwrapper displays it in statistics. If the dig utility is installed, ip-addresses are automatically resolved to names (if there is such a match in dns).



If transparent squid is installed and some users are redirected to it, then lightwrapper scans the squid log and assigns the names to the found ip-addresses. You can configure the lightwrapper so that it resolves ip-addresses in the names for users who are not redirected to squid. At the same time, for obvious reasons, it is impossible to vouch for the accuracy of such recognition.



I will be glad to any reviews and bug reports.