Netgraph ipfw and flexible traffic accounting via netflow

Hello again!

Let's continue the review of FreeBSD netgraph modules.
Today our guest is ng_ipfw, ng_one2many and old friend ng_netflow.
With their help, we will try to calculate the traffic under certain conditions and see how it can be used in practice.

What kind of modules?

ng_ipfw is a module that creates an interface for accessing the netgraph subsystem from the IPFW firewall. When loaded into the kernel, the module automatically creates its node in the space of the netgraph subsystem with the name "ipfw:", and can be present only in a single copy. Ng_ipfw accepts connections to arbitrary number hooks, hook numbers are specified in the ipfw rules. Each packet entering the module is tagged by several parameters (ipfw rule, interface, direction), so that it can be identified when returning to ipfw. Non-tagged packets returned to the firewall are destroyed. Control messages module does not accept.

ng_one2many - a module that allows you to distribute packets across multiple connections. It accepts connections to one “one” hook and to numerous “many0”, “many1”, “manyN” ... As long as there are two packet distribution algorithms within the module.
')
1. Packages included in “one” alternately go through different “manyN” (xmit_roundrobin). All packages included in "manyN" go in "one".
2. Packages included in “one” are copied through all “manyN” (xmit_all). All packages included in "manyN" go in "one". (seems to work ng_hub).

There are also algorithms for determining up / down links, and the ability to turn them on / off via commands, this is necessary for specific tasks, more details on this in man ng_one2many.

Ng_one2many accepts control messages: setconfig, getconfig, getstats, clrstats, getclrstats. In principle, intuitive. Read more in man.

Module ng_hole - does the elementary actions, counts the packets included in it and destroys them.

I wrote about the ng_netgraph module in my past FreeBSD Netgraph topic using the example of an Ethernet tunnel. .

IPFW

There is no sense for system administrators working with FreeBSD to talk about ipfw, and who do not know what it can read without any problems in Wikipedia , and Google will display tons of information with examples and ready-made developments. Let me just say that for the interaction between ipfw and netgraph in ipfw there are two rules ngtee and netgraph, for example:

ipfw add netgraph 10 all traffic from any to any - will send all traffic caught in the rule to the ipfw hook: 10 module ng_ipfw

ipfw add ngtee 20 ip from any to any - copies all traffic that fell into the rule to the ipfw hook: 20 of the ng_ipfw module

Count a little

We want to count traffic passing through the interface of each user in both directions. At the same time exclude subnets 192.168.5.0/24 and 192.168.10.0/24 from accounting, because they have free resources in them. Users are terminated on the ngX interfaces by the mpd daemon.

In ipfw, it will look like this:
ipfw table 1 flush
ipfw table 1 add 192.168.5.0/24
ipfw table 1 add 192.168.10.0/24

ipfw add ngtee 10 ip from not "table(1)" to any in via "ng*"
ipfw add ngtee 10 ip from any to not "table(1)" out via "ng*"

That is, we copy to ipfw: 10 of the netgraph subsystem all traffic that does not belong to the networks listed in table 1.

Draw a graph. It is quite simple.

We create a simple structure of the ng_ipfw and ng_netflow and ng_ksocket modules. Since this is a collection of statistics, the data will go only in one direction, processed and sent to the right place.

We copy all the traffic passing through IPFW rules to ng_ipfw, which in turn will be connected to ng_netflow, connected to ksocket and then to netflow collector.

We collect in system

Create a netflow module and connect its iface0 hook to the ipfw 10 hook.
ngctl mkpeer ipfw: netflow 10 iface0
We call the netflow module created “netflow”, it is located at the creation address “ipfw: 10”
ngctl name ipfw: 10 netflow
We send the “netflow:” module a command to receive net IP packets via iface0. (More details in my past topic about the ng_netflow module)
msg netflow: setdlt {iface = 0 dlt = 12}
Create a ksocket module and connect its inet / dgram / udp hook to “netflow:” the export hook.
ngctl mkpeer netflow: ksocket export inet / dgram / udp
Call the created module "netflow_socket"
ngctl name netflow: export netflow_socket
Send the "netflow_socket" command to connect to the collector.
ngctl msg netflow_socket: connect inet / 3.3.3.3: 9996

It would seem that everything is simple - it will work, but it turns out to be not quite as it should be. The package that came out of the user interface and entered into the interface of our neighboring user will be counted twice. Here we can stop counting, for example, outgoing traffic ... but the ng_one2many module comes to the rescue.

New graph

Since ng_one2many can work in roundrobin mode, and even does it by default, we can insert it between ng_ipfw and ng_netflow.

To split traffic in half, you need to connect one2many hooks of many0 and many1 to ng_netflow and ng_hole, since the mandatory condition of packet distribution for one2many are hooks in the active state.

It turns out that only half of the packages will be taken into account, with a small margin of error, and the second part will be destroyed.

For the second graph, we will use the new hook of the ipfw module - 20. Also, when creating the graph, we will need to give the nodes new names so that they do not overlap with the graph already created above. Creating it is similar, and I will not describe it in detail.

As a result, we get the final scripts for work.


The first script to collect full netflow graph.
#!/bin/sh
case "$1" in
start)
echo "Starting full netflow."
ngctl mkpeer ipfw: netflow 10 iface0
ngctl name ipfw:10 netflow
ngctl msg netflow: setdlt {iface=0 dlt=12}
ngctl mkpeer netflow: ksocket export inet/dgram/udp
ngctl name netflow:export netflow_socket
ngctl msg netflow_socket: connect inet/3.3.3.3:9996
echo "Ok."
exit 0
;;
stop)
echo "Stopping full netflow."
ngctl shutdown netflow_socket:
ngctl shutdown netflow:
echo "Ok."
exit 0
;;
restart)
sh $0 stop
sh $0 start
;;
*)
echo "Usage: `basename $0` { start | stop | restart }"
exit 64
;;
esac


The second script creates a graph of accounting for half of the traffic.
#!/bin/sh
case "$1" in
start)
echo "Starting half netflow."
ngctl mkpeer ipfw: one2many 20 one
ngctl name ipfw:20 one2many
ngctl mkpeer one2many: netflow many0 iface0
ngctl name one2many:many0 netflow_half
ngctl mkpeer one2many: hole many1 black
ngctl name one2many:many1 blackhole
ngctl mkpeer netflow_half: ksocket export inet/dgram/udp
ngctl name netflow_half:export netflow_socket_half
ngctl msg netflow_half: setdlt {iface=0 dlt=12}
ngctl msg netflow_socket_half: connect inet/3.3.3.3:9996
echo "Ok."
exit 0
;;
stop)
echo "Stopping half netflow."
ngctl shutdown netflow_socket_half:
ngctl shutdown blackhole:
ngctl shutdown netflow_half:
ngctl shutdown one2many:
echo "Ok."
exit 0
;;
restart)
sh $0 stop
sh $0 start
;;
*)
echo "Usage: `basename $0` { start | stop | restart }"
exit 64
;;
esac


In IPFW we get a new design. In the second label we add networks, the traffic between them should be reduced by 2 times.

ipfw table 1 flush
ipfw table 1 add 192.168.5.0/24
ipfw table 1 add 192.168.10.0/24

ipfw table 2 flush
ipfw table 2 add 192.168.50.0/24
ipfw table 2 add 192.168.51.0/24
ipfw table 2 add 192.168.52.0/24
ipfw table 2 add 192.168.53.0/24

ipfw add ngtee 10 ip from not "table(1)" to any in via "ng*"
ipfw add ngtee 10 ip from any to not "table(1)" out via "ng*"

ipfw add ngtee 20 ip from "table(2)" to "table(2)" via "ng*"


That's all

Thanks to everyone who heroically read the entire article.