FreeBSD Netgraph on the example of an Ethernet tunnel

Hello.

I think many system administrators working with FreeBSD are aware of the existence of the nuclear subsystem Netgraph. But not many know / understand how it works, and what can be built from this.

I'll tell you what it is, as well as analyze a simple example of an Ethernet bridge assembly via the Internet.
')

A bit of theory.

Wikipedia tells http://ru.wikipedia.org/wiki/Netgraph

Netgraph is the modular network subsystem of the FreeBSD kernel, based on graph principle. In Netgraph, a graph is constructed from nodes of various types, each node type has a certain number of inputs / outputs (hereinafter referred to as hooks). The netgraph node allows you to perform certain actions on a packet passing through it. Some Netgraph nodes provide support for various protocols, encapsulations, such as L2TP, PPTP, PPPoE, PPP, ATM, bluetooth, others are used for connecting modules and sorting / routing between netgraph nodes, such as bpf, split.

The Netgraph subsystem is a set of modules, each with its own specific task. This is such a handful of Lego cubes that can always be connected.

A striking example of using multiple netgraph modules is the massively used Russian mpd demon providers to terminate client tunnels, such as PPPoE, PPTP, PPP, and so on. In versions of the 3 mpd branches, I did many operations myself, but by 5 of the current version, I mainly deal with switching netgraph modules.

Let's look at our cubes.

[root@bsd1] /boot/kernel/> ls | grep ng_
ng_UI.ko
ng_async.ko
ng_atm.ko
ng_atmllc.ko
ng_bluetooth.ko
ng_bpf.ko
ng_bridge.ko
ng_bt3c.ko
ng_btsocket.ko
ng_car.ko
ng_ccatm.ko
ng_cisco.ko
ng_deflate.ko
ng_device.ko
ng_echo.ko
ng_eiface.ko
ng_etf.ko
ng_ether.ko
ng_fec.ko
ng_frame_relay.ko
ng_gif.ko
ng_gif_demux.ko
ng_h4.ko
ng_hci.ko
ng_hole.ko
ng_hub.ko
ng_iface.ko
ng_ip_input.ko
ng_ipfw.ko
ng_ksocket.ko
ng_l2cap.ko
ng_l2tp.ko
ng_lmi.ko
ng_mppc.ko
ng_nat.ko
ng_netflow.ko
ng_one2many.ko
ng_ppp.ko
ng_pppoe.ko
ng_pptpgre.ko
ng_pred1.ko
ng_rfc1490.ko
ng_socket.ko
ng_source.ko
ng_split.ko
ng_sppp.ko
ng_sscfu.ko
ng_sscop.ko
ng_sync_ar.ko
ng_sync_sr.ko
ng_tag.ko
ng_tcpmss.ko
ng_tee.ko
ng_tty.ko
ng_ubt.ko
ng_uni.ko
ng_vjc.ko
ng_vlan.ko
[root@bsd1] /boot/kernel/> ls | grep ng_ | wc -l
58
[root@bsd1] /boot/kernel/>

Currently 58 modules +1 main module netgraph.ko (FreeBSD 7.3). All of them can be built into the kernel statically, or dynamically loaded. Since the modules are nuclear, they also work right in the core, due to which you can get amazingly productive results.

For each module there is a man, which describes the capabilities of the module, inputs / outputs (hook, hooks, hooks, I will call them “hooks”), received control messages and their actions.
To control the netgraph subsystem, there is a ngctl utility that accepts various commands, such as list, mkpeer, connect, name, etc., you can read about them separately in man ngctl.

Task.

Let's set a simple task: it is necessary to connect two physical Ethernet networks via the Internet.
We have 2 servers.

BSD1
2 network interfaces
Interface 1 em0 - external IP 1.1.1.1
Interface 2 em1 - internal IP 192.168.1.1

BSD2
2 network interfaces
Interface 1 em0 - external IP 2.2.2.2
Interface 2 em1 - internal IP 192.168.1.2

Between the servers there is a routable Internet connection through external interfaces.
You need to connect the network on the em1 interfaces of both servers.

Let's look at our cubes.

We will need:

netgraph.ko - the foundation.

ng_ether.ko is a module for connecting to physical network interfaces. When this module is loaded in graph space, one node is automatically created for each physical interface, which has the name of this interface. In our case it will be "em0:", "em1:". Each ng_ether node has 3 hooks: lower, upper, orphans. The lower hook is a direct exit to the location of the network interface driver, from where the packets are sent or received by the device. The upper hook is a direct access to the place of the system kernel, from where packets should be sent or received by the network device driver. Hook orphans, this is the same lower one, only erroneous, corrupted packets from a network device get into it. We do not take it into account.

In simple terms: when ng_ether is connected, a gap appears in the exchange between the kernel and the network card driver. On the side of the core, this gap is called upper, on the side of the network device driver, lower. This gap is connected, until the moment when the netgraph subsystem connects to the lower or upper subsystem.


ng_bridge.ko - the module of the most real Ethernet switch. Everyone, I think, can imagine (and someone even now sees) a box with Ethernet ports, blinking lights, standing on a table and connecting computers to a network. ng_bridge is the box in the netgraph subsystem. ng_bridge is a simple implementation of an Ethernet switch, equipped with an arp table and a simple loop detection algorithm. Hooks ng_bridge are called link0, link1, etc. We will connect our ng_ether hooks to them. The module accepts control messages such as setconfig, getconfig, reset, getstats, clrstats, getclrstats, gettable. Message actions are intuitive, details in man ng_bridge.


ng_ksocket.ko is a module that allows you to listen to a socket directly from the kernel (any socket supported by the system function socket ()) and connect to another socket. Accepts the only connection to the hook "<family> / <type> / <proto>". In man socket it is told what are family, type, proto. In our case it will be "inet / dgram / udp". inet - ipv4, dgram - datagrams, proto - udp. The module also accepts several control messages, such as bind, listen, connect, accept, getname, getpeername, setopt, getopt. Read more in man ng_ksocket. Using the message “connect” we can connect our inet “inet / dgram / udp” to a remote one that was created in the same way, but with the “bind” command.

We make the graph.

For a better understanding of the work of your graph system, it is advisable to draw them before assembling in the system.



1. We connect both hooks of the ng_ether module of em1 interface to ng_bridge so that the system can see both the physical network connected to the interface and our virtual network. It is done the same on both servers.
2. The next free ng_bridge link is connected to ng_ksocket with inet / dgram / udp parameters. (Why did I choose UDP? Because no one guarantees the successful delivery of a signal in a cable or radio network, just as in the IP protocol, no one guarantees delivery of UDP.) It is done the same on both servers.
3. Commanding the ng_ksocket module to take a specific port, on a specific IP address, and also connect to a remote IP using the required port. We do this on both servers, in the difference that IP addresses and ports are changing to opposite to each other.

We collect the graph in the system.

I have already mentioned the ngctl utility above. With its help, we will create our modules in the system and link them.
We need the commands mkpeer, connect, name, msg. I will describe them in simple language.

Mkpeer command
Syntax: ngctl mkpeer module1 module_type2 hook1 hook2
Creates a “module2” with the specified type and connects its hook “hook” to the “hook 1” of the module “module 1”

Command connect.
Syntax: ngctl connect module1 module2 hook1 hook2
It connects the "hook1" of the module "module1" to the "hook2" of the module "module 2".

Command name.
Syntax: ngctl name module: hook name
Assigns a name to the module created via mkpeer.

The msg command.
Syntax: ngctl msg module: message parameters
Transmits a control “message” to a “module” with “parameters”.

Well, now what our graph will look like live for the server "bsd1". For the tunnel we will use udp port 7777

Create a node ng_bridge and connect to its hook “link0” hook of the network interface “em1” “lower”.
ngctl mkpeer em1: bridge lower link0
We call the newly created node the name "switch", it can be found along the path "em1: lower".
ngctl name em1: lower switch
We connect to the "link1" of our "switch" upper network interface "em1".
ngctl connect switch: em1: link1 upper
Create the ng_ksocket node and connect to its hook “inet / dgram / udp” “link2” of our “switch”
ngctl mkpeer switch: ksocket link2 inet / dgram / udp
Call the newly created ksocket "switch_socket", it can be found along the path "switch: link2"
ngctl name switch: link2 switch_socket
We send the command "bind" to our "switch_socket", with parameters. ksocket will take port 7777 on IP 1.1.1.1.
ngctl msg switch_socket: bind inet / 1.1.1.1: 7777
We send the "connect" command to our "switch_socket", with parameters. ksocket will connect to port 7777 at the IP address 2.2.2.2.
ngctl msg switch_socket: connect inet / 2.2.2.2: 7777
We send a command to the ng_ether module of the em1 network interface to switch to the listening mode for packets not addressed to it. We now need to accept packets for devices located in our virtual network.
ngctl msg em1: setpromisc 1
ngctl msg em1: setautosrc 0

For the server "bsd2" we just need to swap the parameters of the bind and connect commands in places.

For ease of use, I designed this into a sh script.
The script uses another command ngctl shutdown. This command sends a special control message to the module specified in the parameter. This message is received by each module, in more detail in the "man module". Usually this command causes the destruction of the module and the rupture of all its connections.

  #! / bin / sh
 self = 1.1.1.1
 peer = 2.2.2.2
 port = 7777
 if = em1

 case "$ 1" in
         start)
             echo "Starting netgraph switch."
             ngctl mkpeer $ {if}: bridge lower link0
             ngctl name $ {if}: lower switch
             ngctl connect switch: $ {if}: link1 upper
             ngctl mkpeer switch: ksocket link2 inet / dgram / udp
             ngctl name switch: link2 switch_socket
             ngctl msg switch_socket: bind inet / $ {self}: $ {port}
             ngctl msg switch_socket: connect inet / $ {peer}: $ {port}
             ngctl msg $ {if}: setpromisc 1
             ngctl msg $ {if}: setautosrc 0
             echo "Ok."
             exit 0
             ;;
         stop)
             echo "Stopping netgraph switch."
             ngctl shutdown switch_socket:
             ngctl shutdown switch:
             ngctl shutdown $ {if}:
             echo "Ok."
             exit 0
             ;;
         restart)
             sh $ 0 stop
             sh $ 0 start
             ;;
         *)
             echo "Usage:` basename $ 0` {start | stop | restart} "
             exit 64
             ;;
 esac

Let's see what happened
[root@bsd1] /usr/local/etc/rc.d/> ngctl list
There are 5 total nodes:
Name: em0 Type: ether ID: 00000001 Num hooks: 0
Name: em1 Type: ether ID: 00000002 Num hooks: 2
Name: switch Type: bridge ID: 000000f6 Num hooks: 3
Name: ngctl16408 Type: socket ID: 00000100 Num hooks: 0
Name: switch_socket Type: ksocket ID: 000000fa Num hooks: 1


The ngctl16408 module with the socket type is used by ngctl for management, do not pay attention to it.

Bags run:

 [root @ bsd1] / root /> ping 192.168.1.2
 PING 192.168.1.2 (192.168.1.2): 56 data bytes
 64 bytes from 192.168.1.2: icmp_seq = 0 ttl = 64 time = 3.760 ms
 64 bytes from 192.168.1.2: icmp_seq = 1 ttl = 64 time = 3.527 ms
 64 bytes from 192.168.1.2: icmp_seq = 2 ttl = 64 time = 3.479 ms
 64 bytes from 192.168.1.2: icmp_seq = 3 ttl = 64 time = 4.052 ms

 [root @ bsd1] / root /> ngctl msg switch: getstats 2
 Rec'd response "getstats" (4) from "[f6]:":
 Args: {recvOctets = 49333 recvPackets = 532 recvMulticast = 467 recvBroadcast = 63 xmitOctets = 580 xmitPackets = 12 xmitMulticasts = 10 xmitBroadcasts = 1}

the end

Wrote a lot of text, hopefully understandable. In practice, such a tunnel will seem to someone unprotected, because bags will run openly over the Internet, but no one bothers you to forward this tunnel inside a VPN connection. For example, I use such a bundle to pass multicast IPTV to a work network.

In the next article I will describe a few more modules of the netgraph - traffic count ng_netflow, nat ng_nat, add shaping via ng_car to the scheme today, or maybe invent something else interesting.

Thanks for attention.