Parsing NetFlow v.9 C # Packages

NetFlow is a network protocol created by Cisco Systems to account for network traffic. The most common versions of this protocol are 5 and 9. The ninth version is more flexible because it uses templates according to which the data is sent. In the fifth version of the data are sent according to the specification.


NetFlow 's traffic information collection system consists of the following components:

• Sensor A device (router, L3 switch) that collects statistics on traffic passing through it;
• Collector . It collects data from the sensor and puts it in the storage;
• Analyzer . Analyzes the data collected by the collector and generates reports.

I will tell you about the development of some functions of the analyzer in C # , and more precisely the analysis of NetFlow packets
')
As the sensor was used MikroTik router.

We enable NetFlow on it for ether1 interface:
/ip traffic-flow set enabled=yes interfaces=ether1

And we add a collector (as a rule, the collector listens on port 2055, 9555 or 9995):
/ip traffic-flow target add disabled=no version=9 address=192.168.0.100:9995

Or the same thing through WinBox :


Now, NetFlow 9 version packets will come to the computer with IP address 192.168.0.100 on port 9995 via UDP (or SCTP ). Packages come and have something to work with.

Parsing incoming packages

Having studied the protocol specification, we learn that each NetFlow packet (N bytes) consists of:

1. Packet Header (20 bytes) - packet header, in a single copy with the fields:
   
   • Version Number ( UInt16 - 2 bytes) - the version number of NetFlow , we always have 9;
   • Count ( UInt16 - 2 bytes) - the total number of records. Further, in the text of the articles with this field was an adventure ;
   • sysUpTime ( UInt32 - 4 bytes) - time in milliseconds from the start of the device - UpTime;
   • UNIX Secs ( UInt32 - 4 bytes) - time in seconds since 0000 UTC 1970, at which the packet was sent;
   • Sequence Number ( UInt32 - 4 bytes) - the counter of transmitted packets, it is constantly increasing from packet to packet, thus you can check whether packets are lost between them;
   • Source ID ( UInt32 - 4 bytes) - the number of data stream, in fact, that from the side of the sensor there can be several data streams.
   
   
2. FlowSet (N-20 bytes) - templates, data ... FlowSet`s can be several or one. In each FlowSet there are two unchanged fields of the type of data transmitted (template, data):
   
   • FlowSet ID ( UInt16 2 bytes) - for the template it is always 0, for the optional template 1, for data it is equal to the Template ID and therefore more than 255 (from 256 to 65535);
   • Length ( UInt16 2 bytes) - the size of the entire FlowSet along with the FlowSet ID and Length fields;
   • Other fields depending on the type of data transmitted.
   
   

Look at the FlowSet ID containing the template, it starts with the fields FlowSet ID , then Length , then:

• Template ID ( UInt16 2 bytes) - a unique ID for each template for which data is transmitted. The number is from 256 to 65535;
• Field Count ( UInt16 2 bytes) - the number of fields in the pattern. Next come alternately field type ( Field Type ) and size ( Field Length );
• Field Type ( UInt16 2 bytes) - the number that determines the type of field. All types are in the protocol specification ;
• Field Length - the length of the field in bytes.

We look at the FlowSet ID containing the data, start with the fields FlowSet ID , then Length , then:

• Data ... data that correspond to the fields and their sizes;
• Padding - zeros up to the 4-byte border.

There is also a so-called optional template and data on it. I will not consider them, they have not met me, for this reason there are no libraries in the library implementation, but everything can be added.

Compiled UML class diagram (using NClass ):

or in pdf
And wrote a library for parsing incoming packages.
The main class that starts it all is Packet . Its only constructor accepts an incoming NetFlow packet in bytes and an object of the Templates class, which is a list of current Templates .

Next, in the constructor of the Packet class, the Parse function is called, which takes an object of the Templates class.
In this function, the packet is divided into a header - 20 bytes and further work with it through the Header class; on FlowSet`s and transfer of each FlowSet`s and processing to the appropriate class FlowSet .

Due to the fact that FlowSet`s can be several, the second part of the packet (without 20 bytes of the header) has to be analyzed and divided into different FlowSet`s . It is noteworthy that in MikroTik ` FlowSet`s in a single copy in the package, but using Netflow Simulator in C # it was possible to work with packages with several FlowSet`s in the package. In addition, thanks to him, a funny bug was found in the implementation of NetFlow v9 on MikroTik `s, what more here .

Netflow Simulator in C # :


Here is the code section breaking part of the package at FlowSet`s :

 this._flowset = new List<FlowSet>(); Int32 length = _bytes.Length - 20; Byte[] flowset = new Byte[length]; Array.Copy(_bytes, 20, flowset, 0, length); byte[] reverse = flowset.Reverse().ToArray(); int templengh = 0; while ((templengh + 2) < flowset.Length) { UInt16 lengths = BitConverter.ToUInt16(reverse, flowset.Length - sizeof(Int16) - (templengh+2)); Byte[] bflowsets = new Byte[lengths]; Array.Copy(flowset, templengh, bflowsets, 0, lengths); FlowSet flowsets = new FlowSet(bflowsets, templates); this._flowset.Add(flowsets); templengh += lengths; } 

In the Header class, the packet header is parsed into its fields. Before you do this, the header is reversed:

 this._bytes.Reverse().ToArray(); 

Next, convert the bits to the type of field it is, for example the version field:

 this._version = BitConverter.ToUInt16(reverse, this._bytes.Length - sizeof(Int16) - 0); 

Yes, in the Header of the sysUpTime field, it has a TimeSpan type, we cast it to this type:

 get { return new TimeSpan((long)this._uptime * 10000); } 

and the UNIX Secs field is of type DateTime :

 get { return new DateTime(1970, 1, 1).AddSeconds(this._secs); } 

Let us turn to processing FlowSet`s . After receiving the FlowSet ID and Length fields, the remaining fields are parsed depending on the FlowSet ID . If it is 0 or 1, then this is a pattern, and if it is a number from 256 to 65535, then this is data.

If this is a template, then we transfer its processing to the Template class and then check our template storage (object of the Templates class) for the presence of a template with the same ID and replace it, otherwise we simply add the template.

If this is data, then we check if there is such a template ( FlowSet ID == Template ID ) in the storage (object of the Templates class) and if there is, then copy this template with the DeepClone function and fill its fields - Field , otherwise we do nothing, because without a template it’s just set of bytes.

DeepClone function:

 public static object DeepClone(object obj) { object objResult = null; using (MemoryStream ms = new MemoryStream()) { BinaryFormatter bf = new BinaryFormatter(); bf.Serialize(ms, obj); ms.Position = 0; objResult = bf.Deserialize(ms); } return objResult; } 

Field is a field, it has the following parameters:

• Type - type;
• Length - size;
• Value - the value.

Moreover, the Field in Template in the repository is without Value parameters, i.e. Value is empty, but when processing Template packages in a FlowSet in a Packet object, it already contains Value fields.

Besides all this, there is also the FieldType enumeration - an enumeration in which the type name corresponds to the number of this type. ( Type parameter in Field )

An example was written for the work of this library:

 using System; using System.Collections.Generic; using System.Linq; using System.Net; using System.Net.NetFlow; using System.Net.Sockets; using System.Text; using System.Threading.Tasks; namespace Consoles { class Program { static void Main(string[] args) { Templates _templates = new Templates(); Socket sock = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp); IPEndPoint iep = new IPEndPoint(IPAddress.Any, 9995); sock.Bind(iep); EndPoint ep = (EndPoint)iep; byte[] data = new byte[2048]; while (true) { int recv = sock.ReceiveFrom(data, ref ep); Console.ReadKey(); Console.Clear(); byte[] bytes = new byte[recv]; for (int i = 0; i < recv; i++) bytes[i] = data[i]; Packet packet = new Packet(bytes, _templates); Console.ForegroundColor = ConsoleColor.Yellow; Console.WriteLine(packet.ToString()); } sock.Close(); Console.ReadKey(); } } } 

We create a socket and listen on UDP port 9995 on our PC. _templates is our template repository. Each incoming packet we feed to the packet object of the Packet class passing also our template storage. And then we execute packet. ToString () This overloaded function outputs the contents of the packet to us and is only needed to check that everything works out for us.

This is all with the library; now it can be used to further write the NetFlow traffic analyzer.

Example with MikroTik :

Received a package without a template in the repository:


Received the template from the sensor:


Received data for which there is a template in the repository:

Error implementing NetFlow v9 in MikroTik `e

In the process of analyzing this topic, an error was found in the implementation of NetFlow v9 in MikroTik `e. The essence of the error:
The Count field in the packet header ( Packet Header ) carries:

Count
The total number of records in the Export Packet, which is
sum of Options FlowSet records, Template FlowSet records, and
Data FlowSet records.

those. contains all records in all FlowSets, ah, and in MicroTik , this field is always 1 (see screenshots above), even if several templates or data are transmitted. those. according to the logic of MikroTik `and the Count field = the number of FlowSet`s (what they wrote to me in the letter and can be seen on the screen), but should be equal to the total number of all templates and data, as it sounds in the specification. For this reason, the use of the Count field in packet parsing is fraught.

Here is an example from Netflow Simulator in C # (I would like to receive data from Cisco as well , but I don’t have such an opportunity, can a reader check this):

Received a package without a template in the repository (note the Count ):


Received a template from the sensor (here at the same time two FlowSet `and it came that in MikroTik` e does not happen. Pay attention to the Count it is equal to 7 = 1 template and 6 data records. According to the logic of MikroTik `and the Count would have been equal to 2 = 2 FlowSet a):


Received data for which there is a template in the repository (note the Count ):


Well, once again the package in Wireshark Marked field Count :


Once again: I will be very grateful to everyone who will send a screen with Wireshark `from Cisco . I'll put it in here.

Source code is available here .

When creating guided:
Wikipedia: Netflow
Caligare: WHAT IS NETFLOW?
Protocol Specification Version 9

Today (19:05 07/30/2013)

MikroTik replied:

MikroTik support [Dzintars] support@mikrotik.com

Hello,

Thank you for reporting a problem.
that count is not set correctly in the netflow packet header). The
problem will be fixed in the next version of RouterOS.

Regards,
Dzintars

UPD: This bug has been fixed in RouterOS 6.2.

Today (13:22 13/09/2013)

MikroTik wrote:

MikroTik support [Dzintars] support@mikrotik.com

Hello,

The NetFlow V9 count field problem was fixed in version 6.2

Regards,
Dzintars