Netstat, where are my datagrams?

It is probably difficult to find another, as useful and poorly documented program as Netstat , meaning the option to display network traffic statistics. When we inspect the network status on a single Linux node, you can always be sure that this utility is available. And here we want to understand whether the network stack is coping with the load, or the problem on the upper floors of OSI , in fact, where the wheels of the business logic of our application are spinning.

 (5:562)$ netstat -s |wc -l 124 

Hooray, we have a lot of useful information, now we quickly figure out what's what. If only to understand what kind of a beast it is, such a timeout in transit , obviously something is not good.

 Icmp: 11475275 ICMP messages received 327527 input ICMP message failed. ICMP input histogram: detination unreachable: 2233840 timeout in transit: 5612259 

Now I look at the manual.

 # man netstat |grep timeout # 

The empty set suggests that it is time to learn from Yandex, Google and even DuckDuckGo. The search result was about the same, but only Google gave 111 thousand variants of the empty set, and Yandex succinctly limited itself to 326 variants.

But we didn’t have chosen the open source software in vain, you can always consult the original source. In our case, the source code is in the nertstat.c file of the Net-tools package.

This is the part of the int main function that defines the action for displaying statistics.

  case '?': usage(E_OPTERR); case 'h': usage(E_USAGE); case 's': flag_sta++; ... if (flag_sta) { if (!afname[0]) safe_strncpy(afname, DFLT_AF, sizeof(afname)); if (!strcmp(afname, "inet")) { #if HAVE_AFINET parsesnmp(flag_raw, flag_tcp, flag_udp, flag_sctp); 

The parsesnmp function parsesnmp defined in the statistics.c file and parsit files:

 /proc/net/netstat /proc/net/snmp /proc/net/sctp/snmp 

Clearly understandable, so you can just look in / proc / net / snmp and see what's behind the timeout in transit .

 # cat /proc/net/snmp |grep -iw icmp Icmp: InMsgs InErrors InCsumErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps Icmp: 127 2 0 90 0 0 0 0 25 0 8 0 4 0 1763 0 1730 0 0 0 0 0 25 0 8 0 0 InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps # cat /proc/net/snmp |grep -iw icmp Icmp: InMsgs InErrors InCsumErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps Icmp: 127 2 0 90 0 0 0 0 25 0 8 0 4 0 1763 0 1730 0 0 0 0 0 25 0 8 0 0 

Comparing these values ​​with the output of the netstat command, we can conclude that the timeout in transit corresponds to the InTimeExcds variable, and the chain further leads to the /usr/include/linux/snmp.h file, where all these fields are defined.

 /* icmp mib definitions */ /* * RFC 1213: MIB-II ICMP Group * RFC 2011 (updates 1213): SNMPv2 MIB for IP: ICMP group */ enum { ICMP_MIB_NUM = 0, ICMP_MIB_INMSGS, /* InMsgs */ ICMP_MIB_INERRORS, /* InErrors */ ICMP_MIB_INDESTUNREACHS, /* InDestUnreachs */ ICMP_MIB_INTIMEEXCDS, /* InTimeExcds */ ICMP_MIB_INPARMPROBS, /* InParmProbs */ ICMP_MIB_INSRCQUENCHS, /* InSrcQuenchs */ ICMP_MIB_INREDIRECTS, /* InRedirects */ ICMP_MIB_INECHOS, /* InEchos */ ICMP_MIB_INECHOREPS, /* InEchoReps */ ICMP_MIB_INTIMESTAMPS, /* InTimestamps */ ICMP_MIB_INTIMESTAMPREPS, /* InTimestampReps */ ICMP_MIB_INADDRMASKS, /* InAddrMasks */ ICMP_MIB_INADDRMASKREPS, /* InAddrMaskReps */ ICMP_MIB_OUTMSGS, /* OutMsgs */ ICMP_MIB_OUTERRORS, /* OutErrors */ ICMP_MIB_OUTDESTUNREACHS, /* OutDestUnreachs */ ICMP_MIB_OUTTIMEEXCDS, /* OutTimeExcds */ ICMP_MIB_OUTPARMPROBS, /* OutParmProbs */ ICMP_MIB_OUTSRCQUENCHS, /* OutSrcQuenchs */ ICMP_MIB_OUTREDIRECTS, /* OutRedirects */ ICMP_MIB_OUTECHOS, /* OutEchos */ ICMP_MIB_OUTECHOREPS, /* OutEchoReps */ ICMP_MIB_OUTTIMESTAMPS, /* OutTimestamps */ ICMP_MIB_OUTTIMESTAMPREPS, /* OutTimestampReps */ ICMP_MIB_OUTADDRMASKS, /* OutAddrMasks */ ICMP_MIB_OUTADDRMASKREPS, /* OutAddrMaskReps */ ICMP_MIB_CSUMERRORS, /* InCsumErrors */ __ICMP_MIB_MAX }; 

Who needs it, and for me RFC 1213 as a chemist, the Periodic Table - you need to know by heart, since this is one of the cornerstones of monitoring network devices. Here is the definition of the item from RFC 1213 writing from memory .

 icmpInTimeExcds OBJECT-TYPE SYNTAX Counter ACCESS read-only STATUS mandatory DESCRIPTION "The number of ICMP Time Exceeded messages received." ::= { icmp 4 } 

So far, it seems that all these definitions as oil, do not reveal the essence, but fortunately ICMP Time Exceeded is part of the definition of the ICMP protocol .

If the gateway processing the datagram sees that the TTL field contains a zero value, the datagram should be discarded. The gateway can notify the sender of the datagram using a time exceeded message.

So, we found what you were looking for. The output of the netstat command, in which this mysterious field was located, means the number of discarded ICMP packets with zero TTL . IRL ^[1] this means that there is a ring on the network, where the packages profukali the entire TTL ^[2] .

 timeout in transit: 5612259 

Linux MIB

During this dive into the wilds of the Linux kernel, I was surprised to find the SNMP subsystem. No, not an SNMP agent, save us Alan Cox, namely a small SNMP stack. This is written in the comment to the file /usr/src/linux/include/net/snmp.h .

 /* * SNMP MIB entries for the IP subsystem. * Alan Cox <gw4pts@gw4pts.ampr.org> * * We don't chose to implement SNMP in the kernel (this would * be silly as SNMP is a pain in the backside in places). We do * however need to collect the MIB statistics and export them * out of /proc (eventually) */ 

SNMP stands for Simple Network Management Protocol , but as the bearded joke says, the letter S has never lied like that in the abbreviation. I plan to write about this brainstorming unit separately, since it is on it that all the monitoring software of computer networks and network nodes is running. In the meantime - the minimum necessary to make it clear where the netstat variable statistics come from.

In the simplest case, we have a client-server architecture, where MRTG or Munin can act as a client, and the server part is an SNMP agent on a network node. Almost all modern network devices have an SNMP agent onboard, even home WiFi routers. Windows has its own SNMP Service , and Linux and open Unix systems use Net-SNMP .

SNMP client and server exchange messages: the client sends a request, and the server returns a response. Usually, this exchange looks like this.

K. - Cheburashka, instruments!
S. - Forty
K. - What is forty?
S. - And what about the devices?

The client asks to give the value of some variable from the Great Dictionary. The server looks in VS, finds a variable, looks in the register and returns value. The Great Dictionary is the MIB ^[3] Database from the figure.

This is how polling system variables, uptime and others look like on the man page .

 snmpwalk -Os -c public -v 1 zeus system sysDescr.0 = STRING: "SunOS zeus.net.cmu.edu 4.1.3_U1 1 sun4m" sysObjectID.0 = OID: enterprises.hp.nm.hpsystem.10.1.1 sysUpTime.0 = Timeticks: (155274552) 17 days, 23:19:05 sysContact.0 = STRING: "" sysName.0 = STRING: "zeus.net.cmu.edu" sysLocation.0 = STRING: "" sysServices.0 = INTEGER: 72 

It is necessary to understand that behind the euphemisms of the Great Dictionary, the registry and the SNMP protocol itself, dozens of RFCs are hidden and the abyss of their possible implementations. Returning to our snmp.h file, only the tip of the iceberg is visible:

 # grep RFC /usr/include/linux/snmp.h * RFC 1213: MIB-II * RFC 2011 (updates 1213): SNMPv2-MIB-IP * RFC 2863: Interfaces Group MIB * RFC 2465: IPv6 MIB: General Group * RFC 1213: MIB-II ICMP Group * RFC 2011 (updates 1213): SNMPv2 MIB for IP: ICMP group * RFC 2466: ICMPv6-MIB * RFC 1213: MIB-II TCP group * RFC 2012 (updates 1213): SNMPv2-MIB-TCP * RFC 1213: MIB-II UDP group * RFC 2013 (updates 1213): SNMPv2-MIB-UDP LINUX_MIB_TCPMINTTLDROP, /* RFC 5082 */ 

And if that were all, the decoding of the netstat statistic could be like this:

   netstat -->     /proc: * /proc/net/snmp * /proc/net/netstat * /proc/net/sctp/snmp -->  RFC --> . 

However, even such a multistage decryption is not enough, since a significant part of snmp.h is taken up by the mysterious Linux MIB , the variables of which are not defined in any RFC, and they are not in the Great Dictionary - MIB Database either.

 /* linux mib definitions */ enum { LINUX_MIB_NUM = 0, LINUX_MIB_SYNCOOKIESSENT, /* SyncookiesSent */ LINUX_MIB_SYNCOOKIESRECV, /* SyncookiesRecv */ LINUX_MIB_SYNCOOKIESFAILED, /* SyncookiesFailed */ LINUX_MIB_EMBRYONICRSTS, /* EmbryonicRsts */ LINUX_MIB_PRUNECALLED, /* PruneCalled */ LINUX_MIB_RCVPRUNED, /* RcvPruned */ LINUX_MIB_OFOPRUNED, /* OfoPruned */ ... LINUX_MIB_TCPMTUPSUCCESS, /* TCPMTUPSuccess */ __LINUX_MIB_MAX }; 

We are ready for the next stage of the dive, in the next article we will try to find out what lies behind the buffer memory errors, and how queue violators collapse in tcp.

1. ↑ In real life, from English in real life.
2. ↑ Package lifetime, from English time to live
3. ↑ Management Information Base - a structured set of management information of node objects. More here .