Tyuni memory and network stack in Linux: the history of the transfer of high-loaded servers to a fresh distribution

Until recently, Odnoklassniki used the partially updated OpenSuSE 10.2 as the main Linux distribution. However, it became increasingly difficult to maintain it, so from last year we moved to active migration to CentOS 7. At the preparatory stage of the transition, all internal procedures for CentOS were worked out, configs and configuration policies were prepared (we use CFEngine). Therefore, in many cases, the migration from one distribution kit to another consists in installing the OS via kickstart and deploying the application using the deployment system of our development - everything else is done without human intervention. This happens in many cases, although not in all.

But we faced the biggest problems when migrating video distribution servers. It took us half a year to solve them.

Briefly about their configuration:

• 4 x 10 Gbps to users
• 2 x 10 Gbps to the storage
• 256 GB RAM - memory cache
• 22 x 480 GB SSD cache on SSD
• 2 x E5-2690 v2

')
Several metrics characterizing video distribution:

• 600 Gbit / s - total peak video upload rate
• 220M video views per day
• 600K simultaneously viewed videos

Problem 1 - a strong increase in CPU system time

Shortly after the launch of the first server, the processor load began to grow strongly at system time.
At the same time, a lot of migration and ksoftirqd processes were visible in the top. At first we tried to spin the kernel settings. What did not help us:

• increase sched_migration_cost
• disable transparent_hugepage
• increase vfs_cache_pressure
• NUMA off

With the next increase in load, perf top showed a 50% load on isolate_freepages_block. In itself, the name of the call to us, unfortunately, does not say anything. But the word freepages was a bit confusing, because There was about 45 GB of free memory on the server. From our experience, we already knew that if there is a lot of free memory on the server, and the kernel still complains about its lack (sometimes this is expressed in the launch of the OOM killer), then, most likely, the problem is in fragmentation . Resetting the disk cache ( echo 3 > /proc/sys/vm/drop_caches ) instantly healed the server, which only confirmed our assumptions.

Memory fragmentation is a common problem (and not only characteristic of Linux), and changes are regularly made to the kernel to combat it. One of the culprits of fragmentation is the kernel itself, or rather the disk cache, which can neither be disabled nor limited in size. This does not mean that the fragmentation in our case was caused by the disk cache, but the exact reasons were not so important. More important was the decision - defragmentation. Such a mechanism is in the kernel, but it is obvious that it did not cope (or instead of it, a memory release was launched - global reclaim). Defragmentation starts only when the free memory drops below a certain mark (zone watermark), and in our case it happened too late. The only way to make it start earlier is to increase min_free_kbytes via sysctl. This parameter tells the kernel to try to keep part of the memory free, and to satisfy this requirement, it has to start defragmentation earlier. In our case, enough values ​​in 1 GB.

Problem 2 - swap care

For a start it is worth mentioning that we use vm.swappiness = 0, because we know for sure that for us the swap is evil. So, there is about 45 GB of free memory on the server and yet it periodically goes into a swap. How is this possible? For a start it is worth recalling that memory in Linux is not one big chunk.

• Memory is divided into nodes: one physical processor - one node.
• Each node is divided into zones (view for 64-bit systems) - ZONE_DMA (0-16 MB), ZONE_DMA32 (0-4 GB), ZONE_NORMAL (4+ GB).
• Each zone is divided into memory areas of the size of a power of two (order of 2), i.e. 2 ⁰ * PAGE_SIZE, 2 ¹ * PAGE_SIZE ... 2 ¹⁰ * PAGE_SIZE (the current distribution can be found in / proc / buddyinfo). The lack of free areas of large size - this is the fragmentation, which we discussed in the previous section.

At first, we again blamed fragmentation, but a further increase in min_free_kbytes, as well as an increase in vfs_cache_pressure, did not help us. I had to get acquainted with the numastat ( numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product utility numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product	numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

1. numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
2. numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
3. numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
4. numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ). . tmpfs ( ), 1 . numastat? , : Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0 NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) . 3 — , , , . ( , , ): CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , . ?.. . 16 , Intel ( 10 ), , , 16. 16 — , RSS. RSS- , 4 , , — 16. . Intel, . -, ( , ). , .. , , . -, Flow director. , , , , . , . Flow director Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir Perfect filter (ntuple) 8 000 . Perfect filter / ethtool -u/U flow-type . Signature Filter 32 000 . , ATR , ( ) __ SYN . During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow. [1] .., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS. , — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , . RPS (receive packet steering) . 2 ( RSS) — scaling.txt. ( irq_balancer). . , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ). .. , , , , , .. — — . , scaling.txt RFS. RFS (receive flow steering) ? , , , . .. , , , . [2]: RPS RFS Accelerated RFS scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox. , . , kernel panic. , . , firmware . 4 — broken pipe broken pipe OpenSuSE, CentOS . , , “” . Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . . — "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe . , — interrupt coalescing. , . 5 — , Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% . , CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , . , softirq , , . : [rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] — : , 2024 [3] broken pipe 40 ( 50 ) ( CFEngine ) . , , -, . , . . https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

• numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
• numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
• numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
• numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

1. numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
2. numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
3. numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
4. numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product

1. numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
2. numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product
3. numastat -m ).

. tmpfs ( ), 1 .

numastat? , :
Per-node process memory usage (in MBs) for PID 7781 (java) Node 0 Node 1 Total Huge 0 0 0 Heap 0.20 0.14 0.34 Stack 118.82 137.87 256.70 Private 80200.73 123323.81 203524.55 Total 80319.76 123461.82 203781.58 Per-node system memory usage (in MBs): Node 0 Node 1 Total MemTotal 131032.75 131072.00 262104.75 MemFree 1228.93 639.84 1868.78 MemUsed 129803.82 130432.16 260235.98 Active 23224.13 121073.42 144297.55 Inactive 101138.88 3753.98 104892.85 Active(anon) 1690.50 120997.86 122688.36 Inactive(anon) 79528.66 3560.95 83089.61 Active(file) 21533.63 75.57 21609.20 Inactive(file) 21610.21 193.03 21803.24 Unevictable 0 0 0 Mlocked 0 0 0 Dirty 0.11 0.02 0.13 Writeback 0 0 0 FilePages 122397.46 124295.47 246692.93 Mapped 78436.03 122947.26 201383.29 AnonPages 1966.62 532.02 2498.64 Shmem 79251.21 123964.70 203215.90 KernelStack 2.44 2.57 5.01 PageTables 158.62 252.29 410.91 NFS_Unstable 0 0 0 Bounce 0 0 0 WritebackTmp 0 0 0 Slab 1801.95 1932.29 3734.23 SReclaimable 1653.13 1818.79 3471.92 SUnreclaim 148.82 113.49 262.31 AnonHugePages 1856.00 498.00 2354.00 HugePages_Total 0 0 0 HugePages_Free 0 0 0 HugePages_Surp 0 0 0
NUMA, 1 tmpfs . interleave (numactl —interleave=all, ) .

3 —
, , , . ( , , ):

CPU0 ( softirq). , , , . ( , ethtool -l/-L ). , 1 Intel 8, 10 — 128. . , , CPU0. irq_balancer. . , , . set_irq_affinity ( ) — RSS (receive side scaling) — https://www.kernel.org/doc/Documentation/networking/scaling.txt. 8 , 8 , 16 , .
?.. .
16 , Intel ( 10 ), , , 16.

16 — , RSS. RSS- , 4 , , — 16. .

Intel, . -, ( , ). , .. , , .

-, Flow director.

, , , , . , .

Flow director
Flow director, Intel, 2 — Signature Filter ( ATR — Application Targeted Receive) Perfect filter. Flow director -IP , . / flow director: ethtool -S ethN|grep fdir

Perfect filter (ntuple)
8 000 . Perfect filter / ethtool -u/U flow-type .

Signature Filter
32 000 . , ATR , ( ) __ SYN .

During transmission of a packet (every 20 packets by default), a hash is calculated based on the 5-tuple. The (up to) 15-bit hash result is used as an index in a hash lookup table to store the TX queue. When a packet is received, a similar hash is calculated and used to look up an associated Receive Queue. For uni-directional incoming flows, the hash lookup tables will not be initialized and Flow Director will not work. For bidirectional flows, the core handling the interrupt will be the same as the core running the process handling the network flow.
[1]

.., , SYN, 20 ( , AtrSampleRate). ATR, RSS (.. 16 ). . flow director ethtool -S ethN | grep fdir , fdir_miss. .. ATR, RSS. 16 . ( https://sourceforge.net/p/e1000/bugs/464/ ), , — RPS.

, — ethtool -k ethN ntuple on . ATR, Perfect filter, , , , .

RPS (receive packet steering)
. 2 ( RSS) — scaling.txt.

( irq_balancer).

. , , , 16. , , , , . RPS, RSS, 16 (.. ), ( , rps_cpus , ).

.. , , , , , .. — — . , scaling.txt RFS.

RFS (receive flow steering)
? , , , . .. , , , .

[2]:
RPS RFS

Accelerated RFS
scaling.txt, Accelerated RFS. ? RFS . Accelerated RFS , . Mellanox.

, . , kernel panic.

, . , firmware .

4 — broken pipe
broken pipe OpenSuSE, CentOS . , , “” .

Broken pipe — , , pipe. pipe , — . , , , , pipe broken pipe. , , ( half-duplex tcp close sequence ), - . ? ? , .. , , . .

— "packet reordering" ( http://en.wikipedia.org/wiki/Out-of-order_delivery ), . .. , , , , RFS broken pipe .



, — interrupt coalescing. , .

5 —
, Counter Strike latency, ( , ) . , ( ) (), softirq . softirq 50% .

, CPU, — . interrupt coalescing, ethtool -c/-C . interrupt coalescing — , .



, softirq , , .
:
[rx|tx]-usecs — [rx|tx]-frames — , *-irq — *-[low|high] —


:
, 2024 [3] broken pipe 40 ( 50 ) ( CFEngine )
.


, , -, . , . .


https://networkbuilders.intel.com/docs/network_builders_RA_packet_processing.pdf https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.centos.org/About/Product