How to get and measure high-speed TCP connection
Reliable data transmission on the Internet is based on the TCP ( Transmission Control Protocol ) protocol , the specification for which was published almost 30 years ago. The TCP algorithm ( RFC793 ) allows the connected device to adapt to work on the network at speeds of tens of megabits per second and delays of up to 100 seconds. With the rapid development of new data transmission technologies, already 10 years after implementation, it became clear that the performance of the protocol would not be enough for wider channels.
Data transfer rate depends on network and system characteristics.
img1 The process of transferring data over the network
a) Buffers
The original TCP configuration limits the transfer rate to the buffer (the Window Size option is “window size”) and is a field with a size of 2 ^ 16 bytes (up to 64 KB). Maximum throughput in this case:
Example: You have a 100 megabyte connection to the Internet and there is a 100 ms delay to the server.
Standard TCP stack, maximum data transfer rate does not exceed 10 Mbps
(524288 bits / 0.1 sec = 5.24 Mbps in spite of the fact that you have a 100 megabit link).
b) Bandwidth-delay product (BDP)
TCP performance in principle depends not so much on the speed of the channel as on the so-called “bandwidth * delay product” or BDP (the result is bandwidth * delay), which is the number of bytes needed for the sender and receiver for maximally filling the TCP connection.
Performance problems arise in cases of so-called “long and wide pipes” (LFN “long fat network”), since the BDP in this case exceeds the TCP window size, thereby limiting the transmission rate.
img2 Effect of delay on maximum TCP throughput
')
An example would be the mobile Internet or a fast optical link.
Example of BDP calculations:
a) Mobile broadband internet: 10 Mb / s, 100 ms RTT
b) high-speed ground network: 1 Gb / s, 10 ms RTT
You can calculate BDP here .
The “TCP window size” must exceed the BDP in order to reach the maximum channel load.
c) Protocol Overhead
By some estimates, about 95% of the computers in the world are connected via Ethernet technology.
Ethernet MTU (Ethernet frame payload) = max. 1500 bytes.
If we take into account all the Ethernet, IP, TCP headers, the picture will look like this:
img3 Single Ethernet frame transmission
The numbers indicate the size (in bytes) of the header for a particular protocol.
IFG (Interframe gap) - mandatory interframe space.
Headers: preamble, frame delimiter, Ethernet header / FCS - 26 bytes, IFG - 12 bytes, IP header - 20 bytes, TCP header - 20 bytes.
If you exclude VLAN tagging, TCP timestamp and other optional features, the maximum payload (TCP) on Ethernet networks will be:
Max TCP Payload = (MTU – TCP – IP) / (MTU + Ethernet + IFG) = (1500–40) / (1500 + 26 + 12) = 94.9%
d) Packet Delay and Loss
Since we are talking about reliable transmission of information, the loss of packets on the network forces TCP to resend segments and directly affects the reduction in speed.
The dependence of TCP speed in relation to packet loss is determined by the Mathis formula:
where: MSS (Maximum segment size) - the maximum size of a TCP segment (MSS = MTU - packet headers = 1460 bytes),
MTU is the maximum size of the transmitted OSI lower layer block (Ethernet MTU = 1500 bytes),
RTT - time of bilateral delay (from one end to another and back, from the English. Round Trip Time)
Ploss - Loss probability.
It can be noted that the formula is not valid with Ploss = 0. This is normal, as there are always packet losses in the real world.
img4 Impact of delay and packet loss on maximum TCP throughput
Most providers will not guarantee a loss of less than 0.01% (1 package out of 10'000).
You can check protocol statistics using the “netstat –s” command.
a) protocol enhancements
In this regard, protocol extensions have been developed, described in the TCP Extensions for High Performance ( RFC1323 ) standard , which are designed to solve constraints.
Among them:
- TCP Window Scale Option: the ability to increase the size of the window to 2 ^ 30 (1 GB)
- TCP selective acknowledgment ( SACK ) options: the receiving party specifies which packets in the stream are confirmed (positive or negative) ( RFC2018 ),
- TCP timestamps : Improved RTT (Round Trip Time Measurement - RTTM) measurements, preventing overlapping of sequence numbers ACK (Prevention Against Wrapped Sequence numbers - PAWS),
- Path MTU discovery : determining the maximum MTU all the way,
- Explicit Congestion Notification ( ECN ): Indicates path overload without dropping packets ( RFC3168 ).
You can check the current TCP / IP settings of your computer here .
b) Adaptation of operational systems
Despite the fact that RFC1323 was published back in 1992, the OSs did not implement the changes immediately.
OS Windows
RFC1323 support appeared since Windows 2000 (XP, Server 2003) and to activate the option you need to rotate in the registry.
Windows Server 2008, Vista, 7 includes a new implementation of the TCP / IP protocol stack, known as the new generation TCP / IP protocol stack (“Next Generation TCP / IP Stack”). It is designed to provide Windows networking technology for several years to come. Among the innovations:
- auto tuning of the receiving window (Receive Window Auto-Tuning),
- compound TCP: solves the problem of poor performance in high-bandwidth networks with the help of a new algorithm, instead of algorithms used on other platforms,
- enhancements for high loss environments and whatnot.
Many options are enabled by default. Configuration via command line.
Detailed configuration procedures for various operating systems (Windows XP, FreeBSD, Linux, Solaris, Mac OS X) can be found on the site of the Pittsburgh supercomputer center .
Note: Although routers basically pass all traffic (between different networks), they have only an indirect relation to TCP optimization (in some cases), since they work at a lower level of the OSI network model and perform the function of determining the best routes and only delivering packets to destination.
In the network there are many methods for measuring the speed of connection to the Internet.
Here we will look at the case of using the nuttcp utility ("New TTCP"), since it has several nice advantages:
- simple and effective method of measuring bandwidth via TCP or UDP,
- cross-platform single-file program (CLI),
- the ability to verify the effectiveness of the local TCP / IP stack (loopback),
- stable server operation (correct termination of TCP sessions), without hangs and crashes
(as in the case of iperf)
- client's work from NAT.
A bit of history: In 1980, Mike Muuss (ping-a) created ttcp ("Test TCP") - one of the first TCP bandwidth testing tools. Many changes have since been created in various implementations and with new features. Nuttcp is one of them. Last beta is April 2010.
Testing works according to the client-server scheme.
Payload is measured - payload (without headers).
Connection via port 5000. Data transfer - 5001 (and higher if you specify a multi-threaded test).
Server - just specify #nuttcp -S
Client - you can specify many options .
Example:
FreeBSD server, Windows XP SP3 client, FastEthernet (100Mbps).
server-ip # nuttcp –S
client options:
-w128 - TCP receive window size = 128 KB
-r - receiving (for client)
-F - fixes possible connection problems if you are in NAT
-i5 - show result every 5 seconds
-T15 - test duration (15 seconds).
TX% and RX% are the processor load on the transmitter and receiver.
Performance check for hardware and TCP / IP OS stack:
C: \> nuttcp.exe -w1m 127.0.0.1
205.0625 MB / 10.00 sec = 172.0189 Mbps 19% TX 12% RX
Examples of results:
- Intel Core 2 Duo (2 core) @ 1.6 GHz / 1 GB RAM / Windows XP = 1300/1400 Mbps
- AMD Athlon X2 Dual-Core 4600 @ 2.4 GHz / 2 GB RAM / Windows 7 = 2000/2100 Mbps
- Intel XEON X5650 (24 cores) @ 2.67GHz / 8 GB RAM / FreeBSD = 16600/18000 Mbps
Check Download (from server to client) :
C: \> nuttcp.exe -w128 -r -F –i5 -T15 server-ip
56.0166 MB / 5.00 sec = 93.9803 Mbps
56.0575 MB / 5.00 sec = 94.0489 Mbps
56.0338 MB / 5.00 sec = 94.0090 Mbps
168.2676 MB / 15.00 sec = 94.1020 Mbps 3% TX 10% RX
some-unix-client # nuttcp -r -F -i5 -T15 server-ip
429.0000 MB / 5.02 sec = 717.3541 Mbps
526.0000 MB / 5.00 sec = 882.0518 Mbps
1371.1741 MB / 15.00 sec = 766.6703 Mbps 26% TX 39% RX 3153 host-retrans 0.29 msRTT
Check Upload (from client to server) :
C: \> nuttcp.exe -w128 –i5 -T15 server-ip
55.6250 MB / 5.00 sec = 93.3169 Mbps
55.8125 MB / 5.00 sec = 93.6562 Mbps
55.6875 MB / 5.00 sec = 93.4277 Mbps
167.2500 MB / 15.12 sec = 92.7664 Mbps 17% TX 6% RX
some-unix-client # nuttcp -i5 -T15 server-ip
422.9375 MB / 5.00 sec = 709.5294 Mbps
420.6875 MB / 5.00 sec = 705.9357 Mbps
456.3750 MB / 5.00 sec = 765.6674 Mbps
1305.3853 MB / 15.06 sec = 727.0077 Mbps 20% TX 48% RX 24478 host-retrans 0.29 msRTT
In the process of testing, you need to remember that the maximum speed of an individual TCP connection is determined by various factors:
- maximum throughput of the slowest part of the road,
- time between sending a request and receiving a response (RTT),
- most delays at long distances are caused by the speed of light in the fiber (~ 200 km / ms),
- additional delays may occur at the time of network or device overload (server, router, PC),
- automatic speed reduction upon detection of packet loss (standard TCP overload prevention mechanism),
- the absence of other negative effects (the minimum number of bit errors at the physical level (Bit Error Rate <10 ^ -8),
- the health of the network card and the correct operation of the driver),
- one physical link can carry many simultaneous TCP connections,
- one host can have several simultaneous connections, even with the same remote host (you can quickly check with TCPView or TCPEye ).
Popular speedtest usually work in a browser, flash + geo-location to the nearest available server, which creates:
- additional load on the local machine (CPU, memory) and network (WWW headers, etc.),
- server selection may not be optimal,
- the possibility of erroneous results.
And lastly, I would like to note the frequently encountered problems with low network performance:
- narrow TCP window (Window Size),
- Ethernet Duplex mismatch,
- bad network cable.
Related Links:
technet.microsoft.com/en-us/magazine/2007.01.cableguy.aspx
fasterdata.es.net/fasterdata/host-tuning
1) TCP restrictions
Data transfer rate depends on network and system characteristics.
img1 The process of transferring data over the network
a) Buffers
The original TCP configuration limits the transfer rate to the buffer (the Window Size option is “window size”) and is a field with a size of 2 ^ 16 bytes (up to 64 KB). Maximum throughput in this case:
Example: You have a 100 megabyte connection to the Internet and there is a 100 ms delay to the server.
Standard TCP stack, maximum data transfer rate does not exceed 10 Mbps
(524288 bits / 0.1 sec = 5.24 Mbps in spite of the fact that you have a 100 megabit link).
b) Bandwidth-delay product (BDP)
TCP performance in principle depends not so much on the speed of the channel as on the so-called “bandwidth * delay product” or BDP (the result is bandwidth * delay), which is the number of bytes needed for the sender and receiver for maximally filling the TCP connection.
Performance problems arise in cases of so-called “long and wide pipes” (LFN “long fat network”), since the BDP in this case exceeds the TCP window size, thereby limiting the transmission rate.
img2 Effect of delay on maximum TCP throughput
')
An example would be the mobile Internet or a fast optical link.
Example of BDP calculations:
a) Mobile broadband internet: 10 Mb / s, 100 ms RTT
BĂ—D = 10^7 b/s Ă— 10^-1 s = 10^6 b, or 1 Mb / 125 kBb) high-speed ground network: 1 Gb / s, 10 ms RTT
BĂ—D = 10^9 b/s Ă— 10^-2 s = 10^7 b, or 10 Mb / 1.25 MBYou can calculate BDP here .
The “TCP window size” must exceed the BDP in order to reach the maximum channel load.
c) Protocol Overhead
By some estimates, about 95% of the computers in the world are connected via Ethernet technology.
Ethernet MTU (Ethernet frame payload) = max. 1500 bytes.
If we take into account all the Ethernet, IP, TCP headers, the picture will look like this:
img3 Single Ethernet frame transmission
The numbers indicate the size (in bytes) of the header for a particular protocol.
IFG (Interframe gap) - mandatory interframe space.
Headers: preamble, frame delimiter, Ethernet header / FCS - 26 bytes, IFG - 12 bytes, IP header - 20 bytes, TCP header - 20 bytes.
If you exclude VLAN tagging, TCP timestamp and other optional features, the maximum payload (TCP) on Ethernet networks will be:
Max TCP Payload = (MTU – TCP – IP) / (MTU + Ethernet + IFG) = (1500–40) / (1500 + 26 + 12) = 94.9%
d) Packet Delay and Loss
Since we are talking about reliable transmission of information, the loss of packets on the network forces TCP to resend segments and directly affects the reduction in speed.
The dependence of TCP speed in relation to packet loss is determined by the Mathis formula:
where: MSS (Maximum segment size) - the maximum size of a TCP segment (MSS = MTU - packet headers = 1460 bytes),
MTU is the maximum size of the transmitted OSI lower layer block (Ethernet MTU = 1500 bytes),
RTT - time of bilateral delay (from one end to another and back, from the English. Round Trip Time)
Ploss - Loss probability.
It can be noted that the formula is not valid with Ploss = 0. This is normal, as there are always packet losses in the real world.
img4 Impact of delay and packet loss on maximum TCP throughput
Most providers will not guarantee a loss of less than 0.01% (1 package out of 10'000).
You can check protocol statistics using the “netstat –s” command.
2) TCP optimization
a) protocol enhancements
In this regard, protocol extensions have been developed, described in the TCP Extensions for High Performance ( RFC1323 ) standard , which are designed to solve constraints.
Among them:
- TCP Window Scale Option: the ability to increase the size of the window to 2 ^ 30 (1 GB)
- TCP selective acknowledgment ( SACK ) options: the receiving party specifies which packets in the stream are confirmed (positive or negative) ( RFC2018 ),
- TCP timestamps : Improved RTT (Round Trip Time Measurement - RTTM) measurements, preventing overlapping of sequence numbers ACK (Prevention Against Wrapped Sequence numbers - PAWS),
- Path MTU discovery : determining the maximum MTU all the way,
- Explicit Congestion Notification ( ECN ): Indicates path overload without dropping packets ( RFC3168 ).
You can check the current TCP / IP settings of your computer here .
b) Adaptation of operational systems
Despite the fact that RFC1323 was published back in 1992, the OSs did not implement the changes immediately.
OS Windows
RFC1323 support appeared since Windows 2000 (XP, Server 2003) and to activate the option you need to rotate in the registry.
Windows Server 2008, Vista, 7 includes a new implementation of the TCP / IP protocol stack, known as the new generation TCP / IP protocol stack (“Next Generation TCP / IP Stack”). It is designed to provide Windows networking technology for several years to come. Among the innovations:
- auto tuning of the receiving window (Receive Window Auto-Tuning),
- compound TCP: solves the problem of poor performance in high-bandwidth networks with the help of a new algorithm, instead of algorithms used on other platforms,
- enhancements for high loss environments and whatnot.
Many options are enabled by default. Configuration via command line.
Detailed configuration procedures for various operating systems (Windows XP, FreeBSD, Linux, Solaris, Mac OS X) can be found on the site of the Pittsburgh supercomputer center .
Note: Although routers basically pass all traffic (between different networks), they have only an indirect relation to TCP optimization (in some cases), since they work at a lower level of the OSI network model and perform the function of determining the best routes and only delivering packets to destination.
3) Measuring TCP / IP stack performance
In the network there are many methods for measuring the speed of connection to the Internet.
Here we will look at the case of using the nuttcp utility ("New TTCP"), since it has several nice advantages:
- simple and effective method of measuring bandwidth via TCP or UDP,
- cross-platform single-file program (CLI),
- the ability to verify the effectiveness of the local TCP / IP stack (loopback),
- stable server operation (correct termination of TCP sessions), without hangs and crashes
(as in the case of iperf)
- client's work from NAT.
A bit of history: In 1980, Mike Muuss (ping-a) created ttcp ("Test TCP") - one of the first TCP bandwidth testing tools. Many changes have since been created in various implementations and with new features. Nuttcp is one of them. Last beta is April 2010.
Testing works according to the client-server scheme.
Payload is measured - payload (without headers).
Connection via port 5000. Data transfer - 5001 (and higher if you specify a multi-threaded test).
Server - just specify #nuttcp -S
Client - you can specify many options .
Example:
FreeBSD server, Windows XP SP3 client, FastEthernet (100Mbps).
server-ip # nuttcp –S
client options:
-w128 - TCP receive window size = 128 KB
-r - receiving (for client)
-F - fixes possible connection problems if you are in NAT
-i5 - show result every 5 seconds
-T15 - test duration (15 seconds).
TX% and RX% are the processor load on the transmitter and receiver.
Performance check for hardware and TCP / IP OS stack:
C: \> nuttcp.exe -w1m 127.0.0.1
205.0625 MB / 10.00 sec = 172.0189 Mbps 19% TX 12% RX
Examples of results:
- Intel Core 2 Duo (2 core) @ 1.6 GHz / 1 GB RAM / Windows XP = 1300/1400 Mbps
- AMD Athlon X2 Dual-Core 4600 @ 2.4 GHz / 2 GB RAM / Windows 7 = 2000/2100 Mbps
- Intel XEON X5650 (24 cores) @ 2.67GHz / 8 GB RAM / FreeBSD = 16600/18000 Mbps
Check Download (from server to client) :
C: \> nuttcp.exe -w128 -r -F –i5 -T15 server-ip
56.0166 MB / 5.00 sec = 93.9803 Mbps
56.0575 MB / 5.00 sec = 94.0489 Mbps
56.0338 MB / 5.00 sec = 94.0090 Mbps
168.2676 MB / 15.00 sec = 94.1020 Mbps 3% TX 10% RX
some-unix-client # nuttcp -r -F -i5 -T15 server-ip
429.0000 MB / 5.02 sec = 717.3541 Mbps
526.0000 MB / 5.00 sec = 882.0518 Mbps
1371.1741 MB / 15.00 sec = 766.6703 Mbps 26% TX 39% RX 3153 host-retrans 0.29 msRTT
Check Upload (from client to server) :
C: \> nuttcp.exe -w128 –i5 -T15 server-ip
55.6250 MB / 5.00 sec = 93.3169 Mbps
55.8125 MB / 5.00 sec = 93.6562 Mbps
55.6875 MB / 5.00 sec = 93.4277 Mbps
167.2500 MB / 15.12 sec = 92.7664 Mbps 17% TX 6% RX
some-unix-client # nuttcp -i5 -T15 server-ip
422.9375 MB / 5.00 sec = 709.5294 Mbps
420.6875 MB / 5.00 sec = 705.9357 Mbps
456.3750 MB / 5.00 sec = 765.6674 Mbps
1305.3853 MB / 15.06 sec = 727.0077 Mbps 20% TX 48% RX 24478 host-retrans 0.29 msRTT
4) Instead of conclusion
In the process of testing, you need to remember that the maximum speed of an individual TCP connection is determined by various factors:
- maximum throughput of the slowest part of the road,
- time between sending a request and receiving a response (RTT),
- most delays at long distances are caused by the speed of light in the fiber (~ 200 km / ms),
- additional delays may occur at the time of network or device overload (server, router, PC),
- automatic speed reduction upon detection of packet loss (standard TCP overload prevention mechanism),
- the absence of other negative effects (the minimum number of bit errors at the physical level (Bit Error Rate <10 ^ -8),
- the health of the network card and the correct operation of the driver),
- one physical link can carry many simultaneous TCP connections,
- one host can have several simultaneous connections, even with the same remote host (you can quickly check with TCPView or TCPEye ).
Popular speedtest usually work in a browser, flash + geo-location to the nearest available server, which creates:
- additional load on the local machine (CPU, memory) and network (WWW headers, etc.),
- server selection may not be optimal,
- the possibility of erroneous results.
And lastly, I would like to note the frequently encountered problems with low network performance:
- narrow TCP window (Window Size),
- Ethernet Duplex mismatch,
- bad network cable.
Related Links:
technet.microsoft.com/en-us/magazine/2007.01.cableguy.aspx
fasterdata.es.net/fasterdata/host-tuning
Source: https://habr.com/ru/post/115527/
All Articles