OvS, DPDK, and Intel HT: 12x Virtual Switch Acceleration
Today we will tell about the Open vSwitch integrated with the Data Plane Development Kit (OvS-DPDK). This is a high-performance open source virtual multi-layer switch available under the Apache 2.0 license. The integration of OvS with DPDK can significantly increase the speed of processing network packets.
Open vSwitch has found practical application in many projects. The switch supports the OpenFlow protocol, suitable for work in software-defined networks (Software Defined Network, SDN). OvS can be configured using the OVSDB control protocol.
The usual Open vSwitch deals with the transmission of network packets through the Linux kernel space (this is shown in the figure below). In the kernel data channel, the “fast path” (“fastpath”) of data processing is implemented through a simple flow table containing the rules for forwarding and processing received packets. If the first packet in the stream does not match any of the existing entries in the kernel table, it is sent for processing to a daemon that runs in user space. This is the so-called “slow path”. After this packet is processed in user space, the daemon updates the flow table in the kernel space, as a result, the following packets in the flow can go through the “fast path”, they do not need to be passed to user space. Following this approach, the usual Open vSwitch eliminates the resource-intensive context switching operation between the kernel and user spaces for a large percentage of packets received. However, the achievable bandwidth is limited by the bandwidth of the Linux network stack, which is not suitable for use cases that require high packet processing speed. For example - in telecommunications companies.
DPDK is a set of Linux user-space libraries that allow you to create high-performance network packet processing applications. In fact, it offers a set of polled mode drivers (PMD) for various physical and virtual devices that allow direct packet transmission between user space and the network interface, bypassing the kernel network stack. This gives a significant increase in performance compared to transferring data through the kernel. This happens due to eliminating the need to handle interrupts and the need to use the network kernel stack. When integrating OvS with DPDK, the “fast path” of switching passes through user space, and the “first packet path” is the same path in user space that packets travel in normal OvS when the “fast path” passes through the core.
Here is a general scheme for integrating DPDK with OvS:
')
Integration of DPDK data plane with ordinary Open vSwitch
The figure below shows the high-level architecture of OvS-DPDK. OvS switching ports are represented by network devices (netdevs). Netdev-dpdk is a network device using DPDK capabilities, it communicates network data through three different interfaces. The first is the physical interface (working with it is done using the librte_eth library in the DPDK), then two virtual interfaces (librte_vhost and librte_ring). These are interfaces with physical and virtual devices connected to a virtual switch.
High-level Open vSwitch architecture with DPDK
Other architectural levels of OvS provide additional functionality and organization of interaction, for example, with an SDN controller. Dpif-netdev provides data transfer services in user space, ofproto is an OvS library that implements an OpenFlow switch. It interacts with OpenFlow controllers over the network and with the switch software or hardware through the provider ofproto. The ovsdb server maintains up-to-date information in the switching tables for a given OvS instance and interacts with the SDN controller.
Details on the OvS architecture can be found on openvswitch.org .
Now let's talk about switching tables OvS-DPDK.
A packet arriving at the OvS-DPDK from a physical or virtual interface receives a unique identifier, or hash, based on its header fields, which are then compared with the entries in the three main switching tables. These are the exact match table (Exact Match Cache, EMC), the data transfer classifier table (Data Path Classifier, dpcls), and the ofproto classifier table (Ofrproto Classifier).
The packet identifier is compared with the table contents, in the order in which they are given here, until a match is found. If a match is found, the action indicated by the entry in the table containing the processing rule for the package will be performed. After completing all the actions on the packet, it leaves the switch. This process is shown in the figure below.
The hierarchy of switching tables in Open vSwitch with DPDK
These three tables have different bandwidth and latency characteristics. The EMC table offers the fastest processing for a limited number of entries. The packet identifier must exactly match the entry in the table. Among the compared data are the exact IP address of the source and port, the address and port of destination, the protocol. Matching all of this with writing to EMC will give the highest processing speed. Otherwise, no match will be found and a search will begin in the dpcls table.
There are many more entries in dpcls (they are organized in many sub-tables), it supports identifier comparison using patterns. For example, it can be used to process packets for which a target IP address and port are specified in it, but any source of such a packet is allowed. With dpcls, you have to maintain a much larger set of table entries, and as a result, if the decision to process a packet is made on the basis of this table, you can achieve about half the EMC bandwidth. If a suitable rule is found in dpcls, it is written to the EMC, as a result, the following packets with the same identifier can be processed at a higher speed.
If the rule for the package identifier is not found and in dpcls, the verification of the table of the classifier ofproto begins, as a result, the OpenFlow controller can decide how to process the package. This is the slowest data path; it is more than 10 times slower than when a match is found in the EMC table. When a match is found in the ofproto classifier, this leads to the creation of new entries in faster switching tables, as a result, subsequent packets of the same stream can be processed faster.
At the moment when this material was written, the following high-level features were available in the OvS-DPDK master branch:
The figure below shows the latest performance test results for conventional OvS and OvS-DPDK. This shows the bandwidth (packets per second) for the usage scenario, when data enters the OvS from the physical port, is processed, and transmitted to another physical port (Phy-OvS-Phy). You can see that OvS-DPDK works about 10 times faster than regular OvS. However, if Intel Hyper-Threading is enabled on the server, OvS-DPDK is about 12 times faster than normal OvS. In the diagram, this option is marked as “1C2T”, which means one physical core with two threads.
Performance comparison of conventional OvS and OvS-DPDK
Similar results were obtained in more complex use cases: (Phy-OvS-VM-OvS-Phy), when data from the physical interface goes to OvS, then goes to a virtual machine, then returns to OvS, and, as a result, is transferred to physical interface. Namely, here the OvS-DPDK achieves an approximately 9-fold increase in productivity.
Details of the hardware and software configuration on which the tests were conducted, the results of which are mentioned above, can be found in this report . There is also test data for other use cases.
OvS-DPDK can be found in the upstream branches of the openvswitch.org repository, it is available in various Linux distributions. The latest stable release is OvS 2.6. It was released in September 2016. Similar releases are published every six months.
Please note that you can upload a zip file with the master OVS branch, or a branch of version 2.6 . There are also installation instructions for the master branch and for version 2.6 . OvS packages with SPDK are available in the following repositories:
Normal Open vSwitch can be downloaded on openvswitch.org , on GitHub , it can be found in Linux distributions.
We have presented a brief overview of OvS-DPDK - a virtual switch that is characterized by high performance and reliability. Integration with DPDK significantly accelerates the usual Open vSwitch. If you want to know more about OvS, here is a list of useful materials in English.
User Guides:
Developer Guides:
Articles:
Webinars on stable OvS releases with DPDK:
INB University materials:
Technical documentation:
Open vSwitch has found practical application in many projects. The switch supports the OpenFlow protocol, suitable for work in software-defined networks (Software Defined Network, SDN). OvS can be configured using the OVSDB control protocol.
OvS-DPDK high-level architecture
The usual Open vSwitch deals with the transmission of network packets through the Linux kernel space (this is shown in the figure below). In the kernel data channel, the “fast path” (“fastpath”) of data processing is implemented through a simple flow table containing the rules for forwarding and processing received packets. If the first packet in the stream does not match any of the existing entries in the kernel table, it is sent for processing to a daemon that runs in user space. This is the so-called “slow path”. After this packet is processed in user space, the daemon updates the flow table in the kernel space, as a result, the following packets in the flow can go through the “fast path”, they do not need to be passed to user space. Following this approach, the usual Open vSwitch eliminates the resource-intensive context switching operation between the kernel and user spaces for a large percentage of packets received. However, the achievable bandwidth is limited by the bandwidth of the Linux network stack, which is not suitable for use cases that require high packet processing speed. For example - in telecommunications companies.
DPDK is a set of Linux user-space libraries that allow you to create high-performance network packet processing applications. In fact, it offers a set of polled mode drivers (PMD) for various physical and virtual devices that allow direct packet transmission between user space and the network interface, bypassing the kernel network stack. This gives a significant increase in performance compared to transferring data through the kernel. This happens due to eliminating the need to handle interrupts and the need to use the network kernel stack. When integrating OvS with DPDK, the “fast path” of switching passes through user space, and the “first packet path” is the same path in user space that packets travel in normal OvS when the “fast path” passes through the core.
Here is a general scheme for integrating DPDK with OvS:
')
Integration of DPDK data plane with ordinary Open vSwitch
The figure below shows the high-level architecture of OvS-DPDK. OvS switching ports are represented by network devices (netdevs). Netdev-dpdk is a network device using DPDK capabilities, it communicates network data through three different interfaces. The first is the physical interface (working with it is done using the librte_eth library in the DPDK), then two virtual interfaces (librte_vhost and librte_ring). These are interfaces with physical and virtual devices connected to a virtual switch.
High-level Open vSwitch architecture with DPDK
Other architectural levels of OvS provide additional functionality and organization of interaction, for example, with an SDN controller. Dpif-netdev provides data transfer services in user space, ofproto is an OvS library that implements an OpenFlow switch. It interacts with OpenFlow controllers over the network and with the switch software or hardware through the provider ofproto. The ovsdb server maintains up-to-date information in the switching tables for a given OvS instance and interacts with the SDN controller.
Details on the OvS architecture can be found on openvswitch.org .
Now let's talk about switching tables OvS-DPDK.
Switching tables OvS-DPDK
A packet arriving at the OvS-DPDK from a physical or virtual interface receives a unique identifier, or hash, based on its header fields, which are then compared with the entries in the three main switching tables. These are the exact match table (Exact Match Cache, EMC), the data transfer classifier table (Data Path Classifier, dpcls), and the ofproto classifier table (Ofrproto Classifier).
The packet identifier is compared with the table contents, in the order in which they are given here, until a match is found. If a match is found, the action indicated by the entry in the table containing the processing rule for the package will be performed. After completing all the actions on the packet, it leaves the switch. This process is shown in the figure below.
The hierarchy of switching tables in Open vSwitch with DPDK
These three tables have different bandwidth and latency characteristics. The EMC table offers the fastest processing for a limited number of entries. The packet identifier must exactly match the entry in the table. Among the compared data are the exact IP address of the source and port, the address and port of destination, the protocol. Matching all of this with writing to EMC will give the highest processing speed. Otherwise, no match will be found and a search will begin in the dpcls table.
There are many more entries in dpcls (they are organized in many sub-tables), it supports identifier comparison using patterns. For example, it can be used to process packets for which a target IP address and port are specified in it, but any source of such a packet is allowed. With dpcls, you have to maintain a much larger set of table entries, and as a result, if the decision to process a packet is made on the basis of this table, you can achieve about half the EMC bandwidth. If a suitable rule is found in dpcls, it is written to the EMC, as a result, the following packets with the same identifier can be processed at a higher speed.
If the rule for the package identifier is not found and in dpcls, the verification of the table of the classifier ofproto begins, as a result, the OpenFlow controller can decide how to process the package. This is the slowest data path; it is more than 10 times slower than when a match is found in the EMC table. When a match is found in the ofproto classifier, this leads to the creation of new entries in faster switching tables, as a result, subsequent packets of the same stream can be processed faster.
About the features and performance of OvS-DPDK
At the moment when this material was written, the following high-level features were available in the OvS-DPDK master branch:
- DPDK support 16.07 (when new releases are released DPDK is updated and the version supported by OvS)
- Support for vHost-user ports.
- Support for reconnecting vHost.
- Support for multiple queues vHost.
- Native tunneling support: VxLAN, GRE, Geneve
- VLAN support
- MPLS support
- Ingress / egress QOS Policy Support
- Jumbo frame support
- Connection tracking
- Statistical information: DPDK vHost data and extended DPDK statistics
- Debugging: support DPDK pdump
- Channel aggregation
- Channel Status Information
- VFIO support
- ODD / OpenStack DPDK port detection
- Accounting for NUMA features when working with vHost-user ports
The figure below shows the latest performance test results for conventional OvS and OvS-DPDK. This shows the bandwidth (packets per second) for the usage scenario, when data enters the OvS from the physical port, is processed, and transmitted to another physical port (Phy-OvS-Phy). You can see that OvS-DPDK works about 10 times faster than regular OvS. However, if Intel Hyper-Threading is enabled on the server, OvS-DPDK is about 12 times faster than normal OvS. In the diagram, this option is marked as “1C2T”, which means one physical core with two threads.
Performance comparison of conventional OvS and OvS-DPDK
Similar results were obtained in more complex use cases: (Phy-OvS-VM-OvS-Phy), when data from the physical interface goes to OvS, then goes to a virtual machine, then returns to OvS, and, as a result, is transferred to physical interface. Namely, here the OvS-DPDK achieves an approximately 9-fold increase in productivity.
Details of the hardware and software configuration on which the tests were conducted, the results of which are mentioned above, can be found in this report . There is also test data for other use cases.
Where to get OvS-DPDK
OvS-DPDK can be found in the upstream branches of the openvswitch.org repository, it is available in various Linux distributions. The latest stable release is OvS 2.6. It was released in September 2016. Similar releases are published every six months.
Please note that you can upload a zip file with the master OVS branch, or a branch of version 2.6 . There are also installation instructions for the master branch and for version 2.6 . OvS packages with SPDK are available in the following repositories:
Normal Open vSwitch can be downloaded on openvswitch.org , on GitHub , it can be found in Linux distributions.
Results and additional materials
We have presented a brief overview of OvS-DPDK - a virtual switch that is characterized by high performance and reliability. Integration with DPDK significantly accelerates the usual Open vSwitch. If you want to know more about OvS, here is a list of useful materials in English.
User Guides:
Developer Guides:
Articles:
- Rate limiting configuration and usage for OvS with DPDK
- QoS configuration and usage for OvS with DPDK
- Configure vHost User multiqueue for OvS with DPDK
- vHost User NUMA awareness in DPDK
- DPDK Pdump in OvS with DPDK
- Enabling OvS with DPDK in OpenStack
- Jumbo Frames in Open vSwitch * with DPDK
- vHost User Client Mode in Open vSwitch * with DPDK
- OVS-DPDK Datapath Classifier
Webinars on stable OvS releases with DPDK:
INB University materials:
Technical documentation:
Source: https://habr.com/ru/post/316004/
All Articles