Separation of control and data plane in network equipment

In the operation of a network device, two abstractions can be distinguished - the control level (control plane) and the transmission level (data plane). ontrol plane is responsible for the logic of the network device to further ensure the possibility of packet transmission (filling in various tables, for example, routing, working out various service protocols ARP / STP /, etc.). Data plane, in turn, is directly responsible for the transmission of useful traffic through our network device. Those. The control plane provides us with information on where and how to send network traffic, and the data plane is already fulfilling its tasks. These abstractions can be distinguished at both the logical and physical levels. But is there always such a separation on the network equipment and where exactly are the functions of each of the abstractions performed? Let's try to figure it out.

This division appeared quite a long time ago in order to improve the performance of network devices. It became clear that using one abstraction to control and transmit network traffic is inefficient. The control level has a rather complicated logic of work and does not perform a huge number of operations per second. The transmitting level, on the contrary, performs relatively uniform operations, but there are a lot of them. Thus, intelligent hardware is required for the control level, and high-performance hardware is required for the transmitter. Since it is difficult and often expensive to achieve both parameters on a single chip, it was decided to separate the logic of network devices. This would allow to implement complex logic, for example, on the basis of general-purpose processors, and to get high performance on specialized microcircuits. Before the manufacturers of network equipment on one side of the scale is functionality and performance, and on the second - the cost of the solution. I propose to consider the workplaces of the control and transmission levels on the example of various types of network devices: switches and routers. Working as a network engineer, you do not always delve deeply into the hardware of the device. But an understanding of the general principles of architecture is necessary for proper selection of devices, network design, and an approach to solving network problems.
')
If we look at the software-defined networks (Software-defined Network - SDN), we find that the control level is fully or partially transferred to the dedicated device in general. However, the consideration of such decisions will remain outside the scope of this article.

Switches

Let's start with the switches, because here we get the most visible separation of our abstractions. The main thing for the switch is the data transfer rate. All packet processing must be implemented at the port speed (wire-speed), otherwise the switch will be an inhibiting element in our network. In this regard, it is in the switches that we can detect the implementation of the transmitting level on individual chips - ASICs (ASIC is a special-purpose integrated circuit). In fact, on the switch, the control level is performed on the basis of a general-purpose processor, and the transmitting level, as we have already noted, is based on the ASIC.

Processors that are installed in network equipment, often have differences from those that are in our PCs and servers. These are most often specialized processors that are designed for use within various devices (network, storage systems, etc.) and belong to the class of embedded processors. Usually they are small, consume some energy and are part of a single-chip system (System on a chip - SoC). SoC - almost full-fledged computer, made on the basis of a single chip (with (micro) processor, RAM, I / O controller, interfaces, etc.). Some of these processors are sharpened to perform operations in network devices, others have a wider range of applications. In this case, most often they can be run on, for example, some solutions based on Unix / Linux, since they still remain primarily general-purpose processors.

Classic ASIC has a predefined set of functions that are performed by hardware. In fact, the general logic of packet processing is laid in the ASIC at the stage of chip production, which is quite difficult to change. In ASIC, we get an acceptable level of logic and at the same time high packet processing speed. Thus, high performance in the switch is achieved by performing the functions of the transmitting level on ASICs. And it is ASICs that are the cause of the relatively limited logic of the switch, which is difficult to change further. It would be possible to use FPGA (Field-Programmable Gate Array) microcircuits instead of ASIC, which can be reprogrammed. But they are expensive and energy intensive. Therefore, manufacturers of network equipment, in order not to increase the cost of their devices, on the one hand, are trying to transfer part of the packet processing to a general-purpose processor (that is, where the control level works), which does not always have a good effect on device performance. On the other hand, they are trying to modernize ASICs, making them more functional and even programmable (for example, Cisco UADP ASIC).

Typically, the switch is one or more ASICs. For example, for every 12/24 port put your ASIC. Programming the logic of the ASIC'a performs the control level. It is he who fills all the tables inside the ASIC (routes, access lists, etc.). An ASIC may have sufficient intelligence to switch packets within itself, or it can perform packet switching through an external bus / switch fabric. This architecture is used primarily in fixed configuration switches (non-modular). Examples of such switches can be Cisco Catalyst 2960/3650/3850.

The block diagrams of switches and routers in the article are simplified to focus attention primarily on the location of the control and transmission levels. They do not include all the structural elements of the devices.

If we are dealing with a modular switch (a switch in which you can install cards with different types of ports), the architecture can be more complex. More ports means more performance and more ASICs. There are at least two approaches to implementing the architecture of such switches.

In the first case, the transmitting level is performed centrally on dedicated ASICs, which are located on a separate board. In this case, ASICs on line cards are less intelligent and perform a very limited set of functions. Programming logic continues to be managed by the controlling level, which in turn is run on its hardware capacity (again, a general-purpose processor is used (and there may be several of them) located on a separate module — the supervisor). Examples of such switches are Cisco Catalyst 4500 and Cisco Catalyst 6500/6800 (centralized switching).


* Port ASIC microcircuits installed on line cards are not very intelligent and perform a very limited set of functions.

A variant is possible, where on each module with linear ports, there is its own specialized transfer card. In this case, each module has its transmitting level, which improves the performance of the entire “box”. We can say that this is an intermediate option between the first and second approaches to the implementation of the modular switch architecture. An example of such switches could be the Cisco Catalyst 6500/6800 (distributed switching).


* Port ASIC microcircuits installed on line cards are not very intelligent and perform a very limited set of functions.

The second approach is to use fairly intelligent ASICs on line cards. In this case, each ASIC can independently handle network traffic, performing the basic set of functions. Those. we immediately have a distributed transmitting level. This may be a more expensive solution, but often more productive. We also minimize with this architecture, the delay in packet transmission. An example of such a switch could be the Cisco Nexus 9500.



The architecture of modular switches is quite complex. In particular, several different ASICs can be used to implement the transmitting layer within a single line card. Each of them performs its own range of tasks or unites downstream ASICs. The switching factory can also be built on the basis of ASICs, which perform both communication functions between line cards and certain types of processing.

Note that in the switches we can have the distribution of the management level. For example, in the Cisco Nexus 9500 switch, the control level within one device is separated: some functions are performed on the supervisor, and some on the line port card (each board has its own general-purpose processor).

Up to this point all consideration went within the framework of one device. But many switches can merge into one logical device by means of stacking. If we have a stack of switches assembled, the control level is usually run on the main switch (it is also called the active / master). And the transmitting level will be launched separately on each switch in the stack. Those. via the stack communication channel, the control layer located on the main switch distributes control information to all switches in the stack for the operation of the transmitting layer locally. An example of such a work model is the Cisco StackWise stack or HPE IRF stack.

Routers

Let's now see how things are with our abstractions in routers. If we consider relatively low-cost routers, the control and transmission levels will run on the same hardware — a general-purpose processor (most often in SoC format). Processor time will be distributed in this case between both abstractions. We will not find any specialized microcircuits for the transmitting level, as it was in the switches. In this regard, we get a very flexible logic of the device, but not the most outstanding performance values ​​(tens and hundreds of megabits per second). Moreover, various vendor tricks (for example, Cisco Express Forwarding) are only the optimization of packet processing at the software level based on the standard hardware base. Examples of such devices are Cisco 800, 1900, 2900, etc. The situation changes if our general-purpose processor becomes multi-core (for example, in the Cisco ISR 4300), and there may be several such processors (for example, in the Cisco ISR 4400). In this case, the controlling and transmitting levels can be performed on different cores and processors. Moreover, the transmitting level is allocated several cores at once in order to obtain parallel processing of packets, which means to improve the performance of our device. It is worth noting that some cores can be given altogether under third-party services (of course, if the processor allows it).


Modern SoC have multi-core processors. 48 cores already surprise no one. And together with batch processing accelerators integrated into SoC, on the basis of a single SoC, you can get very good performance: there are SoC solutions on the market that allow you to process network traffic at speeds up to 40 Gbit / s.

A separate conversation - these are high-performance routers. In this case, the usual general-purpose processor capacity may not be enough. Network equipment manufacturers transfer the transmitting level to a separate hardware, more adapted to handle a large flow of traffic. In fact, we are going to switch architecture. But since the router is more functional, ordinary ASICs are not enough. In this regard, each manufacturer offers its own solutions.

One option is to use specialized network processors (Network Processor - NP or Network Processing Unit - NPU). Network processors are much more functional than ASICs, but at the same time more productive than general-purpose processors.

As an example, consider the Cisco ASR 1000 routers, where the basic functions of the transmitting layer are performed on a separate board. This card hosts one or two specialized Cisco QuantumFlow Processor (QFP) network processors that handle traffic directly. This processor has a RISC architecture and is designed specifically for transmitting traffic. The second-generation QFP includes up to 128 processor cores, each of which can run four separate processes. Those. we have up to 256 cores on a single board (in the case of two processors). Comparing with the architecture of simpler routers, where everything is performed on several cores, you can immediately conclude that such routers are more productive.

Network processors are manufactured by various companies (Cisco, EZChip, Broadcom, etc.) and are used to perform the functions of the transmitting level by many network equipment manufacturers. For example, network processors are used in Huawei equipment (both in routers, for example, in NetEngine40E, and in S12700 switches).

In addition to network processors, specialized chipsets are on the market, for example, the Juniper Trio chipset. They are positioned between network processors and ASICs. By and large, the general meaning is preserved - the transmitting level is performed on a specialized hardware, in this case the Trio Chipset chipset. Note that before the Trio chipset was introduced to the market, Juniper actively used programmable ASICs of its own design (Internet Processor ASIC and I-Chip) in its routers.

It should be noted that in the top solutions, we will have not only the separation of the levels of control and transmission between different iron, but also the distribution within each level. For example, in the Cisco ASR 9000 router, the control level within one device is separated: some of the functions are performed on the processor board, and some on the line port card. The same applies to the transmitting level: there are many network processors and they are located directly on the line cards.


Finally

Since even a single vendor has a lot of implementations of architectures, it is extremely difficult to consider them all. However, if we need high performance, most often the transmitting level will be performed on a specialized hardware: be it a network processor, a specialized chipset, a regular or programmable ASIC, or something else. In some devices, we will find even a combination of these chips. Often, manufacturers of network equipment in their equipment use third-party chips (for example, ASICs or network processors).