Introduction to Distributed Switch Architecture: Network Management Technology as a Single Device

The Promwad electronics design center team returns to Habr after the winter holidays with new articles on the development of embedded software and new devices for mass production. Today we will share our experience in the topic of network technologies.



The average home network, as well as the network of a small enterprise, is no longer just two or three computers connected through the first Chinese switchboard. Along with the growth of content (databases, streaming audio / video, etc.) and an increase in the number of devices (VoIP devices, servers, NAS, IP cameras, and in home networks - TVs and other “Internet of things” ) the amount of transmitted data through the network infrastructure is growing. Data streams need to be divided among themselves, while not forgetting about traffic prioritization: for example, it is desirable to start up VoIP traffic with a higher priority than IPTV, and IPTV in turn - than torrents. Therefore, it is not surprising that, over time, even small local area networks become more complex, and the port capacity of single switches becomes insufficient ...



To increase the capacity of ports of large network nodes, as well as to increase the overall network bandwidth, they most often resort to building so-called. stacks - combining switches into hierarchical structures, depending on the data flows.

')

The most "classical" network building architecture is a tree, as shown in the figure:







At the same time, traffic from node A to node D passes through a chain of upstream switches, which imposes additional performance requirements — the throughput of each next (upstream) node of the tree must be higher than the previous one.



In order to offload the main nodes of the network, most often they introduce additional connections between switches. For example, the “core” of a network is often organized as a ring of switches, as shown in the diagram below:







If this is done using the good old Ethernet, then in order to avoid ring overflow by broadcast traffic, the Spanning Tree Protocol is used, which places one of the links between the ring switches into an inactive state. This allows you to provide an alternative route for traffic in the event of a break in the central ring at any of the points. If any of the protocols specifically designed for “rings” are used, for example, FDDI or Token Ring, the ring is not broken, and traffic is transmitted over the ring with the most uniform load on the switches included in the ring.



And of course, no one forbids, if necessary, to combine different topologies within the same network, up to and including the use of multiple intersecting rings, each of whose nodes is a vertex of the tree.



When building a network, it becomes necessary not only to connect passive switch nodes, but also to ensure data transfer between nodes, taking into account priority and separation of different types of traffic. At the same time, it is desirable to maintain the possibility of centralized network management from one point to set these priorities. There is a need to combine all the network equipment in a logical entity.



Technology Distributed Switch Architecture provides such an opportunity, allowing you to manage an extensive network as a single device, as well as set the traffic paths of the entire system. At the same time, switch nodes can be simple single-chip devices with one control processor common to all nodes, which has a positive effect on the cost of the system and the cost of maintaining it.



The essence of the DSA technology can be briefly reduced to the introduction of an additional level of addressing within the nodes of the entire network, and in fact within the entire port capacity of all the nodes. As an example of this technology, consider the implementation of the Distributed Switch Architecture from one of the manufacturers of switch-microchips - Marvell. There are many implementations of this technology from different vendors (different manufacturers call this technology differently), but the essence of these solutions is similar.



For the successful operation of this technology, each switch node is assigned a unique Device ID, and each port of a single device is assigned a Port ID. Each packet entering into any external port of any network node that cannot be processed within this node (that is, based on local tables and rules) is supplemented with 4 bytes: a DSA tag. For normal traffic, Marvell devices use the Forward DSA tag. Device ID and Port ID are placed in this tag, which unambiguously determine where exactly within the network the packet got into the network. The tag itself is placed immediately after the Source Address in the MAC header.







Marvell's DSA implementation supports 802.1Q — if there is a VLAN header in the packet, its VID and priority are transferred to the DSA header, and the VLAN header from the packet is removed, that is, the packet size remains unchanged. A special flag in the new header indicates that the source packet was tagged, which allows you to restore the VLAN header at the output from the external port if necessary. When transferring the priority of a packet, you can use a special replacement table, so the distribution of packets on queues can be set arbitrarily.



Since, in the presence of addressing nodes, you can access each of them separately, for the entire network of switches you can use one common control CPU physically connected to only one of the nodes. The CPU is connected to one of the switch ports, and for the exchange, taking into account the additional addressing, either the DSA header mentioned above or the 4-byte extended header type Etherty DSA is used, which additionally includes the ethertype value. This ethertype is selected by the administrator and is set in the switch configuration, as well as in the kernel module on the control CPU. Using Ethertype DSA allows you to simultaneously transmit both traffic with a DSA tag and normal network traffic through a port.



Centralized management is one of the most important conditions when building a complex data network. Marvell's DSA implementation solves this problem with special packages - Management frames, which are used with two types of DSA tags: From_CPU and To_CPU. As the name implies, the first type is used to transfer packets from the CPU to the managed node, and the second type is from the node to the controlling CPU. Their main difference is that the From_DSA tag indicates the Target Device ID and Target Port ID of the device, and the To_DSA tag indicates the Source Device ID and Source Port ID. The fact is that each switch on the network knows which port (directly or through other nodes) is connected to the controlling CPU, which means that it suffices to address the device being controlled.



Additional features of Marvell's DSA implementation also include multi-chip traffic monitoring: traffic from any port on any network node can be duplicated to any other port, including the port of the managing CPU. At the same time, it is possible to monitor any number of ports, and to identify the source, a DSA tag of the type To_Sniffer is used, in which the Source Device ID, Source Port ID and the VLAN number of the incoming packet are recorded.



Starting with version 2.6.28, the Linux kernel mainline supports the use of Distributed Switch Architecture on network interfaces. Initially, only some models of Marvell switches were supported, but then Broadcom chip support was also added. There is also a patch to support Micrel switches. Support consists of an additional layer in the network stack, which adds to the packet, or removes the DSA header from it, sending data to the virtual network interface, depending on the header data. Thus, from the point of view of the operating system, each external port of all the switches of the DSA network (except for the DSA ports that the switches are connected to each other) is “visible” as a separate network interface. If there is support from the hardware side, you can assign your own MAC and IP address to this interface, which means you can also build a router with a large port capacity using DSA.



It is worth noting that DSA implementations from different vendors are not compatible with each other, and also have different functionality. For example, the Micrel KSZ8993M can additionally address only the port number of one device directly connected to the control CPU. Therefore, in the case of using the network using the described technology, you will have to choose any one chip maker.



Consider an example of a DSA description in the Device Tree configuration for kernel 3.10 or higher:

dsa@0 { compatible = "marvell,dsa"; #address-cells = <2>; #size-cells = <0>; interrupts = <10>; dsa,ethernet = <&ethernet0>; dsa,mii-bus = <&mii_bus0>; switch@0 { #address-cells = <1>; #size-cells = <0>; reg = <16 0>; /* MDIO address 16, switch 0 in tree */ port@2 { reg = <2>; label = "lan2"; }; port@4 { reg = <4>; label = "lan4"; }; port@5 { reg = <5>; label = "cpu"; }; switch0uplink: port@6 { reg = <6>; label = "dsa"; link = <&switch1uplink>; }; }; switch@1 { #address-cells = <1>; #size-cells = <0>; reg = <17 1>; /* MDIO address 17, switch 1 in tree */ port@1 { reg = <1>; label = "lan1"; }; port@3 { reg = <3>; label = "lan3"; }; switch1uplink: port@5 { reg = <5>; label = "dsa"; link = <&switch0uplink>; }; }; 

In this case, the architecture of two switches is described, which can be represented as follows:







Since two external switches define 2 external ports, with this configuration, the operating system will additionally create 4 virtual network interfaces, which are ports in which computers A, B, C and D are included, each of which can be used independently. In this case, by default, a separate independent MAC table will be allocated for each port of each switch.



In this case, the configuration of each individual switch can be made locally via the MDIO bus (for example, if all the equipment is located within one unit, or the chips are located on the same printed circuit board), or with the help of special configuration packets via the Ethernet network. Thanks to this, it is possible to build both a switch with a large port capacity and a complex network of switches with centralized control.



It is worth noting that the current DSA implementation in the kernel only supports the configuration of switches via the MDIO bus, which somewhat limits the scope of technology use. However, if necessary, adding a remote control is not difficult.



On this theoretical part of the description of Distributed Switch Architecture can be considered complete, next time we will try to figure out how to use this technology in practice. So if network technologies and electronics development are your topic, join our Habré subscribers .