Own traffic exchange point in the data center. Part 1. How it works
In the 90s, when the Russian Internet was only emerging, providers from one city had to exchange traffic through Europe and pay for it to transit and higher operators (uplinks). The end user in the end received high prices for Internet access, restrictions on the amount of available traffic, a great response time of web resources. In order not to drive national traffic to foreign cities and villages, local providers began to unite and create traffic exchange points in major cities.
Today we will talk about them: let's tell you why we needed our own DataLine-IX traffic exchange point, how it is arranged, and share the first results.
ISP-A and ISP-B providers exchange their network traffic through a traffic exchange point. Without it, the exchange would occur through the “big” Internet.
The Internet Exchange Point (Internet Exchange Point, IX) is an infrastructure that allows autonomous systems to exchange traffic (peering) directly, bypassing higher-level telecom operators (uplinks). Peering helps IX participants reduce packet transfer routes between networks and reduce traffic costs. If you need to connect with several participants at once, then through a traffic exchange point it is easier to do: instead of a separate connection to each participant, one to IX, where these participants are present. This is the organizational side. Financially, participant IX also benefits from savings on channels and the purchase of traffic from uplinks.
')
For the end user, there are also pluses: network delays are reduced, the response time of resources on the Internet is reduced.
Traffic exchange points are a positive thing. The more of them, the better the connectivity of the Internet and the more accessible it is for the end user.
The first traffic exchange points began to appear in 1994 in major European cities: London (LINX), Frankfurt (DE-CIX), Amsterdam (AMS-IX), Moscow (MSK-IX). Now around the world there are about 580 IX.
The largest traffic exchange points in Russia. Source: internetexchangemap.com
When in 2015 we thought about our own traffic exchange point, more than 40 operators were present in our data centers (there are 53 of them now). Worked Meet-Me-Room, which simplifies connection to any operator on our sites. Its own traffic exchange point would give even more opportunities for our customers, especially for Internet providers and content generators (gaming services, CDN, video hosting, media, social networks):
The infrastructure of the DataLine-IX traffic exchange point is distributed over two sites - OST and NORD.
There are 6 Extreme BlackDiamond X8 switches at two sites. Two of them connect data centers into a single network and form the core of the DataLine-IX network infrastructure (Core). Two access points (Access) from each site are connected to the core. New members join the traffic exchange point through these access points.
8 links of 10G are organized between the nodes of the core, which are combined into a logical channel with a total bandwidth of 80 Gbps. The capacity of the channels connecting the core and access nodes is 40 G.
Topology of the traffic exchange point DataLine-IX.
One of the Extreme BlackDiamond X8 at the OST site.
On one BlackDiamond X8 switch chassis, 768 ports of 10 GbE (7.68 Tbps) or 192 ports of 40 GbE (7.68 Tbps) can work. By default, we use 10 GbE ports to connect new members, but we can provide 40 and 100 GbE ports on request. Here are all available connection standards:
DataLine-IX participants exchange routes with each other through a routing server (Route Server, RS) using the BGPv4 protocol. RSs also provide member filtering for routes according to Internet Routing Registry (IRR) policies and other attributes of the BGPv4 protocol (AS_PATH, Next-hop, etc.).
Routing servers are deployed on two Huawei RH1288 V2-8S servers with a UNIX-based environment.
Server routing
The infrastructure of the traffic exchange point unites the participants into a single broadcast domain (L2-domain), so there is a high risk of broadcasting storms due to garbage BUM traffic (broadcast, unknown destination address, multicast). At a minimum, a storm can lead to a decrease in the bandwidth of the participants' channels. In the worst-case scenario, communication with the routing server will be lost, BGP sessions will break and all infrastructure IX will be closed. To prevent DataLine-IX from shutting down, we use multi-level protection against BUM traffic, limiting traffic as follows:
On the third and fourth levels of OSI, dynamic routing protocols, other than BGP, and other protocols that threaten the participants and the infrastructure itself IX are filtered. When sending routing information via BGP, both the prefixes themselves and the set of attributes for this prefix are analyzed (elements AS_PATH, Next-hop, etc.).
For new IX participants, the standard setting rules apply:
The connection procedure itself is built so that we have the opportunity to double-check the correctness of these settings. First, the new member connects to the port in the quarantine VLAN. We analyze its traffic, and if everything is configured correctly, the port is translated into a productive VLAN. There, the new participant is still isolated from the rest: its prefixes are not announced to the other participants of IX, and the participant himself does not receive anything either. If everything is OK, then the sessions are transferred to a productive mode of operation.
Ways to connect to DataLine-IX
New members can choose the following options for connecting to DataLine-IX:
DataLine-IX is still at a very early stage of development. At the moment we have 38 participants, the total number of routes is 6889. In addition to increasing the number of participants, the nearest plans include the organization of access nodes in external sites.
DataLine-IX statistics at ru.map-ix.net
Statistics participants DataLine-IX by occupation.
Ask questions in the comments if something interesting is left behind the scenes. In the second part, we’ll tell you about useful DataLine-IX tools for managing outgoing and incoming announcements, protection against DDoS attacks.
Today we will talk about them: let's tell you why we needed our own DataLine-IX traffic exchange point, how it is arranged, and share the first results.
ISP-A and ISP-B providers exchange their network traffic through a traffic exchange point. Without it, the exchange would occur through the “big” Internet.
The Internet Exchange Point (Internet Exchange Point, IX) is an infrastructure that allows autonomous systems to exchange traffic (peering) directly, bypassing higher-level telecom operators (uplinks). Peering helps IX participants reduce packet transfer routes between networks and reduce traffic costs. If you need to connect with several participants at once, then through a traffic exchange point it is easier to do: instead of a separate connection to each participant, one to IX, where these participants are present. This is the organizational side. Financially, participant IX also benefits from savings on channels and the purchase of traffic from uplinks.
')
For the end user, there are also pluses: network delays are reduced, the response time of resources on the Internet is reduced.
Traffic exchange points are a positive thing. The more of them, the better the connectivity of the Internet and the more accessible it is for the end user.
The first traffic exchange points began to appear in 1994 in major European cities: London (LINX), Frankfurt (DE-CIX), Amsterdam (AMS-IX), Moscow (MSK-IX). Now around the world there are about 580 IX.
The largest traffic exchange points in Russia. Source: internetexchangemap.com
When in 2015 we thought about our own traffic exchange point, more than 40 operators were present in our data centers (there are 53 of them now). Worked Meet-Me-Room, which simplifies connection to any operator on our sites. Its own traffic exchange point would give even more opportunities for our customers, especially for Internet providers and content generators (gaming services, CDN, video hosting, media, social networks):
- Improved connectivity of the DataLine network infrastructure: when connected to our network, client traffic goes through shorter routes with a minimum number of transit sections;
- reduction of participant IX's expenses for the purchase of IP transit from uplinks: part of the traffic will pass through DataLine-IX;
- building a more reliable network infrastructure: using DataLine-IX, the participant will be able to organize backup routes and unload channels to other traffic exchange points;
- access to the traffic of participants who are not present at other traffic exchange points;
- fast connection to several service providers via a dedicated VLAN.
How it works
The infrastructure of the DataLine-IX traffic exchange point is distributed over two sites - OST and NORD.
There are 6 Extreme BlackDiamond X8 switches at two sites. Two of them connect data centers into a single network and form the core of the DataLine-IX network infrastructure (Core). Two access points (Access) from each site are connected to the core. New members join the traffic exchange point through these access points.
8 links of 10G are organized between the nodes of the core, which are combined into a logical channel with a total bandwidth of 80 Gbps. The capacity of the channels connecting the core and access nodes is 40 G.
Topology of the traffic exchange point DataLine-IX.
One of the Extreme BlackDiamond X8 at the OST site.
On one BlackDiamond X8 switch chassis, 768 ports of 10 GbE (7.68 Tbps) or 192 ports of 40 GbE (7.68 Tbps) can work. By default, we use 10 GbE ports to connect new members, but we can provide 40 and 100 GbE ports on request. Here are all available connection standards:
- 1 / 10GbE SFP / SFP + (SR / LR / ER / ZR);
- 100/1000/10000 MbE (10GBaseT) RJ45;
- 40GbE QSFP + with SR4 / LR4;
- 100GbE CFP2 with SR10 / LR4.
DataLine-IX participants exchange routes with each other through a routing server (Route Server, RS) using the BGPv4 protocol. RSs also provide member filtering for routes according to Internet Routing Registry (IRR) policies and other attributes of the BGPv4 protocol (AS_PATH, Next-hop, etc.).
Routing servers are deployed on two Huawei RH1288 V2-8S servers with a UNIX-based environment.
Server routing
The infrastructure of the traffic exchange point unites the participants into a single broadcast domain (L2-domain), so there is a high risk of broadcasting storms due to garbage BUM traffic (broadcast, unknown destination address, multicast). At a minimum, a storm can lead to a decrease in the bandwidth of the participants' channels. In the worst-case scenario, communication with the routing server will be lost, BGP sessions will break and all infrastructure IX will be closed. To prevent DataLine-IX from shutting down, we use multi-level protection against BUM traffic, limiting traffic as follows:
- The ban on the reception on the ports of participants of all multicast frames, except for protocols and specific types of messages that ensure the correct operation of network services (LACP, ICMPv6 NS, ICMPv6 NA).
- Restriction on broadcasting frames (broadcast storm-control / broadcast rate-limit). In DataLine-IX, they are used by the ARP protocol to determine the MAC address by a known IP address.
- Filtering the ether-type field. Transmission of IPv4, IPv6 and ARP frames is usually allowed.
- Ensuring the reliability of information in ARP messages (ARP inspection). The participant responds only to ARP requests related to its IP address on a specific interface. For ARP packets, the rate-limit tool is also used to limit the number of ARP packets per second.
On the third and fourth levels of OSI, dynamic routing protocols, other than BGP, and other protocols that threaten the participants and the infrastructure itself IX are filtered. When sending routing information via BGP, both the prefixes themselves and the set of attributes for this prefix are analyzed (elements AS_PATH, Next-hop, etc.).
For new IX participants, the standard setting rules apply:
- On the participant's port in the direction of DataLine-IX, STP, IP redirects, LLDP, CDP, ARP proxy and other link-local protocols should be disabled, except for ARP and IPv6 ND.
- Ethernet frame announcement allowed: 0x0800 - IPv4, 0x0806 - ARP, 0x86dd - IPv6.
- One port - one member MAC address.
- Forbidden announcement network IX to other AS, not members of IX.
- It is forbidden to announce the default routes and full view.
The connection procedure itself is built so that we have the opportunity to double-check the correctness of these settings. First, the new member connects to the port in the quarantine VLAN. We analyze its traffic, and if everything is configured correctly, the port is translated into a productive VLAN. There, the new participant is still isolated from the rest: its prefixes are not announced to the other participants of IX, and the participant himself does not receive anything either. If everything is OK, then the sessions are transferred to a productive mode of operation.
Ways to connect to DataLine-IX
New members can choose the following options for connecting to DataLine-IX:
- Shared Peering - traffic exchange with all IX participants via routing servers.
General peering scheme. - Private Peering - traffic exchange with individual participants IX. In this case, traffic exchange is organized not through a routing server (RS), but through the establishment of direct BGP sessions between participants. This method of connecting to a traffic exchange point is useful when you need to improve connectivity with one or more specific participants IX.
Direct peering scheme. - Access to a dedicated VLAN (Private VLAN) - organization of communication channels through a dedicated VLAN with one or several participants. This method may be needed to combine several ports of one participant in one broadcast domain (VLAN) or to combine several participants in one VLAN.
Connection diagram through a dedicated VLAN . - A point-to-point channel (p2p) is organized through a switching fabric IX using the EoMPLS technology. Thanks to connectivity through a dedicated VLAN, you do not need to lay additional crossings for organizing a p2p channel; you just need to have a connection to DataLine-IX.
P2p wiring diagram
What's next?
DataLine-IX is still at a very early stage of development. At the moment we have 38 participants, the total number of routes is 6889. In addition to increasing the number of participants, the nearest plans include the organization of access nodes in external sites.
DataLine-IX statistics at ru.map-ix.net
Statistics participants DataLine-IX by occupation.
Ask questions in the comments if something interesting is left behind the scenes. In the second part, we’ll tell you about useful DataLine-IX tools for managing outgoing and incoming announcements, protection against DDoS attacks.
Source: https://habr.com/ru/post/317122/
All Articles