The architecture of the Aggregation-Access network of large providers

The architecture of networks of modern telecom operators is perfectly described in all sorts of manuals, guides on preparing for Cisco certifications, and just smart and good books. But many of them focus precisely on MPLS Core with interesting features of this technology (such as Traffic Engineering, MPLS BGP Multipath, etc.), bypassing the distribution-access segment. I propose to talk about the architecture of the access network, adopted by large providers. As examples, we will consider the access networks of one of the UAE operators (let's call it UAE Telecom) and Tier 1 operator from the USA (say, USA Telecom), with whom I was lucky to work. According to the information, the same aggregation-access architecture has an IP network of one of the largest Ukrainian operators.

The carrier network consists of a core, usually using MPLS technology, and segments of aggregation-access networks that allow users to connect to the network. USA Telecom has several networks, one of which, however, is built without using MPLS — hardware packet-switching technology using Routing Plane and Forwarding Plane separation (Cisco Express Forwarding and similar to Juniper) allows you to route regular IP packets at the same speed as it is switched by MPLS routers.

Provider Edge (PE) routers are located on the border of the MPLS core. They are more productive than core routers (P Routers), because they need to “hold” the BGP table. So USA Telecom uses GSR 12000 and even CRS-1 as PE routers - Tier 1 status and tasks are required.

After PE routers, there is a network consisting of switches to which users connect. Today we will talk about her.
')
The main task: to make the network as scalable as possible, since resources are becoming more expensive with each following “step”: the switch distribution port is more expensive than the switch access port, and the PE router port is more expensive than the switch distribution port. The applied architecture allows to expand the network most efficiently. In addition, such an extension is carried out with the provision of communication redundancy in the most efficient way.

UAE Telecom uses Cisco ME 4924 as a distribution device, and Cisco 4506 as an access switch. I suggest using them in our model.



It is important to say that the operator uses several architectural approaches, and only one of them is described in this article. The second, with a similar configuration, implies connecting two access switches in each ring. This increases the number of access ports and does not affect reliability:



The operator provides Triple Play services to individuals, but the customer can determine for himself exactly which Play from these Triple ones he needs. Enterprises are offered a Layer 2/3 VPN and separate access service configurations with redundant access. In this case, the customer receives a link to two different access rings.

Connecting the Internet, each customer receives a separate VLAN. All VoIP traffic of all customers, however, “walks” in a separate VoIP VLAN with the appropriate QoS settings. This is what the access switch configuration looks like for an average customer:

policy-map mark-voice class class-default set dscp ef ! interface FastEthernet3/47 description 00000000-Customer-Loc switchport access vlan 434 switchport mode access switchport voice vlan 808 logging event link-status qos vlan-based no snmp trap link-status storm-control broadcast level 1.00 tx-queue 3 bandwidth percent 30 priority high spanning-tree portfast spanning-tree bpduguard enable ! interface Vlan808 no ip address service-policy input mark-voice 

The use of level 3 switches at the access level allows you to configure flexible QoS policies and prioritize key traffic, such as VoIP and routing protocol updates, already at the access level.

VLANs for a typical set of services are terminated on distribution switches. For each of them, SVI is the default gateway. Immediately - at the level of distribution of switches, provides traffic shaping.

STP in the face of PVST supports link redundancy. For individuals, only this type of redundancy is provided - if the link to “its” distribution of the switch “falls”, traffic will be able to get there along the ring through the second switch.

For organizations, the distribution of switches is supported by redundancy - between the pair of SVIs that act as the default gateway for the organization on each distribution router, the HSRP is configured. If one of the distribution of switches fails, individuals whose VLANs are tied to its SVI will be left unconnected, but the organization will pick up the second distribution switch by making their SVI default gateway for their VLANs.

 router ospf 100 ! passive-interface Vlan333 network 123.102.12.49 0.0.0.3 area 23.123.15.0 ! interface Vlan333 description Customer-SVI ip address 123.102.10.133 255.255.255.252 ip access-group 101 in no ip redirects ip ospf message-digest-key 1 md5 7 0200005700002120 service-policy input TECOM-Policy-2MB service-policy output TECOM-Policy-2MB ! interface Vlan281 description Enterprise-SVI bandwidth 2000 ip address 123.102.12.49 255.255.255.252 ip access-group 101 in no ip redirects ip ospf message-digest-key 1 md5 7 033000070200004D standby 170 ip 123.102.12.48 standby 170 priority 110 service-policy input TECOM-Policy-512K service-policy output TECOM-Policy-4MB 

OSPF also works on distribution switches. This is necessary due to the fact that these switches are engaged in routing traffic between VLANs within the same distribution pair of devices. But there are exceptions: if a client orders a Layer 3 VPN, the CE router must establish an OSPF neighborhood with the PE router of the provider. To ensure this, such a client is allocated a VLAN, which is not terminated on the SVI switch, but is forwarded further, to the PE port of the router, by a trunk. Total, from distribution of a switch to PE of a router one Vlan leaves, with the traffic which the switch routes to PE, and set of VLAN of clients who use Layer 3 VPN:

 interface GigabitEthernet1/3 description Uplink to PE Router switchport trunk encapsulation dot1q switchport trunk allowed vlan 1,12,19,31-33,36-41,70-79,88,98,112,113,125,126 switchport trunk allowed vlan add 135,143,198,199,314,316,320,322,399,440-444 switchport trunk allowed vlan add 454,456,459,480,490,510,511,530,550,557-559 switchport trunk allowed vlan add 563,567,571,578,579,582,588,602-604,616,626 switchport trunk allowed vlan add 637,660,661,666,670,672,675,677-679,688,699 switchport trunk allowed vlan add 705,713,728,739-749,753,754,757,760,762 switchport trunk allowed vlan add 769-778,783,788-790,793-795,799,806,812,822 switchport trunk allowed vlan add 823,878,888,900,923,925,950,955-958,987,990 switchport trunk allowed vlan add 993-995,998,999,1100,1206,1209,1252,1301 switchport trunk allowed vlan add 1302,1313,1325,1326,1350,1351,1360,1361,1400 switchport trunk allowed vlan add 1705,2100,2400,3432,3900 switchport mode trunk qos trust dscp udld port aggressive mac access-group filtermac in tx-queue 1 bandwidth percent 10 tx-queue 2 bandwidth percent 20 tx-queue 3 bandwidth percent 30 priority high tx-queue 4 bandwidth percent 40 service-policy input police-in-ge 

Thus, each distribution pair of devices forms a separate VLAN domain, and in the network controlled by another distribution pair of switches, you can use VLANs with the same numbers again.

In the UAE, the Internet and other IP services are very popular, because the coverage is very, very dense. So a pair of switch distribution can cover one building. And for one of the largest skyscrapers - business centers, even a separate pair of PE routers was allocated. For comparison, according to the stories, at Kyivstar, a pair of PE routers provides coverage for one area. Although, of course, IP services are not the largest part of the business of this operator, the comparison is significant.

PS Company names, IP addresses, descriptions, MD5 hashes and other information that can be kept confidential are changed. All technological details, however, are preserved in the original.