Projection of corporate conflict on network connectivity

The corporate conflict arose on 10.06.2019 due to the increasing cost of delivering SMS to VimpelCom users by Mail.RU Group. In response, Mail.RU Group ceased to “serve” direct Russian IP channels in the direction of the VimpelCom network.

Below is a brief analysis of the situation from the perspective of a network engineer.

Update : 06/14/2019 18:45 - emphasis on Russian routes to VimpelCom network, corrected conclusions, added an explanation by Sergey Kubasov (CIO Vkontakte) .
Update : 06/14/2019 19:48 - Added description of a method for limiting the distribution of routes along the “short” Russian route via Rostelecom, MTS, RETN. NO ..
Update : 06/15/2019 09:39 - Mail.ru has blocked its Looking Glass.
')

Introductory:
VimpelCom has an AS3216 autonomous system, all others (8402 - home Internet, 16345 - mobile Internet) are located behind 3216.

Mail.RU Group - autonomous systems AS47541, AS47542 and AS47764. The main content generator is 47,542, the so-called. CDN VKONTAKTE (movies, music). All autonomous systems are independent of each other (from the standpoint of an external autonomous system).

First, we look at the situation from the VimpelCom network. Looking Glass Vimpelcom will help us to do this.

We look at the first autonomous system - AS47541.

2914 47541 79.104.32.251 (metric 10500) (79.104.32.251) Origin IGP, metric 30, localpref 87, valid, internal, best, group-best, import-candidate, import suspect Received Path ID 0, Local Path ID 1, version 2865394342 Community: 2914:410 2914:1214 2914:2213 2914:3200 3216:3000 3216:3103 47541:1 47541:40000 47541:50078 

AS-PATH contains two autonomous systems - 2914 (NTT) and 47541 (VKONTAKTE-SPB-AS).
The localpref metric is set to 87, which, according to the description in the RIPE DB for the AS3216 object, corresponds to the international peering.

...
remarks: International peer's routes get local preference in the
remarks: range of 81-89.
...

The same information is confirmed by the 3216: 3000 and 3216: 3103 community (source - RIPE DB for AS3216 ):

...
remarks: 3216: 3000 Receieved from an international uplink or peer, specifically:
...
remarks: 3216: 3103 AMS-IX
...

That is, the route to VKontakte Vimpelcom sees through the European joint.

Let's look at another VKontakte 47542 VK (VKONTAKTE-MSK-CDN-AS).


And what about Mail.ru itself? Let's look at AS47764 (mailru-as).

  3356 47764 194.67.0.215 (metric 10501) (194.67.0.215) Origin IGP, metric 0, localpref 77, valid, internal, best, group-best, import-candidate Received Path ID 0, Local Path ID 1, version 2867605667 Community: 3216:3000 3216:3007 3356:2 3356:22 3356:100 3356:123 3356:519 3356:2094 47764:1 47764:40000 47764:50077 

Vimpelcom sees Mail.ru through as3356 (Level3 uplink, Tier1 operator). This information is confirmed by localpref 77:

...
remarks: Uplink's routes get local preference in the range of 71-79.
remarks: Last Update: February 2012 ...

and the community (3216: 3000 and 3216: 3007):

...
remarks: 3216: 3000 Receieved from an international uplink or peer, specifically:
...
remarks: 3216: 3007 Level 3 Communications
...

From the information received, it is clear that the traffic from the Vimpelcom network to VKontakte and MailRu spreads through European junctions according to the routes received via the BGP protocol. There are no alternative routes through the internal Russian joints in Looking Glass. Measures to artificially redirect traffic through the obviously worst routes were not detected.

And how does Mail.ru Group see the VimpelCom network?
Use the Looking Glass from Mails . Update : as of the morning 06/15/2019 Looking Glass does not work, when prompted, the 500th error is issued:

From AS47764 (mailru-as) routers:

  Path #6: Received by speaker 0 Advertised to peers (in unique update groups): 188.93.60.188 1299 1299 1273 3216 3216 217.20.147.250 (metric 100) from 217.20.147.250 (217.20.147.253) Origin IGP, metric 500, localpref 200, valid, internal, best, group-best Received Path ID 0, Local Path ID 0, version 1172721494 Community: 1299:20000 47764:701 47764:41100 47764:41108 47764:50077 

AS-PATH contains AS1299 (Telia, Tier1 operator, uplink Mail.RU) and as1273 (Vodafon, Tier1 operator, uplink VimpelCom).

LocalPreference 200 is the standard for external Mail.ru joints (https://net.mail.ru/bgp.html), and the MED 500 matches everything that was received not from a direct interface, not from IXs, not from peering.

But why are there no local routes through Russian telecom operators ???
There are they, but their priority is “non-standard”!
Here is the route through Rostelecom (as12389):

  Path #1: Received by speaker 0 Not advertised to any peer 12389 3216 46.61.178.149 from 46.61.178.149 (213.59.207.79) Origin IGP, metric 500, localpref 199, valid, external Received Path ID 0, Local Path ID 0, version 0 Community: 3216:2001 3216:2999 3216:4100 12389:5 12389:6 12389:1100 12389:1105 12389:1277 47764:701 47764:41100 47764:41104 47764:50077 Origin-AS validity: not-found 

Here through Megaphone (as31133):

  Path #2: Received by speaker 0 Not advertised to any peer 31133 3216 78.25.77.41 from 78.25.77.41 (10.222.253.97) Origin IGP, metric 500, localpref 199, valid, external Received Path ID 0, Local Path ID 0, version 0 Community: 3216:2001 3216:2999 3216:4100 31133:300 31133:46170 47764:701 47764:41100 47764:41105 47764:50077 Origin-AS validity: not-found 

Here through RETN. NO:

  Path #3: Received by speaker 0 Not advertised to any peer 9002 9002 3216 87.245.253.24 from 87.245.253.24 (87.245.225.1) Origin IGP, metric 500, localpref 199, valid, external Received Path ID 0, Local Path ID 0, version 0 Community: 9002:9002 9002:64667 47764:701 47764:41100 47764:41101 47764:50077 Origin-AS validity: not-found 

And even through the MTS!

  Path #5: Received by speaker 0 Not advertised to any peer 8359 3216 212.188.61.105 from 212.188.61.105 (195.34.52.77) Origin IGP, metric 500, localpref 199, valid, external Received Path ID 0, Local Path ID 0, version 0 Community: 8359:200 8359:609 8359:5012 47764:701 47764:41100 47764:41103 47764:50077 Origin-AS validity: not-found 

The metric localpref on these Russian routes is understated, that is, routes are worsened compared to foreign ones!

In addition, Mail.Ru Group prohibits the distribution of its prefixes in VimpelCom through Russian operators!

RETN. NO (http://lg.retn.net/):
On announcements from Mail.RU Group there is a 3216: 65535 community.

Routes marked by such a community VimpelCom does not accept on its network. Extract from RIPE DB by AS3216:

...
remarks: Internal comminuties are only internally assigned.
remarks: They are in range 3216: 0000-3216: 4999 and 3216: 6000-3216: 65535
remarks: at the border
remarks: routers.
...

Towards Rostelecom (http://lg.ip.rt.ru) Mail.RU Group gives similar routes with the 12389: 8350 community.

According to the entries in the RIPE DB for the as12389 object, this community means “do not announce as3216 on the network”:

...
remarks: | 12389: 835y When advertising to GoldenTelecom (AS3216) |
...
remarks: | ... y = 0 - do not advertise |
...

Similarly, in the direction of MTS (http://lg.mtu.ru):
Community 8359: 2120 means:

...
remarks: 8359: 212x when announcing to Sovam (Beeline)
...
remarks: x = 0 - do not announce
...

It is impossible to look at the announcements of Mail.RU Group in the direction of Megaphone - the latter does not have Looking Glass.

Let's see AS47541 (VKONTAKTE-SPB-AS).


AS-PATH points to AS174 - Cogent (uplink Mail.RU, Tier1), then AS6762 - Telecom Italia (uplink VimpelCom). Local Preference everywhere 150, but so on all external joints, regardless of the written policy .

Let's see AS47542 (VKONTAKTE-MSK-CDN-AS).

  Router: mx960-m9-0 Command: op lg-sh-bgp prefix 81.211.56.202 0.0.0.0/0 LP:151 MED: NH:87.240.191.222 AS path: 47541 I Communities: Accepted Best 0.0.0.0/0 LP:151 MED: NH:95.142.204.251 AS path: 47541 I Communities: Accepted Inactive-reason: Interior > Exterior > Exterior via Interior 

And from the second router:

  Router: mx960-m9-1 Command: op lg-sh-bgp prefix 81.211.56.202 0.0.0.0/0 LP:151 MED: NH:87.240.191.224 AS path: 47541 I Communities: Accepted Best 0.0.0.0/0 LP:151 MED: NH:95.142.204.250 AS path: 47541 I Communities: Accepted Inactive-reason: Interior > Exterior > Exterior via Interior 

Only default routes (0.0.0.0/0). This situation was explained by an employee of Mail.RU Group greediness , for which he thanks. In short: the Moscow segment of the VKontakte network is caching (and not generating), whose task is to optimize the download speed of popular popular content. Caring for users, yes.

If there is no route to any network, then this network is not served by caching servers. This means that the optimization of download speed does not work, and users are suffering. But here it is necessary to emphasize - users are not only their own, but also VimpelCom.

Findings:

1. On the VimpelCom side, the traffic towards Mail.RU Group is distributed naturally. No artificial redirects were found using Local Preference manipulations.
2. From Mail.RU Group, there are manipulations with VimpelCom prefixes . For existing routes towards the Vimpelcom network through Russian operators, priorities are understated compared to routes through foreign Tier1 operators.
3. Mail.RU Group routes to the Russian operators (MTS, Rostelecom, RETN.) Have been added by BGP management communities to limit their distribution towards VimpelCom

Why does Mail.RU Group put routes through Europe higher priority? Why does Mail.RU Group prohibit short domestic Russian connectivity with VimpelCom?

Are they so cheaper? Drive traffic through foreign channels and pay Tirvana currency?
Or is there a desire to drive traffic away to take it was not so convenient, eh?
This is not known to the network engineer ...