How Verizon and BGP Optimizer made a big offline

Major route leaks affect large sectors of the Internet, including Cloudflare

What happened?

24.06 at 10:30 UTC, there was a collapse on the Internet: a small company in the north of Pennsylvania poured traffic from a variety of routes passing through a large provider Verizon (AS701) - a navigator could send a stream of cars from a multi-lane highway to a narrow street . As a result, many websites on Cloudflare and many other providers have access problems. This should not have happened at all, because Verizon should not have sent these routes to the entire Internet. To find out how it happened, read on.

We have already written about such incidents before, from time to time they happen, but this time the consequences are felt throughout the world. The problem was aggravated by BGP Optimizer from Noction . It has a function that breaks the resulting IP prefixes into smaller and more specific ones. For example, our IPv4 route 104.20.0.0/20 divided into 104.20.0.0/21 and 104.20.8.0/21. As if the road sign for Pennsylvania was replaced by two others: "Pittsburgh, Pennsylvania" and "Philadelphia, Pennsylvania". By dividing large IP blocks into smaller ones, the network manages the traffic within itself, but this separation should not be made publicly available. Otherwise there are such troubles.

To explain what happened next, let's first recall what scheme the Internet works with. In essence, the Internet is a network consisting of networks called autonomous systems. Each autonomous system has its own unique identifier. All networks are interconnected by Border Gateway Protocol (BGP). BGP connects these networks and forms the structure of the Internet, in which traffic passes, say, from your Internet provider to a popular website in another part of the world.

Through BGP networks exchange information about routes, namely: how to get to them from anywhere. These routes can be specific (like a specific city on a map) or general (like a region). It was then that trouble happened.

One Internet provider in Pennsylvania ( AS33154 - DQE Communications) used BGP Optimizer on its network, that is, there were many specific routes on their network. Specific routes have priority over general ones (in the same navigator, for example, the route to Buckingham Palace will be more specific than the route to London).

DQE provided these specific routes to its client ( AS396531 - Allegheny Technologies Inc), and from there they got to a transit provider ( AS701 - Verizon), which carried these “optimal” routes all over the Internet. They seem optimal because they have more details and specifics.

And all this should not have gone beyond Verizon. And although there are effective ways to protect against such failures, Verizon's lack of filters led to a collapse that affected many services, such as Amazon, Linode and Cloudflare .

As a result, on Verizon, Allegheny and DQE collapsed shaft of users trying to access these services through their networks. They were not designed for such powerful traffic, which led to interruptions. And even if resources were enough, DQE, Allegheny and Verizon would not have been telling everyone about the perfect route to Cloudflare, Amazon, Linode, etc.

BGP leakage process with BGP Optimizer.

At the worst moments of failure, we observed a loss of approximately 15% of global traffic.

Cloudflare traffic levels during the incident.

How could leak be prevented?

There are several ways.

For a BGP session, you can set a hard limit for the received prefixes, and if the number of prefixes exceeds the threshold, the router will terminate the session. If Verizon had such a limit on prefixes, nothing would have happened. A provider like Verizon would be worthless to install. Why were there no limits? I have one version: carelessness and laziness.

Another way to prevent such leaks is IRR-based filtering. IRR (Internet Routing Registry) is a distributed database of Internet routes to which networks add entries. Other network operators use these IRR entries to create lists of specific prefixes for BGP sessions with other networks. If IRR filters were used, none of these networks would accept erroneous specific routes. Incredibly, Verizon did not have this filtering at all in BGP sessions with Allegheny Technologies, although IRR filtering has been used (and well documented) for more than 24 years. IRR filters would not have cost Verizon anything and would not have limited their service. And again - carelessness and laziness.

Last year, we implemented and deployed the RPKI platform, which just prevents such leaks. It sets filters on the source network and prefix size. Cloudflare announces prefixes with a maximum size of 20. RPKI indicates that more specific prefixes cannot be accepted, regardless of the path. In order for this mechanism to work, the BGP Origin Validation must be enabled on the network. Many providers, for example, AT & T have already successfully used RPKI in their network.

If RPKI were used at Verizon, they would see that the proposed routes are invalid and the router would automatically reject them.

Cloudflare advises all network operators to deploy RPKI right now!

Prevent route leakage using IRR, RPKI and prefix limit.

All of these recommendations are perfectly described in the MANRS ( Mutually Agreed Norms for Routing Security ) regulations.

How to solve the problem

The Cloudflare network team contacted the affected AS33154 (DQE Communications) and AS701 (Verizon) networks. It was not easy - maybe because when it all began, on the east coast of the USA it was still early morning.

Screenshot email to Verizon.

One of our network engineers quickly contacted DQE Communications, and after a short delay we were connected to someone who could solve the problem. With our phone support, DQE was able to stop sending “optimized” routes to Allegheny Technologies Inc. We are grateful for their help. Everything stabilized and returned to normal.

Screenshot of attempts to contact DQE and Verizon support services

Unfortunately, despite all our attempts to contact Verizon by phone and email, at the time of this writing (more than 8 hours passed after the incident), no one answered us and we don’t know .

We at Cloudflare would not like to repeat this, but unfortunately, very little is being done for this. It’s time for the industry to take more efficient measures to ensure routing security, for example, with systems like RPKI. We hope that large providers will follow the example of Cloudflare, Amazon and AT & T and begin to check routes . This is especially true for you, Verizon. We are still waiting for a response.

And although we could not affect what happened, we apologize for service interruptions. We care about our customers, and engineers in the US, UK, Australia and Singapore got in touch a few minutes after we discovered the problem.

Other articles tagged with BGP .