Four facts about memcached amplification

This is a translation of Artem ximaera Gavrichenkov’s original publication, “Understanding the facts of memcached amplification attacks,” published on the APNIC (Asia-Pacific Network Information Center) blog .

The week from February 25 to March 3 was high-intensity from the point of view of memcached-amplified DDoS attacks in all corners of the world, that is, on the Internet.

Nevertheless, let us once again recall all the facts that we know about amplified attacks.
')

Fact number one: Amplifiers were, are and will be

NTP (Network Time Protocol) was the first protocol maliciously used as an amplifier of DDoS attacks back in 2013. Thousands and hundreds of thousands of NTP servers by that time were deployed throughout the network, so the use of this amplification vector was quite beneficial for attackers. And NTP gave the opportunity that resulted in a wave of NTP-amplified DDoS attacks. At the beginning of 2014, for some time NTP as the main amplifier became even more popular than the DNS protocol (Domain Name System).

In 2015, we saw a massive rise of botnets. They used UDP-based (User Datagram Protocol) amplifiers, setting a record for the band of DDoS attacks observed by that time. Different botnets used different techniques, but a combination of thousands of compromised devices and NTP or DNS amplifications made such attacks extremely dangerous, as shown by the denial of service from Dyn and OVH in 2016. Joint resistance to the botnet command and control centers (C & C) made it possible to avoid potential capture by botnets of control over the entire network.

The same 2016 clearly demonstrated that NTP and DNS are not the only protocols used as amplifiers. Portmap, SNMP, SSDP, Chargen, MSSQL, CLDAP and some other network protocols provide excellent opportunities to strengthen DDoS attacks. Despite this, the next 2017 year brought a lot of different vulnerabilities that allow exploiting the dark sides of other protocols, because attackers with knowledge do not go on vacation and try to look for new, previously unexplored, attack vectors.

Fact number two: Amplification has not changed and has not evolved - just found new vulnerabilities

Moreover, it is vulnerabilities that are the reason why we witnessed the events of the end of February and the beginning of March 2018. After in 2017, a group of Chinese researchers from 0Kee described the amplification path through memcached, it took the attackers only a couple of months to set a world record of an amplified DDoS attack conducted using an unprotected memcached server. Being free, free and present in virtually every Linux distribution, memcached is a powerful tool for destruction, providing a hypothetically infinite amplification factor (according to our data, the amplification factor is more than 10,000x, according to Akamai - more than 50,000x). In this case, we again run into the capacity of transit connections as the main requirement for survival under a similar link-level attack.

Link to the original presentation 0Kee.

When 10 years ago, the interception of UDP traffic was added to the basic memcached configuration, the birth of the new amplifier was simply obvious. Just ten years ago, amplifiers as such did not worry almost anyone. Only now, in the conditions of the growth of transmission and processing speeds of packets that has been going on for the past ten years, this has become a problem.

The general concept of reflection and amplification has not changed at all. First, illegitimate requests are sent to the vulnerable (open to the world on port 11211) memcached server (this technique is traditionally called IP-spoofing ). After this, the UDP server prepares a response and sends thousands of responses to the target with the help of an attacker, flooding the target host, which is quite legitimate from the point of view of memcached flood of responses. Huge values ​​of PPS and BPS cannot be bypassed by any general means and means, since such a volume of traffic quickly makes routing devices at the boundaries of networks unavailable due to resource exhaustion.

Fact number three: When something works, it works; but sometimes against you

So, a new world record has already been set in the DDoS attack band: Arbor Networks showed 1.7 Tbit / s to one of its customers, while GitHub, under the protection of Akamai, survived 1.3 Tbit / s. The attack on GitHub was riveted by the attention of many media, as it is a large and popular resource, the normal operation of which is supported by many services and products around the world. Qrator Labs client - Qiwi payment system, successfully neutralized the 480 Gbit / s attack of the amplified memcached UDP traffic.

The Shodan statistics still show a large number of active memcached amplifiers open to attackers from anywhere in the world:


Shodan statistics for port 11211

As you can see from this screenshot, mainland China and the United States are the two main sources of memcached amplifiers. Qrator.Radar also collected some statistics on the number of accessible and vulnerable memcached servers:


Qrator.Radar statistics on the total number of vulnerable memcached servers

The number of such servers drops very quickly. Why? No one wants to kill or be killed. Operators (classical communications providers and transit operators) are worried about their networks and that they can be used as a tool to enhance what can easily be called "illegal."

In order to make this information even more obvious and understandable, we collected statistics on the number of amplifiers in the customer cone of individual autonomous systems, here's an example of the top of this list:



DDoSmon also provides the ability to receive fairly detailed statistics on the number of targets of memcached-amplified DDoS attacks, where current trends are also visible: https://ddosmon.net/memcached_amplification_attack

Fact number four: everything that can be corrected should be corrected.

Memcached users should disable UDP support in case they do not use UDP to transfer data. Also, memcached should listen only to localhost - not all available interfaces, as it happens in some default settings.

Any player larger than a single company that is responsible only for itself or one person responsible for a specific resource must be 100% sure that no memcached servers are visible from the rest of the network and are closed by a firewall.

Be sure to pay attention to the recommendations of AUSNOG (Australian Network Operators' Group) for the implementation of Explotable Port Filters.

And although the number of vulnerable servers across the network is declining, attackers continue to search for new and stronger ways to exploit vulnerabilities and, in general, are currently succeeding in this search. System administrators, who are supposedly responsible for memcached servers, behave recklessly or even ignorantly, so that the long tail of attempts to close the remaining caches that cause problems, do not go anywhere. That is why it is so important to try to deal with the problem of memcached amplification at the operator level.

Currently, the best way to do this is to use rate limits on all ports looking outside and monitor UDP traffic on port 11211 until it can be managed. It protects your infrastructure, and your customers, and any random victim.

It contains several recommendations at once: http://www.senki.org/memcached-on-port-11211-udp-tcp-being-exploited/

NTT recommends adding memcached UDP / 11211 to the same “exploitable ports” list as NTP, CHARGEN and SSDP. Below is an example configuration for IOS XR, allowing you to set rate-limit on amplification sensitive UDP ports up to 1% of capacity. Qrator Labs considers this method of processing memcached traffic extremely useful and efficient, recommending it for deployment:

ipv4 access-list exploitable-ports
permit udp any eq ntp any
permit udp any eq 1900 any
permit udp any eq 19 any
permit udp any eq 11211 any
!
ipv6 access-list exploitable-ports-v6
permit udp any eq ntp any
permit udp any eq 1900 any
permit udp any eq 19 any
permit udp any eq 11211 any
!
class-map match-any exploitable-ports
match access-group ipv4 exploitable-ports
match access-group ipv6 exploitable-ports-v6
end-class-map
!
policy-map ntt-external-in
class exploitable-ports
police rate percent 1
conform-action transmit
exceed-action drop
!
set precedence 0
set mpls experimental topmost 0
!
class class-default
set mpls experimental imposition 0
set precedence 0
!
end-policy-map
!
interface Bundle-Ether19
description Customer: the best customer
service-policy input ntt-external-in
ipv4 address xxx/x
ipv6 address yyy/y
...
!
interface Bundle-Ether20
service-policy input ntt-external-in
...
... etc ...