DDoS bypassing the Curator: simple steps for a quiet life

Recently, the second Uptime.commuinty information systems operation and administration conference was held in Moscow, where we also shared our experience. We, as usual, about sore - about DDoS.

DDoS attacks on Habr began ten years ago and still present for us an unpleasant problem. At first, there were timid attempts to podzalit a little bit, and now for us the usual DDoS is about 30 Gbit / s. This is not surprising, because now every grandmother in Moscow has 50MB. All the classics: one old woman - 50, 10 old women - 500 ...

Vadim Rybalko , Habr.

This is not about any know-how and special Jedi techniques. Everything is simple and rather prosaic and more like a complex of banal hygienic procedures. Most of the “experienced” administrators are already aware of everything mentioned below, but to summarize and repeat once again is not superfluous. We reached many solutions on our own, based on our experience, so if someone can save some time, this is good luck. Are you still not doing backups? Then we go to you!

A little bit about our architecture

At the main site, we have all our hardware, our own rack, all our servers are powerful enough, we take the maximum. Everything is tightened into a gray network, because we try not to have our own IP-Schnick. There are three telecom operators in the rack from whom we rent small blocks of IP addresses. We thought to take AS with Provider independent addresses, but in that case we would have to buy equipment that stands like a wing from Boeing, and pay for channels that stand like the second wing. And in the end, for the defense, we chose Qrator .

We have known our colleagues since then, when they only started working on the technical platform of the Moscow State University under the HLL brand, since then our relations in some areas have long outgrown corporate relations. They were among the first in Russia to protect against DDoS, and still remain among the most appropriate. Insider information, of course, I will not share, I will say this: they are among the best in this market.

the main problem

We perfectly present how Qrator works, and we have no complaints about the work. This is not about them, but in principle about this type of DDoS protection. Any protection has architectural limitations, and, accordingly, ways of using these restrictions to use the attacker.

No matter how good the defense is, it is only the first frontier and only helps from an attack in the forehead. This helps from hackers-pioneers, but if the site is “ordered”, then the attackers will not limit themselves to swapping around the domain, they will surely get tired of searching for more vulnerable real resource addresses, and they will beat them there.

Where can an attacker find out the real network addresses?

Public whois and other Ripe DB databases

The first is databases, for example RIPE, where a lot of public information is indicated. There is no direct equivalent of “Privacy WHOIS”, as in the domains. There all data is indicated in plain text: admin contacts, company name and other technical data. Very useful information for intruders. Suppose we are called "Habrahabr". He can search for the word "Habr" or something similar, and can find us. Now it is not as easy as before. But there is such an option.

It must be remembered that, in addition to RIPE DB, there are some other hosters (such as “Hetzner”), who always require you to fill out a special form for RIPE, even if you rent one server from them. They can also mark rented addresses in whois somewhere, for example, the hoster can still indicate the name of the organization in remarks to the address. And all this will also be visible with the usual parsing of the database. A slightly more intelligent attacker can sort by the nic-handle of the administrator person or by the mintainer.

Potential protection : you need to depersonalize your IP address blocks as much as possible.

Reverse resolve

Correct PTR is convenient and correct, but this is another attacker's weapon. As a rule, it is used combined with other methods of perimeter research.
Everyone knows that for the normal operation of the mail you need to register PTR. In addition, for example, without PTR there may be a long rezolv. But they need to be depersonalized, because if you register a technical domain in the PTR, then an attacker can start scanning for this domain and find interesting entries in the subdomains. All public PTR is desirable either to close the provider domain, node-0-0-0-0.yatvoidomtrubashatalisp.net , for example. Either write some rubbish that does not tell the intruders anything sensible.

Potential protection : use impersonal PTR, in the reverse zone of the operator.

Address selection, port and service scanning

An attacker can scan a block of addresses on open ports, including the entire LIR (operator), since there are relatively few of them in general. Having received a list of addresses with active web servers, an attacker can make a request a la curl -H "host: example.com" http://INET_ADDR/ in order to get answers from the web server with the virtualhost being attacked. HTTPS can help an attacker with this, especially if there is only one certificate on the server and no TLS SNI. And, let's say, it may be that the attacker does not even try using cURL to pull the name of a specific site with the header, but simply stupidly pulls the IP address on port 443. And he can get the default certificate, which will indicate the name of the site and its domain .

Potential protection : there are many methods of protection against such studies, and it is best to limit incoming connections at the firewall level altogether: everything that comes from trusted networks and not from traffic filtering point networks is simply not to respond. If a data clearing center is used, then you need to take a list of its addresses and add them to the white lists and completely close the entrance for everyone else. For example, the Curator recently introduced automatic testing of the availability of protected hosts not from the addresses of traffic clearing points, for which he thanks.

It should be remembered that if you think that if you specified the address of the data clearing center in the A-record for the site, then your nginx will not give it away - you are mistaken. Of course he will. Once again: the easiest way is simply to hide all the hidden things with a firewall.

Mailboxes

The first is that the mail server can helo to helo technical domains that can be subsequently used for scanning.

The second thing that I suspect once got ourselves - the Received : headers in the headers can be specified every hop where the letter passed. And there it can be stated that a server with such an IP received a letter from the server with such an IP, using a specific protocol, at a specific time, with a specific letter ID. Just looking at the source of the letter, you can see your IP-Schnick.

Potential protection : it is better to display mail through an edge mailer, which is located in the network space that is isolated from the main infrastructure, for example, on a leased server or virtual server. It is necessary to control the headers of letters and mask / delete compromising headers received in automatic mode. Or use an external resource, such as Mailgun , there is no such problem (but there are others).

DNS

A whole bunch of opportunities to attack, and you can’t list them all. The key node is primary NS, which is the source of all updates for all secondary. If primary is set, then it will also block the ability to change the regular A-record. And you can’t do anything, because you first need to redelegate the domain to other NS servers. In general - a lot of problems that we also faced, about 3 years ago. As a result, we simply hide the primary. That is, we do not announce them anywhere. We have them, but they are far away. He is not alone, and their IP addresses and domain names are not indicated anywhere at all. In the SOA record, where we are supposed to indicate primary, we indicate the address of one of the secondary. Everything is obvious. And we do not update the DNS using axfr. Since DNS was invented a long time ago, this technology is not very suitable for our scheme. We use PowerDNS with MySQL backend, where we store all databases. All our secondary are MySQL slaves with PowerDNS. And even if some of our DNS will be ddosit - we still have a lot of them. Including DNS, which are covered with protection from DDoS-attacks from the Curator. We bought a bunch of virtual locks, there are really a lot of them.

Potential protection : as in the case of postal workers, do not keep DNS in the same infrastructure with the main engineer. Master is generally better to hide away and not announce. It is better to delegate domains to secondary, including in the SOA records.

Pro AS

We thought, wondered if we should take our AS. We were even offered to do almost for free with the status of LIR, but, first, we do not need as many addresses (/ 23 - 512 pieces). Secondly, it is unnecessary responsibility, because you need to communicate with the RIPE, and you need to be able to communicate with them. Thirdly, you need to pay RIPE money for IP-Schnick (formally not). And, most importantly: all the data, all IP-Schnick will be accessible to all. Accordingly, you need to put expensive pieces of iron, you need to have very good channels, to have a bunch of uplinks, to increase control over the network. This is not for us at all, but a normal scheme for someone bigger.

They burned, pour 30 gigs into the channel. What to do to minimize losses?

Independent channels

Have several independent channels of independent operators, with independent routers. Two or three small blocks from different parts of the blocks of addresses of the operator. This allows you to quickly and safely "merge" the attacked block into nullroute.
Even having a data clearing center, we specify the addresses of several of our external providers in upstream so that in case we have a problem with one of the providers, for example, a network loss or a DDoS attack, so that the Curator can transfer the entire load to another. This is similar to the upstream from nginx, which is able to throw out non-working upstream.

Sane Operators

It is very important to work only with normal operators with operational support service. There are good operators: as a rule, these are not very large companies, but firmly established in the market. They have everything cool, you can call, understand what exactly is happening and how the problem is solved. You can always get through to the person who makes the decisions. In our experience, large operators are evil. If you are not a giant corporation, they will not work with you normally. A cumbersome bureaucracy, in any case, makes it difficult to make decisions quickly, to get through phone calls to technical support is already a big problem.

Bfg

Have at the junction with the operator BGP, even in the absence of a normal AS and address block. BGP for many is three terrible letters, which are called especially dark forest with a special strong sort of dark magic. In fact, nothing terrible here. Even if you have no AS, BGP is good. Because normal operators can allow their own small block of addresses to announce to themselves from the client’s routers under the “gray” AS. If suddenly a certain prefix is ​​poured, then you remove it from the announcement on the border routers, and the attack shuts down on the operator’s side, and the machine does not reach the router. Even if there is any kind of UDP that is hard to control, it is already terminated at the operator level. He has good channels, and he can handle it. And our routers, although normal, but not a fierce enterprise, of course, they will not take up 30 gigabits. So I advise you to deal with the network: how it works, how routing works, including in the global network; what is BGP, at least the main thing. Without ridicule, a good tutorial on the network is Cisco CCNA, they even tell you about the Flintstone family.

Be calm

In the case of an attack, do everything slowly and thoughtfully. The most important rule is to do it first. If suddenly something happened, you can not lose your head. Especially when the boss is freaking out that something is not working, in no case do not make sudden movements, because you can accidentally lose everything at all. For example, it’s banal to register an incorrect IP-schnick in the wrong access-list, and you can lose the entire grid. Or, when the server is lying, the database has died, you must first understand what exactly died before repairing it, because you can only make it worse, and at the same time break the backup. No wonder there is a saying: “measure seven times, cut one”. You need to do everything thoughtfully.

Order pentest or use WAF

It is never a shame to give a controlled break of the perimeter protection. Most likely the pentest will be an expensive pleasure for the “man from the street”, but always “options are possible”. As for the WAF - a good thing, but only in the complex. Many of the WAF vendors on the market look odd charlatans.

IDDQD mode

If the company really has a lot of money. The entire AS can be advertised via BGP to the data clearing center. This is quite expensive, because it is necessary to sensibly calculate the utilization of the channels, since billing is carried out for the traffic bandwidth. If someone starts to pour something not through that IP address, then he can pour on a lot of money. In addition - it adds hemorrhoids, as you have to thoroughly understand network technologies and be able to prepare them. Most likely it will be an excellent option for an average company with a developed infrastructure and a formed operating department.

As in all important matters, in the case of the perimeter protection of the network, relying on chance is not very good. The statement about the “magic pills” is also true - they do not exist. DDoS is an unpleasant thing, but not necessarily fatal. Follow the safety rules, do not substitute, all that needs to be hidden - hide.

We wish all the good guys to successfully repel all attacks, we wish attackers every failure. And may the force be with you!