Invalid mail server, or life without spam

Fighting spam is the headache of all responsible mail administrators. What they do not invent, to make their lives better loved . However, as shown by the practice of communicating with many system administrators, for some reason, not everyone can imagine how to properly filter spam.

The most common approach is “add a bunch of RBL (DNSBL) and will enjoy life”. The approach is not correct a little more than completely. The second most popular - content filters, often bought for big money. This approach is also in most cases completely unjustified.

But everything is so simple, for a quiet life it is enough just to look closely at the three headers of the incoming SMTP session. Having rummaged on Habré and in the back streets of the Internet, I didn’t find an exhaustive article on how to properly configure the SMTP server in terms of countering spam. Therefore, I decided to paint everything that I know on this subject myself and what I successfully use.
')
By the way: this article is certainly aimed primarily at administrators who want to make a high-quality spam filter. However, on the other hand, it contains very important information for those who simply have to work with the mail, but who are not well versed in all the intricacies of the process of e-mailing correspondence.

So, if you want to protect your users from spam or vice versa, you want someone to accidentally not protect users from your emails - welcome under the cat.

A small note: the article was written with an eye to setting up a Postfix mail server, but on the whole it is rather theoretical. The described Postfix options should be specified in the corresponding * _restriction parameters of the configuration file, for any details, refer to any Postfix configuration guide.

A bit about the SMTP protocol

E-mail has many similarities with ordinary mail. The most important thing for us now is that all the information on the electronic "envelope" is just two addresses: the recipient and the sender, as well as the postman's stamp, the envelope delivered.

Let's get a little distracted: Imagine that a face of extremely repulsive appearance will come to you and hand in a tightly sealed parcel with the return address “Tray from Tilimililitryamdiya”. Dare to accept and open? Hardly. So, email can also be easily checked and sifted out based on address information only, and the scope for possible actions here is much wider.

As you should know, mail on the Internet is transferred between mail servers via SMTP. Any communication on this protocol begins with three mandatory headers: HELO , MAIL FROM and RCPT TO . That is, before starting to transmit any data, the server is first presented (HELO), then it reports the return sender address (MAIL FROM) and then the recipient address (RCPT TO). These three headers are the signature on the electronic envelope, and almost all spam can be eliminated only on the basis of their analysis. Most attempts to send something to my server do not go beyond MAIL FROM, that is, the letters are eliminated even before the actual acceptance, which significantly reduces the load. That is, instead of opening the package from Tryam and discovering anthrax spores there, I immediately send the postman to hell.

So, what should you do in order not to accept emails that are obviously spam? Let's go in order.

A bit about DNS

Sometime in the early days of the Internet, mail was delivered directly to the sites listed in the mailing address. That is, to deliver a letter for user@domain.com, the mail server searched for the IP address of domain.com and tried to send a parcel using the found IP. Then MX records appeared that at once solved most of the problems of similar organization of mail interaction. However, some programs can still work with A records when delivering mail. But you, of course, have at least one MX record for your domain, right?

MX records contain server addresses to which emails for the specified domain should be delivered. However, in order to combat spam, a technology has emerged that allows to indicate in DNS also the addresses of servers from which letters from a specified domain can come. Her name is Sender Policy Framework .

I will not go into all the details of the technology in detail; I will only say that the TXT record

v=spf1 +mx -all

for your domain, it will tell all clients that support SPF checking that emails from your domain can come exclusively from the servers specified in the MX records. You can make the rule softer by writing instead of -all ~ all . For details, contact Google and the official SPF website .

Always register the SPF record for the domain, as well as enable the SPF check on your mail servers. I recommend to strictly forbid sending emails from your domain from all hosts except your MX servers. Coupled with SPF checking on your server, such a setting immediately cuts off all emails sent from third-party hosts on behalf of users of your domain to addresses of users of your domain. And almost half of such spam, because usually SMTP servers are very poorly protected from emails from their own domain, and spammers actively use this. SPF once and for all relieve you of the letters to Vasya Pupkin, written judging by the envelope Vasya Pupkin, but coming from a server in some Nicaragua.

How to configure SPF in Postfix will tell you Google, information in bulk and can not be mistaken, so we will not waste time on technical details.

There are a couple of extremely important remarks about DNS. Most likely you know that DNS master records, the so-called A records, convert the name to an IP address. In addition to these, there are also CNAME records that assign a nickname to an already existing name. It is these two types of records that form the basis of the entire domain name system.

But few users know that there are also reverse entries that convert IP to a domain name. They are called PTR. So, there are two unwritten (strictly speaking) rules, which are still followed by everyone:

• For each A record, there should be a mirrored PTR record, that is, by the host name via DNS we get IP, and by IP - the same host name back.
• The address in the MX record should always be the name (not IP!) Of the host for which an A record exists. That is, it is impossible for an MX record to have an IP or alias (CNAME).

If you do not comply with one of these requirements - be prepared for the fact that at least a quarter of mail from your domain will be recognized as spam. This is due to a simple thesis: the trustworthy sender always adjusts everything by observing the rules, respectively, if the rules are not followed - then the sender should not be trusted, and therefore also to receive mail from him - as well.

Well, to enable PTR checking in your home, use the option

reject_unknown_client_hostname

It requires that the IP from which the connection is made is resolved to the name via PTR, and this name is resolved in turn back to the desired IP.

There is a less severe restriction given by the option

reject_unknown_reverse_client_hostname

In this case, the server will check only the presence of the PTR record, but will not require the existence of the corresponding record A.

Checking the greeting

So, someone wanted to send a letter to your server. The transfer begins with a greeting - the header HELO. In HELO, the full domain name (FQDN) of the sender must be indicated, respectively, if this is not the case, you can safely refuse to accept it immediately. There are two options for this in Postfix:

reject_invalid_helo_hostname
reject_non_fqdn_helo_hostname

The first prohibits the reception of letters from hosts that send greetings with incorrect syntax, the second - from hosts that send non-FQDN in a HELO request.

However, it’s not the FQDN that only the most stupid spammers (and MS products, but, as you know, they don’t write the laws), in the end it’s not difficult to introduce yourself to gmail.com. Therefore, we must take a closer look at HELO. For this are the option

reject_unknown_helo_hostname

Which forbids the reception of letters from servers that appear to be an address for which there is no A or MX record.

Sender - is it worth trusting him?

So, the server successfully presented itself to you, the next item on the program is the sender's address. From it, you can also extract a lot of useful information. Just want to note that the sender's address does not have to be from the same domain as the server itself. This is a common misconception, so keep in mind that it is not. One mail server can easily serve several domains.

However, if a domain is specified in the sender's address, which does not exist at all, then the letter is obviously not worth accepting. And you certainly should not accept a letter in which something is indicated as the return address, which is not a general address at all. For refusing to accept for such letters, there are two options:

reject_non_fqdn_sender
reject_unknown_sender_domain

The first is checking the address for spelling, the second is checking the existence of a domain.

Already not bad, but you can do something else. You can request a server serving the specified sender address for the existence of a user with this address. Indeed, it seems to be a good idea to make sure that the return address really exists, otherwise we may well receive a letter from an ephemeral phantom, about which no one ever heard.

Technically, this is implemented very simply: our server opens a counter SMTP session, trying to send an email to the sender's address. If it is possible to successfully pass the stage of sending RCPT TO with this address, i.e. if the receiving server suddenly does not declare that the specified mailbox is not on it, it is considered that the return address sent to us exists. The data (that is, the letter) of course, during verification, no data is transmitted, the session is terminated after RCPT TO.

For such a return address check is the option

reject_unverified_sender

It follows from the above that for any address from which you send mail from your domain, there must be a box on your server. Otherwise, your letters will not pass the return address check on the recipient side and, accordingly, will not be delivered to the destination. This is true for all mailings and other kind of one-way communication that does not require an answer. Always create mailboxes for all addresses from which you send mail. If you do not want to receive replies to any address, then just send letters to it in / dev / null, but you must accept these letters.

Does the recipient exist at all?

So we got to the last header of the envelope - to the recipient. Everything is simple: firstly, it would be good to check that the information transmitted to us is generally an e-mail address. For this is the directive

reject_non_fqdn_recipient

In addition, we would not like to receive mail to addresses for which we do not have mailboxes. To configure this behavior, you must first create a list of serviced mailboxes, then tell Postfix about it through one of the * _recipient_maps parameters of the configuration file, and then use the configuration file parameter

smtpd_reject_unlisted_recipient = yes

Or prohibit option that has the same effect:

reject_unlisted_recipient

In any case, Postfix will stop accepting emails for accepted domains if there is no mailbox for the recipient. However, this restriction does not affect the transfer of correspondence to addresses that are not in the accepted domains.

And finally, you can even prohibit the open forwarding of letters through your Postfix, leaving only the opportunity to receive letters to known addresses. For these purposes is the option

reject_unauth_destination

It prohibits sending all unregistered users (yes, you will have to configure SMTP authentication). Always use this option! Otherwise, quickly get into all sorts of DNSBL.

As a subtotal

So just based on the analysis of the three envelope headers you can filter out a huge amount of spam. However, spammers are tricky, so this is still not enough.

Greylisting

Sometimes mail servers are overloaded and cannot receive a letter. Do you think that they respond to incoming requests in this case? Oddly enough, they answer - the server is temporarily unavailable, try again later. In this case, no normal sender will ever consider that a letter cannot be delivered with all the ensuing consequences. On the contrary, the sender will attempt to deliver the letter later, putting it in turn to send. This fact can (and no doubt need!) Be used very effectively: each time you try to connect from an unfamiliar host, our server will send a temporary error message, and skip the email only the second time. This will eliminate almost all the remaining spam at once, since spam servers almost never make more than one attempt to deliver a letter (otherwise they would simply “fall” from the queue overflow). This technology is called Greylisting, and it is simply necessary to use it in modern realities.

The downside is a small (usually no more than half an hour) delay in the delivery of letters when you first connect from an unknown host. That is, if a server that is not yet known to our postfix, wants to send a letter, then on the first connection attempt, an error about temporary unavailability is sent to it. If the server retries the connection, the letter is accepted, and the server is logged in reliable nodes, and then letters from it are received without delay.

I also propose to read about setting up greylisting in Postfix on Google, it is easy and it will not turn out to be wrong.

Blocklist, or how not to do

Some mail administrators rely on so-called DNSBL (RBL) for spam filtering - blacklists of sites seen in spamming. So, never add any DNSBL checks to your mail servers. There are two reasons for this: the first, and most basic, lies in the second part of the first sentence of this section. Nodes are entered into these lists completely randomly and there are no guarantees that a normal host will not get there (on which, perhaps, at some point the spamming virus has settled, but now the virus has already been cured, or easier and much more real - one external IP for a large network in which the spammer started). The second reason is more trivial: the filtering mechanism proposed above is much more efficient than any DNSBL and does not rely on unverified data from third parties.

On its head, or look at the other side of the barricades

They learned how to filter spam, but now I will try to bring together all the information on what needs to be done so as not to get into spam.

For mail server administrators:

• Always make MX records referring to A.
• The A record for the mail server should always have a mirrored PTR record.
• The host of the HELO header must have an A or MX record.
• Always create SPF recordings (yes, this is just not necessary, but just a good tone rule).
• For all letters sent from the accepted domain, the box for the return address must exist and receive mail.

For those who send mail (from programs, from sites, etc.):

• Always send mail with an existing return address.
• Never send mail from a domain not under your control without checking the SPF rules for it. For example, gmail.com currently allows you to send letters on its behalf to any server, but yandex.ru and mail.ru report via SPF that sending on their behalf from third-party servers should attract close attention, which is interpreted by smart spam filters as an increase in spam rating for a given letter.
• Never send mail through incorrectly configured SMTP servers. Check the server for lice on the list above - it’s 5 minutes, the dig or nslookup utility will help you.

Summary

Of course, the proposed settings do not filter all spam. Therefore, it is quite possible that you will need to additionally use a context filter, which will analyze the content of the letters, for example, spamassassin , although I do not use anything like that.

However, never forget about the settings described in this article when configuring mail, because the above parameters can reduce the server load by orders of magnitude with respect to using only contextual filters, and additionally provide excellent filtering.

UPD: First of all, thanks to Vanav for critical comments. Secondly, it was originally implied by me, but apparently still an incomplete summary of the following options:

You have to decide for yourself what restrictions you are ready to put on your server and which ones are not. Many are skeptical of many of the above checks. However, they are all used on real SMTP servers on the Internet, so even if you turn on everything at all - you will not be alone. Therefore, if you notice a server that is configured incorrectly, do not be lazy to send it to the administrator a letter on this topic, perhaps you will help him avoid anger from users whose correspondence has not reached (if he has not already been expelled from work by the time you write the letter). And never forget that broken hosts can be added to the whitelist in order to receive mail from them without checks.

Current version of the article

The latest and most current version of this article is on the official Russian-language Wiki-resource for Ubuntu documentation. There you can freely improve and supplement the laid out reference and training materials, as well as add your own. If you have something to tell other Ubuntu users, then a huge request - write or edit the relevant article on help.ubuntu.ru . Even with small improvements and additions you will help thousands of people, and some of them, in turn, will describe something useful and interesting for you.