Technical recommendations for mailing lists

“Even if you receive any letter, you cannot read it.”
(Mark Twain)

We have already written about how to properly do mailings, improve their quality and effectiveness. They cited metrics, on the basis of which the reputation of the sender is built. They talked about the interface Postmaster Mail.Ru , with which they can be monitored. Many companies, both at the beginning of their development and rather large ones, neglect these rules, as a result of which problems start with the deliverability of letters, proceedings with technical support service, etc. But we hope that you are not one of them.
')
So, your project is gaining popularity and users like it, you are going to stay in touch with them. You have read the administrative requirements (about which we wrote earlier ) and are going to organize a mailing list responsibly and without spam for those users who are ready to receive it. Or maybe you're just going to set up corporate email. You pick up a mail server from a distribution kit, write a script, launch it and ... 70% of recipients did not deliver the letter, 15% had it in the Spam folder, and the rest cannot read what is written in it. I will try to tell about what to do to prevent this from happening in this article.

Why is it all can not work out of the box and need some additional recommendations? E-mail is one of the oldest Internet protocols, which has survived to this day with virtually no changes, but it has acquired a huge number of related standards and no documented application practices anywhere. The e-mail standards themselves do not include any method of authenticating the sender or preventing unwanted mailings and spam. Therefore, any mail server performs a number of additional checks in order to weed out unwanted correspondence (which may be more than 90%).

We have compiled a list of tips that will help pass these additional checks. Many of the recommendations listed below are not mandatory, but the more of them you follow, the higher the likelihood that there will be no problems with the delivery of letters.

Recommendations for the mail server administrator:

We start with ... IP addresses. You received an IP address (or a whole network) from a provider, hoster, or registrar. What should I do to use the selected address for the mail server, and is it valid?

1. Make sure the address is real and static.
Most mail servers restrict the reception of mail from dynamic addresses or from networks with address translation. The reason is that in such networks there are end-user computers that do not perform mail delivery. Accordingly, almost all mail that comes from these networks is spam sent through bots. Therefore, the reputation of these networks is hopelessly flawed.

2. Check whois information using whois utility or online services (easy to find). I will give a list of what you need to pay attention to:
2.1 The network must have a valid description. If the description contains information that is dynamic or intended to be distributed to end individual users (for example, an ADSL network), you will most likely also encounter many problems when trying to use it.
2.2 For the network, abuse-contact contacts should be indicated, and these contacts should be live and responsive. Otherwise, the network (and with it your IP) has a big chance of “flying into” black lists if there are spam incidents from any of its hosts.
2.3 If the network is European (RIPE recorder), the status of the network must be ASSIGNED, ASSIGNED PA or ASSIGNED PI - this means that this network is in use. Many registrars forget to mark the network as used, and some recipients prohibit receiving mail from unused networks.
Ask the registrar to enter the correct information in the network description, but remember that many potential recipients already “remember” the old information. Therefore, it is better to initially use IP addresses from conscientiously administered networks with a good reputation.

3. Check and configure the PTR record for the IP address. The PTR record must point to the real host name. It is desirable that
3.1 This name did not resemble the name of a dynamic host, that is, it did not contain the words dynamic, adsl, nat, long sequences of characters or groups of digits. For example, server.example.com is a good, valid PTR record.
3.2 If a really large amount of outgoing mail from several servers is planned, then it is advisable to name the servers in accordance with the draft standard " DNS Naming Convention for Outbound Internet Email Servers " and create the zone and A-record mxout provided by this document.
3.3 The name of the PTR record must be resolved back to the same IP address, that is, an A or CNAME record server.example.com must exist and be resolved to the IP address of the server from the example.
If the server will support multiple domains, there is no need to make PTR-records in each domain - one entry for any of the domains is enough. When checking DNS, it is better to use a third-party DNS server to make sure that the records are really visible from the outside.
PTR records are managed by an IP network operator, for example, an Internet or hosting provider. Usually this can be done independently through the hosting control panel or the support service of the provider.

4. Configure MX records for the domain from which distribution will be made. Even if you do not plan to accept any letters for this domain. At a minimum, you must correctly receive and process messages about the impossibility of delivering your letters and the postmaster @ address. The use of domains without MX records in the sender's address or in the SMTP envelope negatively affects the deliverability of letters. Some recipients validate the sender addresses from the SMTP envelope, so it is desirable that the server does not give errors when sending a letter to such an address.

5. Configure SPF record . SPF allows you to specify which servers are allowed to send mail on behalf of the domain. SPF is configured for the address used in the envelope-from (SMTP envelope).

6. Configure DKIM :
6.1 Generate a DKIM key pair and publish the public key;
6.2 Do not forget to configure the DKIM signature for all outgoing emails (more on this later).
DKIM allows you to sign emails that are sent on behalf of a specific domain. Currently, DKIM is already a must have technology, it is the technical basis for others - FBL, DMARC, as well as for services such as Postmaster Mail.Ru.

7. Post a DMARC policy. DMARC specifies exactly how to use SPF and DKIM, and allows you to completely eliminate phishing on behalf of the domain by publishing restrictive policies. And it also gives you the opportunity to receive in the form of structured reports all the information about attempts to violate your policy.
DMARC has not yet been adopted as an official standard, but it is already supported by the main players in the email market - Mail.Ru, Google, Yahoo and Microsoft. This is a very simple to implement and yet extremely effective solution.
Now you can try to raise the mail server.

8. In the Mail Transfer Agent settings, configure the hostname of the server, which is sent in the HELO command. This name must match the canonical server name from the PTR record (server.example.com). Very often, the default HELO uses names like localhost.localdomain, and many mail servers refuse to accept mail with such a HELO. Set up a SPF record for a name from HELO, this RFC 7208 requirement that allows you to deliver service letters with an empty SMTP envelope address.

9. Enable DKIM support in your mail server. Some servers have built-in support, some can be implemented using free software (for example, OpenDKIM), for Microsoft Exchange / IIS SMTP there are inexpensive and free plug-ins. When setting up, do not confuse DKIM (RFC 4871 / RFC 6376) and DomainKey (RFC 4870). DomainKey is an outdated standard that has never been adopted. Doing his support is optional. DKIM can be recognized by the presence of the DKIM-Signature signature in the letter headers. To sign the headers and body of the letter it is better to use the relaxed mode.

10. Configure the postmaster box.

11. Avoid configurations that lead to unauthorized relay of letters, otherwise all the work will go down the drain along with the reputation of the server.

12. Test your server configuration by sending emails to basic email services through a standard email client, for example, Thunderbird. Check the headers of received emails. Make sure to:
12.1 correctness of HELO;
12.2 availability and passage of SPF-, DKIM- and DMARC-checks on servers that support DMARC;
12.3 impossibility of unauthorized relaying through your server;
12.4 that postmaster @ and addresses for DMARC and FBL reports are received
12.5 The server correctly generates reports about undeliverable emails.

13. Subscribe to FBL (FeedBack Loop). FBL are provided by many major mail services. A subscription will give you the opportunity to receive information about all complaints about spam e-mail from your domain.

Now you can customize the mailing / write scripts to send emails. Getting started.

14. Handle messages about the inability to deliver letters. Remember that there are two ways to get an error: in an SMTP session (in this case, a message about undeliverable can be generated by your server) or by a letter with a undelivered delivery report (NDR) from the recipient server. It is necessary to process both cases. At the same time, it is imperative to react and exclude the user from mailings with repeated or permanent (permanent) delivery errors to his address. The presence of a large number of invalid recipients can lead to a decrease in reputation and / or triggered greylisting.

15. Set the SMTP address. MAIL FROM: (envelope sender) - the sender address that is used at the SMTP protocol level. It may not coincide with the address of the sender from the header of the letter From :. For normal DMARC operation, it is desirable that the From: address and the envelope sender address be from the same domain. And, of course, the letter must have a valid DKIM signature for this domain and pass SPF checks. Incorrect sender address in the envelope (envelope) - the most common and serious error in mailings from various websites.

16. Do not use public mail addresses in the From: and MAIL FROM addresses, since such mailings will not pass SPF / DKIM authentication in DMARC. Use your own domain addresses.

17. In the Content-Type headers for all text parts of the letter, specify the correct encoding. Make sure that the actual encoding of the text matches the one specified in the header. It is advisable to use the same encoding in all headers and parts of the letter. Currently the recommended, most widely supported text encoding is UTF-8. If the encoding is not specified (or is specified incorrectly), the text may be displayed differently by different services and mail programs.

18. Generate From: headers, Message-ID: and Date: headers directly in the script for sending letters. Make sure that these headers, especially Date :, are formed in the correct format. By standards, these headers are formed together with the text of the letter. If they are missing or formed incorrectly, one of the transit servers can add headers, which may violate the integrity of the DKIM signature.

19. Make sure that in all service and other headers, including the subject line (Subject), the names of attached files (Content-Type and Content-Disposition), non-ASCII characters, in particular Cyrillic, are encoded in quoted-printable or base64. According to the standards, non-coded eight-bit characters cannot be found in letter headers; Some servers do not accept such letters. If you are targeting foreign subscribers, then it is desirable that the letter does not contain uncoded 8-bit characters at all.

20. Limit string length and use correct string terminators.
20.1 The maximum allowed string length in email is 998 8-bit characters. When using UTF-8, one displayed character may take several octets, so try to make the text in the strings shorter or encode the text in base64.
20.2 The correct terminator of strings in email is CRLF (characters with codes 13 and 10). In Unix, the standard line terminator is LF. If the script or MTA being used does not replace the line terminators, this can lead to problems — incorrect display of the letter or cutting off part of the text. Errors can cause double replacement of terminators. Check the documentation or test whether the configuration you are using should transcode the ends of the lines.
20.3 String terminators should not split UTF-8 characters so that there is no situation in which a terminator is added to a long string, for example, between two octets of a single Cyrillic character. Therefore, the breakdown of the text should be made before the formation of the letter.
Encoding text parts to base64 increases the size of a letter by about a third, but it saves from all the problems associated with terminators of strings in text parts.

21. Try to adhere to a fairly simple letter structure, avoid deep nesting of multipart parts (that is, including one component in another). If there are more than one multipart-parts of each type in the letter (one multipart / mixed by one multipart / alternative and one multipart / related), most likely, the letter is not optimally formed.

22. Correctly put the Content-Disposition of each part as an attachment (attachment shown separately from the letter) or inline (an object displayed as part of the letter - for example, a picture in the text). It often happens that the document attached to the letter is marked as inline, although it is intended for downloading by the user, or the logo in the text of the letter is marked as attachment, although it should not be visible in the list of attached files.

23. Form the text (plaintext) part of the letter as an alternative to the HTML part. Do not forget that the user can read the letter, for example, from a cell phone. Correctly and beautifully convert HTML content to text is not always possible. By adding a text part to the letter, you can be sure that the text will be displayed exactly as you expect it.

24. Test your script work on test boxes from different email services. Do not forget to download the original letter that came to the box and check:
24.1 correctness envelope sender;
24.2 passing SPF, DKIM and DMARC checks;
24.3 availability and validity of text encodings in the Content-Type headers for all text parts (including HTML);
24.4 in HTML parts - the validity of the encodings indicated in meta tags (if any);
24.5 display of various characters, including Cyrillic in text, HTML-parts and file names (if you plan to send files);
24.6 availability of proper Content-Disposition and Content-Transfer-Encoding for each part of the letter (except multipart);
24.7 display of pictures and attachments;
24.8 string terminator correctness;
24.9 correct splitting and display of long texts in HTML and plain text parts;
24.10 Make sure that the From: Date: and Message-ID headers are formed by your server, signed with DKIM-signature and valid.

Now you can go to the layout of letters. Recommendations for the coder:

25. Keep it simple. Do not forget that webmails are forced to filter some tags and attributes in order to ensure security, and they all do it differently. Avoid using classes, minimize the use of styles, especially for positioning. The good old layout on the tables is the most portable.

26. Do not attempt to typeset for the web interface of a specific mail server. Users often set up redirects, mail collectors, or read email in email programs.

27. Think about the user. He can open your letter both on a huge retina display and on an old phone on top of Everest, trying to grab a GPRS connection. Accordingly, the letter should load quickly, but look beautiful even with images turned off.

28. No inscriptions in small print. They get nervous and make you suspicious. Feel free to show the user that you know and respect their rights and their responsibilities.

29. Choose the most appropriate way to embed images. There are the following options, each with its own pros and cons:
29.1 External images: in terms of sending speed, this option is the most preferred. External pictures do not weigh anything and do not complicate the structure of the letter. However, many applications and email services require user permission to display such images. In addition, the server on which they are located must be sufficiently reliable so that when opening a letter, it does not cause “hangs” due to the inaccessibility of the picture. Use them when having pictures is optional.
29.2 Inline Pictures. Sent along with the contents of the letter. Complicate the structure of the letter and increase its weight, but it can be quite large and is usually displayed by default.
29.3 Data URI. The content of the picture is inserted directly into the tag. Do not complicate the structure of the letter, almost always shown in the web mail, sometimes even with disabled images. As a rule, they have a size limit, suitable for small icons.
29.4 Font pictures. Images of Unicode characters from standard fonts or using embedded custom fonts. Unique property - can be scaled along with the text. . , , , .

30. URI (http://, https:// mailto:), URI , "//example.com/" - , , .. .

31. «» .

, , , - ? — , . , Mail.Ru, abuse@corp.mail.ru. – , , .

, , – .