Riddles and Myths SPF

SPF (Sender Policy Framework), the full name can be translated as "Sender Policy Basics for Authorizing Domain Use in Email" - a protocol through which an email domain can specify which Internet hosts are authorized to use this domain in the SMTP HELO and MAIL FROM commands. Publication of the SPF policy does not require any additional software and is therefore extremely simple: just add a TXT record containing the policy to the DNS zone, an example of the record is at the end of the article. To work with SPF there are numerous manuals and even online designers.

The first version of the SPF standard was adopted more than 10 years ago. During this time, numerous implementations were created, practical applications were developed, and a new version of the standard appeared. But the most surprising thing is that for some reason SPF, more than any other standard, has been overgrown in 10 years with an incredible amount of myths and delusions that wander from article to article and with enviable regularity pop up in discussions and answers to questions on the forums. And the protocol would seem so simple: the implementation takes only a couple of minutes. Let's try to remember and disassemble the most frequent misconceptions.

TL; DR - recommendations at the end.

1. Misconception: SPF protects my domain from fakes

In fact: SPF does not protect the sender’s address visible to the user.

Explanation: SPF does not work at all with the content of the letter that the user sees, in particular with the sender's address. SPF authorizes and checks addresses at the transport mail (SMTP) level between two MTAs (envelope-from, RFC5321.MailFrom aka Return-Path). These addresses are not visible to the user and may differ from those visible to the user from the From header of the letter (RFC5322.From). Thus, a letter with a fake sender to From can easily pass SPF authorization.

2. Misconception: I will implement SPF and it will be safer and less spam

In fact: most likely, you will not notice any changes in security and spam.

Explanation: SPF is initially an altruistic protocol and by itself does not benefit the one who publishes the SPF policy. Theoretically, your SPF deployment could protect someone else from fake emails from your domain. But in practice, even this is not the case, because the results of SPF are rarely used directly (see below). Moreover, even if all domains published SPF, and all recipients forbade receiving letters without SPF authorization, this would hardly lead to a decrease in the level of spam.

SPF does not protect the sender from spamming or spam directly, however, it is actively used and is very useful in spam filtering systems and for protection against fake emails, since allows you to bind a letter to a specific domain and its reputation.

3. Misconception: SPF has a negative (positive) effect on the deliverability of letters.

In fact: It all depends on the type of letter and the way in which it is delivered.

Explanation: SPF by itself does not affect the deliverability of letters by the normal flow, and negatively affects the wrong implementation or indirect flow of letters (indirect flow), when the user receives letters from the wrong server from which the letter was sent, for example, to redirected letters. But spam filtering systems and reputation classifiers take into account the presence of SPF, which on the whole, on the main stream of letters, gives a positive result. Unless, of course, you send spam.

4. Misconception: SPF authorizes the sender of the letter

Actually: SPF authorizes the mail server sending the letter on behalf of the domain.

Explanation: First, SPF works only at the domain level, not for individual email addresses. Secondly, even if you are a legal mail user of a specific domain, SPF does not allow you to send an email from anywhere. In order for a letter to pass SPF validation, you should only send through an authorized server. Thirdly, if you have authorized a server via SPF (for example, you have allowed sending emails from your domain through some ESP or hosting provider) and it doesn’t implement additional restrictions, then all users of this server can send emails on behalf of your domain. All this should be considered when implementing SPF and authentication of letters in general.

5. Misconception: letters that did not pass SPF authorization are eliminated.

Actually: SPF-authorization or its absence in general does not drastically affect the delivery of letters.

Explanation: The SPF standard is only an authorization standard and explicitly indicates that actions applied to unauthorized letters are outside the standard and are determined by the local recipient policy. The refusal to receive such letters leads to problems with letters going through indirect delivery routes, such as redirects or mailing lists, and this fact should be taken into account in local policy. In practice, a strict failure due to the failure of SPF authorization is not recommended for use and is possible only when publishing with the -all (hardfail) policy in the absence of other filtering tools. In most cases, SPF-authorization is used as one of the signs in the classifiers or as one of the factors in the weight systems. Moreover, the weight of this factor is very small, because violation of SPF-authorization is usually not a reliable sign of spam - many spam letters pass SPF-authorization, but not completely legal - and this situation is unlikely to change drastically sometime. With this use, there is no difference between -all and ~all .

The fact of having SPF authorization is important not so much for delivering the letter and deciding whether it is spam, but for confirming the sender's address and contacting the domain, which allows the letter to use not the IP reputation, but the reputation of the domain.

The decision about the action on the letter that has not passed authorization is much more affected by the policy of DMARC. The DMARC policy allows you to drop (or put in quarantine) all unauthorized emails or their percentage.

6. Misconception: in SPF you must use -all (hardfail), is it safer than ?all or ~all

In fact: in practice, -all does not affect anyone’s security, but it negatively affects the deliverability of letters.

Explanation: -all results in blocking emails sent via indirect routes by those few recipients who use the SPF result directly and block emails. At the same time, this policy will not have a significant effect on most spam and fake emails. Currently, the most reasonable policy is considered ~all (softfail), it is used by almost all large domains, even those that have very high security requirements (paypal.com, for example). -all can be used for domains that are not sending legal emails. For DMARC, ~ and ? are equivalent.

7. Misconception: it is enough to register SPF only for domains from which mail is sent.

Actually: it is also necessary to register SPF for the domains used by HELO mail servers, and it is desirable to prescribe a blocking policy for A-records and wildcard that are not used for sending mail.

Explanation: In some cases, in particular, when delivering an NDR (undeliverable message), DSN (delivery confirmation message) and some auto-responses, the sender's address in the SMTP envelope (envelope-from) is empty. In this case, the SPF verifies the host name from the HELO / EHLO command. You need to check what name is used in this command (for example, looking at the server configuration or sending a letter to the public server and looking at the headers) and register the SPF for it.

Spammers do not have to send spam from the same domains from which you send letters, they can send on behalf of any host with an A or MX record. Therefore, if you publish SPF for altruistic reasons, then you need to add SPF for all such records, and preferably another wildcard (*) for non-existent records.

8. Misconception: it is better to add a special type of SPF record to the DNS, and not TXT.

Actually: you need to add a TXT record.

Explanation: In the current version of the SPF standard (RFC 7208), SPF records are deprecated and should not be used anymore.

9. Misconception: the more (a, mx, ptr, include) I include in the SPF record, the better, because there is less chance of a mistake

Actually: ip4 possible, it is necessary to reduce SPF record as much as possible and use in it only network addresses via ip4 / ip6 .

Explanation: A limit of 10 DNS requests has been assigned to the resolution of the SPF policy. Exceeding it will result in a permanent policy error (permerror). In addition, DNS is an unreliable service, and for every request there is a probability of failure (temperror), which increases with the number of requests. Each additional a or include record requires an additional DNS request; for include also necessary to request everything specified in the include record. mx requires a request for MX records and an additional request for A-record for each MX server. ptr requires an additional request, moreover, in principle, it is unsafe. Only network addresses listed via ip4 / ip6 do not require additional DNS queries.

10. Misconception: the TTL for the SPF record should be made smaller (more)

In fact: as with most DNS records, it is better to have a TTL in the range from 1 hour to 1 day, reducing it in advance when implementing or planning changes and increasing it with a stable working policy.

Explanation: A higher TTL reduces the likelihood of DNS errors and, as a result, temperror SPF, but increases the response time if you need to make changes to the SPF record.

11. Misconception: if I do not know from what IP my letters can leave, then it is better to publish the policy with +all

Actually: a policy with an explicit +all or implicit rule allowing mailing on behalf of the domain from any IP addresses will negatively affect mail delivery.

Explanation: This policy has no meaning and is often used by spammers to provide SPF authentication on spam emails sent through botnets. Therefore, a domain that publishes such a policy has a very big chance of being blocked.

12. Misconception: SPF is useless

Actually: SPF is needed.

Explanation: SPF is one of the sender's authorization mechanisms in email and a way to identify a domain in reputation systems. Currently, major postal service providers are gradually beginning to require the authorization of letters, and letters that do not have authorization may be subject to "penalties" for their delivery or display to the user. In addition, emails that have not passed SPF authorization may not return auto-responses and delivery reports or undeliverable. The reason is that these response categories, as a rule, go to the address of the SMTP envelope and require that it be authorized. Therefore, SPF is required even if all letters are authorized by DKIM. Also, SPF is absolutely necessary in IPv6 networks and cloud services: in such networks it is almost impossible to use the reputation of IP addresses and letters from addresses not authorized by SPF, as a rule, are not accepted. One of the main objectives of SPF, defined in the standard, is the use of domain name reputation instead of IP reputation.

13. Misconception: SPF is self-sufficient

Actually: DKIM and DMARC are also needed.

Explanation: DKIM is required for passing letters through various forwarding. DMARC is required to protect the sender's address from counterfeit. In addition, DMARC allows you to receive reports of SPF policy violations.

14. Fallacy: if the server forwards emails, you must use SRS (Sender Rewriting Schema) to save SPF authorization.

In fact: SRS should be used only if you want to give the redirected letters authorization (and, as a result, reputation) of your domain.

Explanation: SRS rewrites the envelope address of the sender to the address from the domain of the forwarding server, so the SPF authorization is performed for the letter with the domain performing the redirection. However, the sender's RFC5322.From domain for such a letter will not match the domain of the envelope for which the SPF authorization is being passed. From the point of view of DMARC, such authorization is not considered valid. In addition, the forwarded letter receives domain authorization if there are situations in which spam letters are redirected (for example, the general flow of letters is forwarded in which even if there is good spam filtering there is a certain amount of spam) - this negatively affects the reputation of the redirecting domain and vice versa, distributes spam domain reputation. It makes sense to use SRS where incoming emails go through additional authorization and spam filtering, for example, in mailing lists with restriction by senders, possibly in conjunction with the From rewrite option to preserve DMARC authentication. In all other cases, it is better to pay attention to the fact that the shipments preserve the integrity of the DKIM signature.

15. Misconception: spf1 is good, spf2.0 is better

Actually: you must use v=spf1 .

Explanation: spf2.0 does not exist. Posting a spf2.0 entry may produce unpredictable results. spf2.0 never existed and was not a standard, but there is a reference to it in a series of experimental standards, the most famous of which is RFC 4406 (Sender ID). This series of standards was developed by an independent working group, in parallel with the adoption of the spf standard, which caused confusion. Sender ID, which was supposed to solve the problem of address spoofing, did not become a generally accepted standard and should be abandoned in favor of DMARC. Even if you decide to use the Sender ID and publish the spf2.0 record, it will not replace the spf1 record.

I almost finished writing this article when I was intercepted by the customer support service and I strongly (with the threat of brute force) recommended recalling the following SPF nuances that they often come across when solving problems:

1. The SPF policy must end with the all or redirect directive. After these directives, nothing should go.
2. all or redirect directives can occur in a policy exactly once, they replace each other (that is, in one policy all and redirect cannot be simultaneously).
3. The include directive does not replace the all or redirect directives. include may occur in a policy several times, but the policy should still be terminated with all or redirect directives. A policy included in an include or redirect must also be a valid policy ending in an all or redirect directive. At the same time, it -all not matter for include which rule ( -all , ~all ?all ) is used for all in the included policy, and for redirect there is a difference.
4. The include directive is used with a colon ( include:example.com ), the redirect directive is with an equal sign ( redirect=example.com ).
5. SPF does not apply to subdomains. SPF. Not. Spreads. On. Subdomains. SPF DOES NOT SPREAD ON SUB-DOMAINS. (and DMARC, by default, is distributed). You need to publish SPF for each A or MX record in the DNS, which is (or can be) delivered mail.

Summary

• Be sure to create a policy for all MX and preferably A records.
• Add SPF policies for names used in the HELO / EHLO mail server.
• Publish the SPF policy as a TXT record with v=spf1 in DNS.
• In policy, mainly use network addresses specified via ip4 / ip6 . Place them at the beginning of the policy to avoid unnecessary DNS queries. Minimize the use of include , do not use a include , a , without extreme and irresistible need, do not use mx and never use ptr .
• Specify ~all for domains from which emails are actually sent, and -all for unused domains and records.
• Use small TTLs for implementation and testing, then increase TTL to reasonable values. Reduce the TTL in advance if you need to make changes.
• Be sure to test the correctness of the SPF policy, for example, here .
• Do not be limited to SPF, try to implement DKIM and be sure to implement DMARC. DMARC helps protect your fake emails and allows you to receive information on SPF violations. This will allow you to identify fakes, indirect flow of letters and configuration errors.
• After implementing SPF, DKIM and / or DMARC, check them using https://postmaster.mail.ru/security/ .
  SPF and DMARC are checked for the current state, the presence of DKIM is checked against the statistics for the previous day and will only be shown if there are correspondence with mailboxes on Mail.Ru in the previous day or earlier.

An example SPF policy: @ IN TXT "v=spf1 ip4:1.2.3.0/24 include:_spf.myesp.example.com ~all"