How the infrastructure of email-mailing services is built: Experience Pechkin-mail.ru

Mailing lists are an effective tool for increasing sales and marketing support projects. That is why more and more companies are resorting to the use of appropriate tools for creating and implementing mailings.

In our today's topic we will talk about how the infrastructure of such services is built, using the example of the Pechkin-mail.ru project.
')

Why do we need such services

A common misconception is that email creation services are used by spammers to send emails with offers to buy Viagra. In fact, services like ours (or foreign Mailchimp) are needed to implement email-mailings to the addresses of people who have subscribed to the company's newsletter (for example, on its website) - that is, they explicitly expressed their consent to receive letters.

Companies can send their customers informational messages about the operation of their systems (for example, notification of new comments on Habré), offer some products or discounts (you can unsubscribe at any time from receiving such letters).

Respectable services of email-mailings with spammers simply do not work and rigidly moderate the content being sent so as not to fall under the sanctions of the postal services.

How it works

In fact, Pechkin-mail.ru service does not independently deliver letters, but creates them in the form of a final html-code, forms them in MIME-format, personalizes, tracks the fate of the letter (was it open, etc.), but not “ Inserts them in mailboxes.



There are many ESPs for this task - for example, andrill, smtp.com, mailgun, sendgrid. With the help of these cloud-based email providers, we are able to achieve an extremely high level of letter deliverability, high controllability of the pool of ip-addresses and low level of operating expenses.

Choice of technology

From the experience of creating high-load web services for business (hundreds and thousands of users), we know that the main technical expenses go to supporting the current code, developing new functions. Trite for salary. That is why often you should not be wise in the choice of programming language and other technologies. The most important thing to understand is that 99% of problems can be resolved by the correct architecture.

Below is information about the main technical characteristics of the Peckin-mail project:

Programming language

Service written in PHP and JS. At the same time, a number of functions use proprietary libraries in C (their creation is a topic for a separate topic).

Databases and OS

The mailing service is a very high load service: we send up to 10 million letters a day. And a significant proportion of these letters are tracked (openings, clicks, etc.). All this generates database queries. Accordingly, it is necessary to avoid problems with it by all means (failures are generally unacceptable).

We solved this problem for a long time and came to the cluster on Percona (there is an excellent article on Habré about its configuration). This is a fault-tolerant, horizontally-scalable cluster with master-master replication (a nice bonus is “hot” backups without performance degradation).

Percona is used to implement business logic and generate reports, in addition, we use MongoDB for queues and online storage of various data. Debian is used as the OS, virtualization is implemented via OpenVZ.

Iron and channels

As already mentioned above, the peculiarity of mailing list services is a large amount of data, with which it is necessary to work a lot and often. This means that we cannot do without fast drives (we are now using SAS, but, of course, SSD is more optimal).

Now Pechkin-mail.ru service runs on DELL PowerEdgeTM R720 DX-150 64Gb RAM, 4x600Gb SAS, Hetzner EX-5, Hetzner EX-10 servers. Additionally, Amazon EC2 (10 instances for peak load) and Selectel Storage are used, which is responsible for hosting client images and their delivery to mailing list recipients (we recommend it to everyone because of the cost).



Also, a high-quality and wide channel is very desirable - SMTP traffic is very heavy, often, these are pictures and content sent to hundreds of thousands of recipients in a short time. If you take the average mailing to 150Kb (there are almost no pictures, they are hosted on a third-party server), then even sending it to 10,000 subscribers is already 1.5GB of traffic. You need to keep this in mind when designing a system.

Important features

A good email service should be able to successfully solve a variety of tasks. Below is a list of the most important features of such tools.

Work with address databases

To work with any service that processes customer data, the speed and ease of downloading this information is important. For the mailing service, it is important to quickly and without problems for the user to download address databases containing email addresses, first names, last names, and other additional data.

We have implemented for this purpose a bunch of Excel and the commercial library libxl - a description of what happened in the end was published in a separate article on Habré .

In addition, Pechkin provides the ability to segment mailings - for example, based on the address base fields, by subscriber activity in previous mailings, or the date the addresses were added to the database.

Create mailings

Layout email-letters - a separate and very interesting topic. We published the rules for the design of mailings and talked about how to embed videos from YouTube into letters.

In addition, there is often a need to create a plain text version of the mailing list - we decided to automate this task and developed a generator of text versions of letters from HTML using the lynx text browser ( here you can read more about this solution).

In addition, based on statistical data on the effectiveness of mailings, it is possible to fine-tune the sending of letters - for example, sending at a certain time, in even chunks, creating A / B mailing (two mailings for testing the effectiveness of different headers, letter texts, etc.) .

Spam fighting

One of the main tasks of the email-marketing service is the fight against spam. In "Pechkin" it is implemented by the following mechanisms:

• Pre-moderation of the sent mailings - all mailings are reviewed by the service staff manually in order to check their compliance with the moderation policies (most of them are based on the Administrative and Technical rules of mail.ru). In automatic mode, moderation is carried out only for the most "trusted" customers;
• Automatic reputation system - we talked about how it works in one of the previous topics .
• Control of beacons spam filters . We work with Mailgun, CloudMark and many others spam filters. In addition, there are automatic APIs that allow you to label potentially “spamming” mailings and monitor them during the sending process.
• Automatic blocking for spam . If the report of the postal provider is evidence that the mailing is detected as spam, it stops immediately and a typical unlock instruction is offered to the user (including filling in the form on the mail provider's website and explaining the legitimacy of address databases and content). According to the results of the check, the mailing is either unblocked or the account is permanently blocked for spam.

No need to reinvent the wheel

A good work of a high-load service can only be achieved if it is properly designed without creating all kinds of crutches. In order to avoid this, we rather actively use the available cloud solutions (Amazon EC2, Selectel Cloud Storage). It is convenient and inexpensive: for example, during peak loads (sending more than 30 mailings at the same time), we include additional instances on Amazon EC2.

Thanks to the competent construction of the infrastructure, we managed to achieve the service uptime indicator in 2014 at 99.98%, and only two support staff manage to cope with more than 2.5 thousand paying customers.