Scalable Mail System Architecture

In this article, we consider one of the options for implementing a scalable architecture of a large mail system.



December 6, 2012. Google stopped registering new accounts for the free version of Google Apps.



Customers of our company constantly have a need for e-mail serving their sites.

We used to set up Google Apps for them, but after December 6, having studied the solutions offered on the market, we decided that it was time to build our own email system.

')

As you know, the appetite comes with eating. If you have already decided to build something of your own, then you should immediately lay the opportunity for growth.

For the projected mail system, the following requirements were formulated:

• scalability (unlimited number of serviced domains, total mailboxes 100 terabytes and more);
• fault tolerance (all intermediate services must be duplicated);
• extensibility (adding new nodes to the system should be easy and simple).

We started by choosing the storage for the letters ...

In fact, the choice was small:

• dovecot / cyrus with file system storage via maildir / mailbox;
• dbmail with storage in the database.

After assessing the number of services provided out of the box, as well as studying the nuances of use, we decided to stop at dbmail .

Here is a short list of goodies that dbmail provides:

• access to the boxes via IMAP, POP3;
• sieve scripts for sorting mail;
• receiving mail via smtp and lmtp protocols;
• administration through cli and SQL queries.

Postfix was selected as MTA. Since we have been working with him for a long time, we confidently believe that it is quite a fast, flexible and easily customizable program. Moreover, the dbmail website has instructions on how to configure the postfix + dbmail bundle.



And the best proxy of all times and peoples works as a load balancer: nginx.

A web interface to mailboxes can be provided by any of the dozens of programs available. For a start, it was decided to use RoundCube.

As a result, the following project architecture emerged:

1. At the forefront, we run nginx with the Mail module enabled, it accepts SMTP, IMAP, POP3 connections from clients and, based on the user name and password, proxies the connection to the correct server.

2. In the depth of the defense works a lot of postfix and dbmail, each of which serves its own domains.



To the outside world, it all looks like one big mail server.

In theory, scalability has been achieved, now the task is to implement.

Each system module will work in its OpenVZ container. As the load increases, we will clone / transfer containers to new physical servers.



In separate containers we put nginx, postfix with different mail filters, dbmail and the storage of letters in the MySQL database we place in the same container, because dbmail does not have to heavily load the processor, and the data volumes transferred between dbmail and mysql will be large (duplication of data in the storage Letters will be implemented through the MySQL replication master-slave mechanism, organizing a hotbackup module for each dbmail module.



Also, for convenience, web servers with a roundcube and a control panel of the entire system can be separated into separate containers.



All containers communicate with each other over a local network with “gray” addresses.

It was planned to issue real Internet addresses only to nginx-containers.

In the process of installing and configuring nginx, it turned out that he cannot proxy SMTP sessions without authorization. Solving this problem slightly changed the originally planned project architecture, which led to increased resiliency and extensibility. We refused to proxy SMTP traffic through nginx, and decided that postfix modules will receive it directly.



All postfix modules of the system were made the same, each with its own real IP address, and we determine which dbmail module to send mail through transport_maps.

As a result, any postfix-module can accept mail for any domain in the system, apply various filters and deliver to the necessary storage.

They will be balanced through round-robin records in the DNS:

nginx1 IN A 1.1.2.1 nginx2 IN A 1.1.2.2 postfix1 IN A 1.1.1.1 postfix2 IN A 1.1.1.2 postfix3 IN A 1.1.1.3 mx1 IN A 1.1.1.1 IN A 1.1.1.2 IN A 1.1.1.3 mx2 IN A 1.1.1.2 IN A 1.1.1.3 IN A 1.1.1.1 mx3 IN A 1.1.1.3 IN A 1.1.1.2 IN A 1.1.1.1 imap IN A 1.1.2.1 IN A 1.1.2.2 pop3 IN A 1.1.2.1 IN A 1.1.2.2 smtp IN CNAME mx1 _spf IN TXT "v=spf1 ip4:1.1.1.1 ip4:1.1.1.2 ip4:1.1.1.3 ?all" 

The necessary records are added to the client's domain:

 @ IN MX 10 mx1.dbmail.io. @ IN MX 20 mx2.dbmail.io. @ IN MX 30 mx3.dbmail.io. @ IN TXT "v=spf1 a mx include:_spf.dbmail.io -all" 

Mail clients of users are configured to receive mail with pop3.dbmail.io or imap.dbmail.io, and send mail via smtp.dbmail.io



Only dbmail-modules were not duplicated and unique, but they are not directly accessible from the Internet, so there is no need to dynamically distribute the load on them yet.

Each of the dbmail modules stores mail for several domains, if the module’s resources will run out,

Some domains can be evicted into a separate module.



And in the future there are plans to move to the storage of mail in a dedicated cluster database and unify dbmail-modules.

The result was such a scalable, fault-tolerant and expandable mail system:

All elements of our postal system communicate with each other using standard protocols and, as it was naively thought, everything should have worked immediately and without the need to build props.

Unfortunately, the design had to finish the file.

History settings and debugs, read the following articles.