POP3 and IMAP: history of mail protocols and collection of IMAP letters in Yandex. Mail

(Dedicated to Mark Crispin 07/19/1956 - 12/28/2012)

Some time ago, IMAP mail collectors appeared in Yandex.Mail. In this post we will talk a little about two important mail protocols - POP3 and IMAP - and how they appeared and are used in modern mail systems. This is not always as simple and obvious as you would expect from technologies that have been developing for almost 30 years.

We begin our excursion into the history of the protocols through which you receive your hundred letters every day.
')



When you chat on the web for many months and years, you will overwrite many letters, and these are hundreds of megabytes and gigabytes of correspondence and files. Often these are not just useful files, you can recall and restore life milestones. These data are more valuable than the contents of the local disk of the computer.

Sometimes there is a task to move to a new and more convenient mail system, but the accumulated archives of letters interfere. Throw them in the same place? Sorry. Passwords are forgotten. It happens that mobile numbers and email addresses entered to restore them are lost. One day you can lose archives forever. Download to local hard drive? Rewrite on disc or flash drive? But they are unreliable: they break, are lost, spoil.

The most logical solution is to import the mail archive into a new mailbox. But it is not enough to do it once, because important letters may continue to arrive in the old box. You can configure unconditional redirection of all incoming mail to another address, and from a technical point of view it is the most preferred option. But we are not robots, and not always technically economical solution is convenient to use.

For a long time, the only way to transfer mail archive between mail systems with constant background synchronization of new letters was POP3. The ability to collect mail from external POP3 servers is implemented everywhere and well. In any case, as good as the protocol allows, and it has a lot of unpleasant restrictions.

POP3 has been leading its history since 1984, when one of the employees at the Institute of Computer Science at the University of Southern California, Joyce Reynolds, published RFC 918, a proposal for the standard Post Office Protocol for receiving e-mail. Four years later, the third edition of the POP protocol appeared, and the current, modern version of the POP3 standard was published in spring 1996, almost 17 years ago.

An important principle of POP3 is optimization for short connections to the mail server in terms of expensive and slow communication. Initially, within each connection, it was supposed to download all the letters from the box to a local disk, and then clear the entire contents of the box. Your mailbox for letters and newspapers works in the entrance of an apartment building in the same way. In the first version of POP there were not even commands for reading individual letters, only for the complete unloading of everything.

Now it is difficult to imagine that emails were not stored on servers. They were transferred to the local computer at the first opportunity and read, sorted by addressees, topics and importance locally.

Interestingly, the POP2 protocol provided for the ability to work with several folders on the server, but it turned out to be unclaimed, and the distribution protocol itself was not received. Therefore, in the POP3 team FOLD, which realized this opportunity, was removed. POP2 overtaken time.

Now in POP3 there is no possibility to download a folder structure from the server, only a “flat” list of letters, consisting, as a rule, of either incoming emails or a combination of user folders. It is impossible to take into account the flags of readability and importance. Despite these limitations, the protocol is still widely used, mainly because of its simplicity and very wide support in any devices.

There have been many attempts to improve POP3, but none of them have achieved such success as the IMAP protocol, which has been developing in parallel since 1985. The history of IMAP is also very interesting. For example, the first implementation was made in Lisp, and its legacy remained forever in the protocol as S-expressions that encode complex server responses, such as BODYSTRUCTURE.

IMAP author and ideologist Mark Crispin laid down the principle of permanent storage of letters on the mail server. IMAP turned out to be one of the earliest "cloud" Internet protocols designed for the fact that local storage on a personal computer is unreliable. In addition, a person can have several personal computers and other terminals for working with mail - the basic things for us today.

The latest version of IMAP, 4rev1, is described in RFC 3501, released in 2003. Despite the apparent age, the protocol turned out to be alive thanks to the extension mechanism provided for in the early stages. This mechanism, of course, is also not without flaws, but nevertheless, it allowed various people to release more than fifty public extensions , many of which were developed quite recently and have found wide application.

Modern mail system without IMAP access is nonsense. For several years, Yandex.Mail has supported IMAP as a server for working from popular client programs such as Outlook, Thunderbird, Apple Mail, as well as numerous mobile clients. By the way, thanks to smartphones IMAP received a second wave of development. If on personal computers for a long time the vast majority of users have opted for the web interface to their mail, then with mobile devices the situation is not at all like that. Fast and beautiful IMAP-clients, for example in iOS, force to reconsider the approach to IMAP as to the choice of exclusively professional and "advanced" users.

Recently, a function of an IMAP client, mail collector from external servers via IMAP, appeared in Yandex.Mail, in addition to the POP3 collector.

The function requires almost no configuration and is designed for background operation after the first power up. When collecting mail via IMAP because of the above described features of the protocol in your mailbox, you will see the folder structure you are used to when you used the previous mail system. All read letters will remain read, and unread - unread.



You can enable collection of folders in Yandex.Mail from all mail systems that support the IMAP protocol. The protocol is not simple, each IMAP server implementation has its own quirks, and it was important for us to first of all process the most massive version of the transition from the old mail to the new one.

As before, besides the letters themselves, contacts from the address books of the most common postal services are imported.

The IMAP protocol is noticeably more complicated than POP3 and simply by a set of features and commands, and by some basic principles of operation. For example, IMAP uses tagging of client commands and server responses to these commands, which allows the server to respond to commands in a random order.

Each command must be preceded by some identifier - a tag, which will then be used by the server when generating a response to this command. This allows the “conversation” of the client with the server to be absolutely asynchronous - the server is entitled to respond to the client’s commands in any order, as the tags allow you to unambiguously match the answer to the previously given command. Moreover, the server can execute such commands at the same time, speeding up the speed of working with mail, and Yandex.Post can use this. At the same time, this requires a special approach to programming both the client and the server. If you remember the mechanism of sequence numbers in TCP, write yourself a +1 in geek cred :)

To implement the IMAP collector inside Yandex.Mail, we used our asynchronous framework to write any high-performance network services called yplatform. While its source code is closed, although we are considering the possibility of publishing it. Follow the news.

Go to Yandex.Mail, configure the IMAP collector - and you can always find any old email. So what, what Yandex is able to search.