⬆️ ⬇️

Why large messengers do not work with XMPP or Reflections on the fate of the protocol

One of the most common questions asked by our users is: “when will the Agent and ICQ switch to the XMPP protocol, and why has this not happened yet?”.

image

On the one hand, Jabber (so often called the XMPP protocol, although Jabber is just an old version of the XMPP specification) is considered by many to be the best alternative to proprietary protocols, on the other hand, world practice shows that large Internet companies are not in a hurry to work with XMPP, and There are certain reasons for this.



Since this question is not as simple as it may seem at first glance, we decided to answer it in as much detail as possible, including the example of our own product, Mail.Ru Agent. Not for the sake of additional PR, but simply because business arguments are always more interesting than conclusions drawn from abstract examples.



So why not XMPP?



Story

')

The first version of Mail.Ru Agent was released more than 8 years ago, in 2003, when the Internet was completely different in the widest sense of the word - from business models to technical solutions.



At that time, XMPP (or Jabber, as it was then called) was completely new, was not used anywhere, and did not seem to be such an indisputable standard for messaging - suffice it to say that the first attempts at its standardization were made at the end of 2001 , and the current standard RFC 3920 is dated 2004 year.



At the same time, XMPP was not the first and only protocol of its kind. The market for instant messengers had already been largely formed by that time, and they all worked on proprietary protocols. In addition, another open protocol for IM, SIMPLE (SIP for Instant Messaging and Presence Leveraging Extensions), already existed and was even standardized ( RFC 3428 ). So the question of which protocol "wins" remained open at that time - and even now the answer to it is, in general, ambiguous.



Both server and client implementations of XMPP were very “raw” in those years and were only suitable for experimental use - in general, the technology was clearly at the very beginning, there was no evidence that it will develop into something more in time, therefore we had no particular reason to build our product on it.



In addition, initially we were not going to make an instant messenger at all! Mail.Ru Agent was conceived as a small utility that would only notify the user about a new mail in the mailbox. The Mail.Ru Agent of the 2003 sample most of all resembled not a server for messaging, but a POP3 server — with the only difference that it was not the client who polled the server about new letters (pull), but the server itself sent such notifications to the client (push) through a permanently established TCP connection (immediately at the moment of receipt of the letter to the MX server). To solve this problem, a very simple protocol was required, which was implemented.



The utility became popular, and since, as a result, we received thousands of simultaneously connected users on its server, a little later, the idea appeared to implement the exchange of messages between them. Then the file transfer, and so on. Of course, each of the new “features” was most easily implemented on the basis of an existing server and its protocol, rather than developing a new server from scratch. Moreover, some of the scalability of the protocol were laid initially.



Well, and then ... then the Mail.Ru Agent’s audience reached several million users per day, and the number of platforms supported by customers gradually reached seven. So even if we needed to suddenly switch to XMPP, it would take a long time, during which we would have to support two versions of the protocol anyway.



However, objective reasons for such a transition never arose. Our own protocol continued to successfully perform its task and completely satisfied us.



Equipment



The main argument of supporters of the XMPP protocol is its openness and the status of the IETF standard. Indeed, it is difficult to overestimate the huge selection of a wide variety of XMPP clients and servers available on any common platform.



But not everything is so simple. The fact is that the XMPP protocol itself provides only a basic set of required functionality. Without going into details, it is the authorization on the server, the presence-statuses, the management of the list of contacts (roster), as well as the direct exchange of messages. In essence, this is just a transport for abstract “events” occurring between two clients (or a client and a server).



Anything that goes beyond these possibilities is described by extensions to the protocol, which are called the XEP (XMPP Extension Protocols), which is strange for the Russian ear. Fortunately, data structures in XMPP are described by a hierarchical XML language and are easily extensible.



There are quite a few “for all occasions” extensions, but since all of them are optional (that is, neither server nor client are required), in reality it turns out that each client and server supports its own, unique set of extensions, and this This results in such intricate tables:



http://en.wikipedia.org/wiki/Comparison_of_instant_messaging_clients#XMPP_clients



http://en.wikipedia.org/wiki/Comparison_of_XMPP_server_software



In other words, there is almost always a situation when the client cannot use some server functions or vice versa. Some confusion with the official status of extensions aggravates the situation: despite all the diversity of XEPs, not all are accepted as the standard or “candidates” for standardization. Many important extensions have experimental status — that is, they can be changed at any time or rejected altogether.



The absence of unambiguous standards and a client-server “zoo” lead to inevitable compatibility problems. And this is not a mere assertion: when implementing the XMPP client in Mail.Ru Agent, we learned from our own experience that the authors of each server interpret the XEP specifications very differently. This is not surprising, because many of the provisions of the specifications are lax, advisory in nature, and besides, they are periodically reviewed. Here is just one of the typical examples: http://xmpp.org/extensions/xep-0153.html#bizrules-image



In addition, the same functionality can easily be implemented with two different XEPs. For example, you can use XEP-0066 (Out-of-Band Data) for transferring files between two contacts, or you can use technically more sophisticated XEP-0096 (Stream Initiation File Transfer). Thus, the presence of the file transfer function in two different clients does not guarantee at all that the transfer of files between them is actually possible.



All these factors cause a very low rate of development of XMPP-based instant messengers. A typical illustration of such stagnation are, for example, voice calls and video calls (a must have for any modern IM application). Despite the availability of appropriate extensions ( XEP-0166 , XEP-0176 , XEP-0266 , XEP-0299 , etc.) of varying degrees of "formality", XMPP-clients with support for calls on the computers of "ordinary users" practically do not occur (I only two are known, and one of them, Empathy , does not even seem to be ported to Win32).



In short, if you want to create a new IM-system based on the XMPP protocol and are going to strictly follow the letter of the standard, then the capabilities of your server and client software will obviously be limited to the framework of existing XEPs. Of course, no one forbids you to create your own, proprietary extensions - however, the more you have such extensions (and the need for them will appear very quickly), the less “open” and compatible with others your system will be. But that was the whole idea, wasn’t it?



On the other hand, the development of a manual scanner on the basis of its own protocol, created from scratch, completely unties the hands of the developer. There was a need to add a new type of package for the functionality invented by the manager? No problems. Did you need to expand the data structure of an existing package? Easily! There is no need to look back at the standards and adjust the design of your software for them - all you need to do is to remember backward compatibility with previous versions of your customers (instead of an army of third-party applications).



In addition to the purely “ideological” problems that hamper the development of XMPP-based instant messengers, this protocol also has objective technical flaws. And above all - its bulkiness. Of course, human-readable XML is more convenient than binary protocols at the design and debugging stage. But in applications like instant messengers that constantly exchange small portions of data with the server, the use of XML leads to the generation of excess traffic, which can be especially critical on low-bandwidth channels and with charging for the amount of data transferred (for example, in mobile networks). Yes, and routine operations such as parsing XML trees and serializing / deserializing XML data require some kind of additional CPU time.



Business



If you look at the situation globally, then the main thing why a large Internet company messenger may be needed at all is to strengthen user loyalty.



Since the user's contact list is associated with the Mail.Ru account, the wider the social graph of this user and the more friends he has in the contact list, the stronger he is “attached” to our account.



And this is very good for us: in spite of the fact that we earn practically nothing directly on the Agent’s services, it is, at the same time, closely integrated with our web services to which users follow links and notifications in the client interface.



Thus, from a business point of view, we are interested in two things:



In the solution of the first task, the client-server protocol, generally speaking, is not particularly important - the main thing is that the quality of the product is high enough compared to its competitors, and the user gets all the features he needs.



However, if we weigh all the pros and cons, then XMPP is less preferable here than the proprietary protocol - for the reasons I mentioned above (limited functionality and compatibility issues between clients). So, for example, if a user cannot make a video call to his interlocutor from our “native” client, then first of all he will be blamed not by the interlocutor's client, but by the Mail.Ru service (“nothing works for them”). This is one of the reasons why we use our own client / server protocol. It is worth noting that the main parts of the specification of our protocol are publicly available, and we do not hinder the development of third-party clients.



As for another problem - the expansion of the social graph - here, theoretically, XMPP could help us a lot. The fact is that in addition to the client-server connection, XMPP provides for the exchange of data between servers (the so-called Server-to-Server Federation or peering). This mode is necessary so that users can add users from other domains to their roster of contacts and exchange messages with them.



Accordingly, by implementing the XMPP-gate on the Mail.Ru Agent server side, we could significantly increase the contact lists of our users through contacts from other IM services that also support peering. But ... alas! In reality, there is only one such service with a significant audience - Google Talk. But according to our data (as well as independent research data), there are too few users (especially Russian-speaking) that the peering support would bring us at least some visible effect. The rest of the services that support XMPP are either even smaller in terms of the user base or do not support the S2S Federation (such services include, for example, Facebook, Vkontakte, LiveJournal).



Ironically, the only service with which we managed to implement a full-fledged S2S peering is the ICQ messenger. And even here, not XMPP, but SIP / SIMPLE was used as the peering protocol.



In other words, the global openness of XMPP and its decentralization are, by and large, a myth: truly large user audiences are concentrated around XMPP servers, isolated from each other and not having interconnections between themselves. Large market participants are partly afraid of spam (in the case of using the S2S federation it is necessary to struggle not only with internal, but also external “garbage”, which creates additional difficulties and requires serious resources), partly do not rush to “share” their audience, and Difficulties (after all, social networks are more than just an XMPP server; and they need to somehow integrate “foreign” users into their architecture, such as tapes of friends, which was not originally intended for this). Small players (for example, enthusiastic admins, raising XMPP servers in their offices) do not make any weather in the market.



Summary



So, let's summarize.



1. The XMPP protocol was born at an unfortunate time from a historical point of view. At the time of its appearance, the messaging market was already formed, and XMPP could not take the place of the unconditional standard (as it happened at one time, for example, with SMTP or HTTP). Appear XMPP earlier - perhaps now the situation would be completely different.



2. When a user selects an instant messenger, the last thing that interests him is what protocol does this instant messenger use. The three main questions the user asks are “what do my friends use?”, “What does this messenger look like?” And “what can it do?”



3. XMPP is not a panacea or a silver bullet. All instant messengers that are now leading the market (both Russian and global) are based on closed (or super-closed, like Skype) protocols, and so far this does not prevent them from developing. On the other hand, the lack of control over XMPP and the absence of dominant players that set the tone for the market lead to the emergence of many products of a rather controversial quality (first of all, it concerns customers) that are compatible with each other only in terms of the most basic capabilities. So openness is now playing rather against this protocol - no matter how paradoxical it may sound.



4. From a business point of view, switching to XMPP as a client-server protocol does not make sense, and launching S2S Federation is premature due to the reasons already described above. However, we are open to such proposals.



Do we know the password and can we see the landmark?



The position of Mail.Ru about messaging technologies is very simple - all the flags are visiting us!



We strive to ensure that Mail.Ru Agent is compatible with any services where there is an opportunity to exchange messages and there is a noticeable audience, and today Mail.Ru Agent offers 4 interaction options - both with Mail.Ru services and with third-party services.



On the client side:



1. Connecting to the Mail.Ru Agent server using our own protocol under the Mail.Ru account. In this mode, the maximum number of functions is available, including audio and video calls, conferences, sending SMS, geolocation, games, searching for music from My World, etc.



2. Connecting to the ICQ server using the ICQ protocol. All the basic functions of ICQ are available, and we are working on expanding their list. The user needs an ICQ account.



3. Connection to any XMPP-servers via XMPP protocol (including XMPP-servers of such social networks as Odnoklassniki, Vkontakte, Facebook, Google Talk messenger, etc.). Implemented XMPP support in almost all, and some XEP.



On the server side:



4. Peering (interoperability) from ICQ at the level of the Agent and ICQ servers. The user connects the Agent’s client to the Agent’s server under the Mail.Ru account and can add ICQ users to his / her contact list. This mode is most similar to the XMPP S2S Federation. Only messaging and status is supported, but we are working on expanding the functionality.



Shortly speaking...



We believe that the future lies with open technology. It seems to us that, sooner or later, messaging must necessarily come to a certain unified standard, as it happened in its time, for example, with telephony (on any phone you can today dial the number in the international format and talk to the interlocutor - regardless of which phone companies do you both use). Whether XMPP will become such a standard is not so important; the main thing is that unnecessary boundaries should be erased.



In our opinion, ultimately this will only benefit all market participants. For example, many no longer remember that not so long ago, in the early 2000s, there were only two mobile operators in Moscow, between which SMS did not go. The calculation was simple: the lack of peering forced the subscriber to buy the contract of the operator, which is used by his closest relatives or friends. Thus, one attracted subscriber potentially resulted in another two or three, so the effect of each successful marketing campaign was essentially multiplied.



Nevertheless, in the end, the operators nevertheless launched peering - and this led to an instant increase in their income, as the social graph of the SMS service increased 1.5-2 times (and, as a result, people began to communicate much more intensively than earlier). We believe that something similar should happen in messaging - the main thing is that the players launching the peering should be approximately comparable in audience, and instead of cannibalization of audiences, their mutual penetration occurs.



Ilya Naumov,

Project Manager Mail.Ru Agent

Source: https://habr.com/ru/post/133274/



All Articles