Why write?

Once upon a time in one distant country was America Online . And she had an amazing private Internet behind the fence, where instead of URLs there were "keywords": something between a web page address and a purchased keyword in advertising. Companies were fighting for interesting keywords, as they are now fighting for domains, and the ads looked like this: "visit us on the worldwide web at www.example.com, or type AOL Keyword: 'banking'".

History tends to repeat. Now the role of America Online is played by the main messengers: they are all behind the fences, incompatible with each other, they all invent their keywords, they want to grab the user and never let go. Companies are not interested in openness: larger players do not want to share users with smaller ones, much less become open. As a result, it is impossible to send a message even from WhatsApp to Facebook Messenger, despite the fact that both belong to the same company. And users value reliability and convenience above abstract openness, although many are annoyed that some of the friends, for example, in Telegram, some in WhatsApp, and parents in Skype.

But the role of the open Internet, unfortunately, today no one plays. I want to change the situation. If XMPP failed, maybe someone else can? And then the story about Tinode .

What is Tinode

Tinode is a fully open source messenger on Github. All client applications ( ReactJS and Android ) are licensed under Apache 2.0, in order to simplify the creation of commercial applications based on Tinode, a server under GPL 3.0. The goal of the project is to create a federated instant messenger that is simple and convenient for both users and operators. Put - and everything works as MySQL or Nginx. In the long term, the goal of the project is to create an open alternative to existing proprietary messengers, to repeat with respect to messengers what Android has done with regard to mobile phone operating systems.

What does he do

Support for multiple devices.

Everyone has a smartphone, sometimes not one, plus it is often convenient to use a web application from the main computer. Therefore, support for multiple devices was one of the main requirements for the project, which determined the main architectural solutions. If the user is authorized with a new device, then you do not want him to start from scratch like in WeChat. This means that both the address book and messages should be stored on the server, which was implemented.

It is obvious that the storage of user information on the server is not suitable for everyone, since it creates risks of unwanted access: the more copies of data are stored in different places, the higher the likelihood that something will go wrong. For this, the possibility of ephemeral messages and messages that are deleted from the server after delivery to the client is provided. Technically, there is the possibility not to keep contacts on the server permanently - the client sends them to the server at the time of connection (login), then they are deleted after logout. However, the authors considered this impractical difficult and did not begin to do.

Online status

Broadcasting online / offline user status in instant messengers is taken for granted, however, this is a very difficult feature to implement. It needs to be "just working", predictably and reliably. Reliability eliminated client status generation, as implemented in some XMPP applications. In the case of Tinode, the server generates online status and sends it to the address book, which, again, requires the storage of contacts on the server and their synchronization with client applications.

Protocol simplicity

The protocol wanted to be made so that the learning curve was flat - no need to know everything to get started. The specification turned out very compact : 10 client requests, 5 server responses. For example, compared to 200+ pages only core XMPP , not counting extensions, this is almost a note on a napkin.

The presentation of the data is separate from the network protocol. The protocol only requires a specific data structure, but does not require that they be transmitted over the network in any particular way. Now the server supports JSON via websocket and long polling, with and without TLS, plus gRPC over TCP. GRPC support was implemented by one developer in two weeks, including writing a text client on Python. Adding support for other data formats and protocols, such as MessagePack or Noise , is unlikely to take much more.

Extensibility

On the one hand, I want everything to work right away, for example, so that the main functionality is comparable to WhatsApp and Telegram right out of the box. On the other hand, the needs of people are different and you need to be able to expand the functionality. Finding a balance is similar to the choice between a monolithic architecture and microservices: it is undesirable to have an immutable monolith, and, similarly, it is not good to get a microservice zoo, whose management becomes a separate task.

It was decided to divide the functionality into three parts - main, network and auxiliary. The main thing is what allows Tinode to perform the main function - to forward messages. Network - the functionality of interaction in servers, as the format of the transmitted data and the network protocol. The secondary is something that solves a local problem, for example, supporting a specific database as a backend or some kind of authorization method, but does not affect other servers or user applications. The main functionality is implemented in the main code. Network functionality is highlighted, but is also stored in the main repository in order to avoid creating incompatible servers whenever possible. Auxiliary is implemented in the form of plug-ins - Go compiled interfaces (support for different databases, different authorizers, push notifications, validators by email or phone, support for captcha, etc.) and gRPC endpoints (chatbot and search interface).

Other

• Possibility, but not the requirement of linking the account to the phone or email or anything else.
• IDs of users who are difficult to guess, and, accordingly, difficult to send spam.
• Tags that allow you to implement a search for people as in WeChat (and, like WeChat, embed a dating service in the messenger) or divide the organization into departments as in Slack.
• The ability to connect users without registration, necessary, for example, for organizing support services via chat.
• Interface and chatbots connection example.
• Plans to create channels in Telegram.

Why go?

Server for the messenger is essentially a router: it receives a message from one channel, somehow processes it, then transfers it to another channel or channels. Go (like Erlang, but that's another story) is ideal for creating such a functional because contains the primitives goroutine and chan, which make organizing streams and exchanging data between them efficient and simple.

Of course, the router can be written in C / C ++, and in Java. However, other things being equal, the code is likely to be more complex and will require more effort to avoid deadlocks.

And then what?

Federation

One of the main tasks for Tinode for the coming year is to create a platform for the federation. So that anyone can start their Tinode server, which could exchange messages with any other server, exactly as it is possible with an email. Already now server clustering is possible. Network communication between the server and clients goes via TLS websocket, which for an external observer is hardly distinguishable from simple HTTPS traffic.

A public DNS is likely to be used, at least initially. However, in the future, the search for chat servers will also be carried out, as is done in Bittorrent - using DHT , a distributed hash table.

I also want to avoid the problems for which XMPP is often criticized. For example, XMPP servers are very talkative. Up to half of messages are duplicate when the XMPP client sends online notifications individually to each contact in the address book.

Reputation and Distributed Decision Making

Messaging systems are most similar to email. As you know, much of the email is spam. I do not want to repeat the mistakes of others, so the mechanisms that limit spam should be immediately integrated into the system. The problem has not been completely solved yet, but there is a general direction:

• Cryptographic identification of the sending server.
  Initially, SMTP did not assume any identification of the sender at all. Do not step on this rake again. Each server wishing to establish contact will be represented by a cryptographic certificate. "Well, yes," you will say, "I’ll gather 100,500 certificates now and every time I’ll introduce myself as a new clean server." And you will be right. Therefore, the next item.
• Distributed reputation accounting.
  When a new, unknown sending server is knocked to us, we will make a request to known servers with a request to provide a rating for the new server. And depending on the answer, let's set, for example, the speed with which the new server can send us messages.

If you look at the distributed decision-making system and taking into account the reputation from a bird's-eye view, you will notice a similarity with the blockchain. Perhaps blockchain (but not cryptocurrency) can be used as the basis for building a distributed reputation system, although it is not yet clear how.

Encryption

Well, what about these days without encrypting messages? Chats between two people are likely to be encrypted by OTR . With group chats is not yet clear. All known group chat encryption schemes either have significant drawbacks or are heavy and difficult to implement. Also, it’s not obvious how important the encryption of group chats is: “If two people know the secret, this is no longer a secret, and if three, this is already a bazaar.”

What do you think about this?