Badoo offline notification system

In order for users, while being offline, to learn about events on the site, we have created a special notification system. Its task is to accumulate events for the user and at the right time to report them via available communication channels, such as email and push notifications to smartphones.
How is event storage organized? What events are notifications coming from? At what point are they sent and on what basis? Today we will try to answer all these and other questions.

The article gives a general description of the system architecture with small technical details and will be of interest to those who are only going to or in some way notify their users about everything new that happened during their absence on the site (in the application, service, etc.)

What are the events?

All events of our site are generated by the actions of some users in relation to others. Having found a nice person, you will want to see his profile in more detail, thereby creating the event "Visiting Profile" . If his profile seemed interesting to you, it is likely that you will want to write to him, thus creating the “New message” event. You can add him to your favorites , and also express a desire to meet him . If he also wants to meet with you, then the “Mutual Sympathy” event will happen for each of the users.
')
Evaluating photos of other users, as well as leaving comments , you will create relevant events.

While on the site, the user sees new events in the form of numbers in red circles near each section containing contacts. If it is not on the site now, then it will be able to receive notifications about all updates in sections.

How are events stored?

In order not to send too many notifications to users, we send a notification not to every event, but group them for a certain period, for example, during the first half hour of the user’s absence from the site. After the first dispatch, we again wait and, if there are new events, we send. However, some particularly important notifications can be sent out of turn.

For the aggregation of events, we developed a special DBMS, a demon in C, which is able to:

• Store an array of events for each user. Each event is characterized by the sender's identifier, numeric type (1 - messages, 2 - visitors, etc.) and an arbitrary string of data to which additional information can be passed, for example, in the form of a serialized array.
• Monitor time, periods, timeouts and, upon request, return data that is ready to be sent. As soon as the time comes, the demon gives all the accumulated events for the next user.
• All communication with the DBMS is carried out using the RPC-like protocol, which transmits data packed using Google Protocol Buffers.

Four of these daemons are launched in each of our two data centers to cope with a load that peaks to 25 thousand requests per second without any problems, as well as to minimize downtime if you need to restart / update the daemon, or if . The load between them is distributed according to a simple principle:

user_id% 4 = <daemon number>

Thus, events for each user always “live” in the same database. User data is stored not only in memory, but also stored on disk, so notifications are not lost. If it is necessary to change the sharding and add a few more demons, then the demons are “extinguished”, the data is moved as needed between the repositories and loaded into memory at startup. This is an extremely rare situation - for all the time we needed to do this only once.

What is the delivery channel?

In order to notify users about new events, we use e-mail, as well as iOS and Android push notifications.

It is worth saying a little bit about what push notifications are. Each iOS application has the ability to send you notifications even when it is not running (of course, if you do not prohibit it). Delivery of notifications in this case is carried out through the Apple server, and not directly. When you install an application on your iPhone, Apple tells the developer the identifier of the installed application on your system, which we use as the address for sending messages. The same system exists for Android-smartphones, but delivery is already carried out through the Google server.

If you have several devices on which our application is installed, the notification will be sent to each of them.

How are notifications generated?

For each of the channels we use, we have our own mechanism for generating notifications.

The general logic is that there are notifications containing information about only one type of event (2 visitors, 5 messages, etc.), and there are group notifications (2 visitors and 1 message; 2 messages and 1 mutual sympathy, etc. ). For emails there are several separate templates for new messages, visitors, etc. plus a group letter template, in which each event is represented by a separate line.

For push notifications, we compiled texts for each type of event in several variants. Since the notification cannot contain a lot of information, we selected several basic combinations for groups of events (for example, visitors + messages, messages + mutual sympathies), for which we wrote versions of texts.

• An example of a “single” notification in several ways:
  
  • Option 1: You have <number> new messages from girls!
  • Option 2: The girls wrote you <number> new messages!
  
  
• Example of "group" notification:
  
  • You have <number> messages and new people want to meet with you! Find out who ...
  
  

Notifications for iOS are different from Android in that the latter may have a title, text, and image, and the first only text, but of greater length.

How is the system architecture?

On the one hand, during the use of the site, users generate events that are immediately added to the database. On the graph, you can see the ratio of events in 24 hours in one of our two data centers (the exact numbers on the graphs were asked not to publish):



On the other hand, on several servers a php script is continuously running, which constantly polls the database, whether there are notifications, the sending time of which has already arrived. Below you can see a graph of the number of users for whom notifications are ready:



If you need to send a notification to your smartphone, then the record in the database is marked with a special flag and delayed for 10 minutes. If the user's time zone is now night, then we will send a notification with the “Silent” parameter in order not to disturb the user's sleep.

If the user has not visited the site within 10 minutes, then we create and send a digest letter containing information about all the accumulated events. After that, the sent notifications are deleted from the database, and we again expect new events.

Each time a page is requested from a registered user, we send a command to the database to clear all its events. It so happens that the user got on the promo page and, not seeing anything new, left the site. On such pages, where not all the counters of new events are visible, we indicate which specific events should be cleaned.

When the time of the next posting comes, and the user still has the status “online”, we mark this with a special flag and postpone the record in the database for a while, expecting that he will either continue to be active on the site or will be in the status “offline”.

What does online mean?

How to determine that the user is now on the site, recently was or is already "offline"? These concepts are quite subjective and selected empirically.

Users can be active on the site and in our mobile applications. In order to store the time of the last activity, we use another DBMS of our own design - Last Access. This out-of-the-box daemon calculates offline status based on all user activity data: 30 minutes after the last action, the user finally ceases to be "online."

In order to send notifications, the more complex logic of determining online status is used. First, we check Last Access: if it reports that the user status has changed to "offline", then we can safely send, because a lot of time has already passed. Otherwise, the following algorithm is used for the site and applications: if a person uses the site, then 15 minutes should pass to send. If the mobile application is used and less than 15 minutes have passed, then we simply check if there is a connection with the running application, and if not, then send a notification.

The described system conveniently and unobtrusively informs users about what's new for them on the site. We are constantly refining and improving it to make it really useful and necessary for our many users.

Alexander Treg Treger, developer.