Ask a question to Mail.Ru Group cloud service developers

Thanks to Reddit, the Ask Me Anything (AMA) format is becoming popular - when a team of specialists responsible for some big, popular project invites everyone to ask them any questions about their work and what the service is under the hood like is arranged. The first format AMA on Habré was used by the developers of Badoo. We also decided to try. We hope that other companies will follow our example and begin to let readers into their internal cuisines.

We decided to start AMA with the Mail.Ru Cloud team. This is a young, actively developing project, about which we are always asked many questions. In addition, since its launch (since August 2013), the Cloud has grown into a large family of projects.

Historically, the Cloud began as a B2C product with web, desktop and mobile versions. However, B2B is also a very promising market. Therefore, we have a Mail.Ru platform for business , combining all the B2B services of Mail.Ru Group, including the Cloud for archives (Icebox), the Cloud for workgroups (Teambox), and Hot storage (Hotbox).

Who and how exactly provides the livelihoods and prosperity of this fun family, how data integrity is achieved, due to which we achieve high download speeds and low latency - you will learn all this from the article, and then you can ask any questions you are interested in.

First, a little history

When we started writing Cloud, we took as a basis a single code base that was used in the Mail, in My World, and in content projects. This code base we call Mpop. It is a bunch of Perl libraries. And several years ago, all Mail.Ru Group portal projects lived on it. When in some of the projects something was refactored in Mpop, it affected the rest. Gradually, each project moved to its version of Mpop, and then we started to leave it. Mail, for example, partially switched to Go. We also initially lived on Mpop, but quietly copied everything to our super-fast Perl server. We have a fully asynchronous architecture, we use AnyEvent. And if you know Perl, then come to us, we have a lot of fun and interesting!

How things are arranged

Structurally, Mail.Ru Cloud consists of several commands:

• The team that writes the server part and builds its interaction with the repositories and mobile clients.
• API development team, a layer between servers and the web. The API is used by both Icebox and Teambox.
• Hotbox development team. It develops both the administrative interface for working with bakery and keys in the Hotbox, as well as the S3 API itself.
• Development team of UI (user interface) and layout. Responsible for the beauty of the interface, its stability, speed and functionality.
• A desktop client development team that makes a cloud client for Windows, Linux, macOS platforms. Also this team is working on a Screenshot and various features for it.
• Two mobile development teams that write client applications for iOS and Android that directly interact with the server side.
• A team of admins that supports all this, not closing eyes for days and nights.
• We also have a testing team and an autotesting team that writes autotests for the API.

Under the hood of the web and desktop versions of Clouds

The Mail.Ru Cloud backend is written in ANSI C, Perl, Lua and some C ++. Perl solves API related tasks. For example, editing documents. By the way, we once told on Habré how we show video in the Cloud , the server part there is written in Perl + Lua.

From the very beginning, when the task to create the Cloud was set, one of the main requirements was the absence of long searches on the database, no seek on the disk and storage on the principle of append only. At the time of launch, we did not find a suitable database for tree data, so we created our own to represent the file system.

Here are some numbers:

• Our users download about 100 terabytes of data per day.
• We have over a thousand disk storage servers - more than 100 petabytes of data, they contain more than 9 billion files. You need to optimally bypass and verify them and be sure that the user content is stored securely and unchanged.
• The number of monthly active users is over 18 million. Every day many hundreds of thousands of unique users visit the web.
• We change an average of ten disks per week, we re-roll the data on them.
• The network capacity is up to 200 Gbit / s (if necessary, we can increase it several times). The download to us now is about 25 Gbit / s.

Cloud was originally written with a focus on desktop and mobile clients. Then it became clear that the web is a zhzhot, you need to redo it. The cloud was completely rewritten asynchronously (in general, our entire codebase is asynchronous, from ANSI C to AnyEvent in Perl), it has become much better and faster. For example, now the web version of Cloud lives on the same dozen of servers as three years ago, despite the fact that the number of users has grown significantly. Only then they lay in the shelf for resources without any stock. Our other new products, for example Hotbox, we also write in Perl. So it goes.

Under the hood of mobile Clouds

We continue to use Objective-C to write the iOS Cloud application. Swift do not touch for several reasons:

1. We still support iOS 7. Yes, Swift is already supported on this version of the operating system, but working with Swift dependencies is very difficult.
2. Swift is still unstable, and we don’t want to repeat the heroic feats of adapting new versions of the language, well described by the Firefox development team.
3. Using Swift at the moment significantly increases the size of the application, and we are proud to weigh only about 20 MB.
4. We actively use dynamic properties of the Objective-C language, and there are no analogues in Swift.

As examples of using Objective-C dynamism, we’ll give the following:

1. Writing unit tests for memory leaks. This is possible thanks to our POSAllocationTracker library. The ability to use in unit tests favorably distinguishes it from the Facebook counterpart in the person of FBAllocationTracker .
2. Automatic detection of access to an object from the wrong stream. Implemented in a POSRx POSSchedulableObject . By the way, in it you can look at the combat examples of unit tests for memory leaks.

Other architectural features of an iOS application:

• Use POSInputStreamLibrary to upload video files to the Cloud. The graph below shows two types of video downloads. Orange pillars - downloads, the contents of which are obtained using the Photos.framework API. Blue pillars - the content is obtained using ALAssetsLibrary and wrapped in a POSInputStreamLibrary. This makes loading heavy video files extremely fast, as described in our long-time article .

• Using disaster recovery tests . The mechanism of random network error generation is built into our network library. Thus, in the debug build, the application will necessarily arrive from the service of five hundred and four hundred. Due to this, it is almost impossible not to test every new feature for these regional cases. Implementation is available in visual form in the POSHTTPGateway object from the POSRx library.

The main architectural task for an iOS application today is to get rid of application hangs when working with large clouds. As a tool for solving this problem, we see the Schedulable Architecture pattern, which we wrote about not so long ago on Habré. In order to immediately see the results and be sure that we are moving in the right direction as it is implemented, we have done monitoring of the suspensions. Now we already have a schedule of the most problematic places in the form of a list of classes in which hangs occur most often. It looks like this:

Thanks to the plugin for HockeyApp , we have complete information about each freeze - call stack and logs.

The Android Cloud client is completely developed in Java (we have not yet found benefits for ourselves in Kotlin). In principle, there is nothing supermode in the architecture: MVP, the entire network layer in the service, for the network we use okhttp, we store the data in SQLite, with the exception of the gallery. There may be a lot of data, and they must be quickly picked up from the cache. Therefore, we apply samopisny serialization for the gallery. For communication within the application, there is an eventbus from green robot. For effective work in the background on new versions of android we use JobScheduler'y, and for not very new - GcmNetworkManager.

For rolling out features for percentages, A / B testing and some analytics, we use Firebase. To debug the interaction with the server, we use it in the debug Stetho builds from Facebook. The latest version almost switched to vector graphics in the app. I think in a couple of versions we’ll move on to it completely. We write tests on junit, uiautomator and espresso.

Under the hood of B2B-Clouds

B2B Cloud is three different products:

• Teambox is a cloud for workgroups, an analogue of B2C-Cloud, in which you can allocate personal space for each account in the domain, create shared folders for departments and groups of employees and set up access to files.
• Hotbox is a fast-access repository for distributing media and any other frequently downloaded content with an S3-compatible access protocol, distributed storage with multiple copies and a guaranteed SLA of 99.99%.
• Icebox - object cloud storage for archiving and long-term storage of data and a simple web interface. It is designed for storing and accessing rarely used data with S3 protocol support, which can be quickly accessed if necessary.

B2B-Cloud consists of two parts - the frontend, which we have built into our Platform for business, and the backend, which ensures the management of the Cloud and its operation. The part of the backend that is responsible for the operation of the Cloud is written in Perl, the management support is written in Python.

Business platform biz.mail.ru provides access to the management of all B2B-Clouds. The platform itself is a general administrative panel for the Mail.Ru B2B services: Mail for the site, corporate Calendar, Agent, DNS service and B2B Cloud. The platform provides a personal account of administrators and is implemented on the principle of plug-ins - there are common elements (authorization, project and domain management, user list), there are separate sections for connected services - Mail, Teambox, Hotbox, Icebox and others. The Platform has its own API and a mechanism for proxying requests from the backend of the Platform to the backends of connected services. Thus, we have a common infrastructure, the functionality of which is developed by different teams of programmers, and for users, all controls are collected in one administrative panel.

So ask!

By tradition, it is necessary to designate the time when we will answer questions. And let it be two days, not one: today and tomorrow from 12.00 to 19.00 (Moscow time). But in the case of particularly harsh discussions, we, of course, will not limit ourselves to this interval. Ask questions about our software, about our servers, about our teams, about our API, and so on. Let's go!