What we will tell on Highload ++ 2017

2,700 participants from different regions of Russia and the world, 150 speakers, more than ten halls, 2 days at the largest Skolkovo venue - this is all about the Highload ++ conference, at which we may meet. We have a big landing in different sections, there will be not only traditional performances, but also master classes.

Made a schedule of speeches from Mail.Ru Group for you - almost 18 speakers! We will talk about detection and recognition technologies, database creation and migration without downtime, writing a streaming video platform, protection from DDoS, a new Tarantool and much more.

Some lectures and master classes overlap with each other, but the recordings of reports are traditionally laid out after the conference. Or connect to the broadcast, it will be too.

November 7

Eduard Tiantov , head of the antispam group and the machine vision group Mail.Ru Group
Face Recognition: From Scratch To Hatch , 3:00 pm, Nairobi + Casablanca Hall

In the mobile Cloud Mail.Ru, face detection and facial recognition technology is already used for clustering photos. We also use recognition technologies in the company's internal services. Based on real experience in our products, Edward will tell:

• how to teach a neural network to find faces in a photo and distinguish them;
• what are the interesting aspects of learning and tricks;
• what you need to know about the optimization of computing to work with tens of millions of queries per day;
• seen problems of the model and our solutions.

Vasily Soshnikov , Senior Developer in Tarantool team
“How and why to create a NginX module - theory, practice, profit. Part 2 " , 15:00, hall" Delhi + Calcutta "

A year ago, Vasily delivered a report “How and why to create an NginX module - theory, practice, profit”. At that time, it was not possible to talk about all the features of NginX ... and this time it will not work - the topic is too big! Instead of all the possibilities, we will focus on issues that we did not have time to talk about last year:

• how and why to create upstream-modules;
• how to properly convert content passing through modules;
• NginX shared memory - why and what to do with it.

It will also discuss the problems and solutions in the project tracking user activity, sessions and traffic accounting in Live Stream video and VOD (on the scale of CDN and three data centers), which was not without several NginX modules.

Anton Reznikov , Mail.Ru Cloud Development Team Leader
Igor Munkin , Cloud Mail.Ru project developer
"How to write a service whose support does not turn into hell" , 17:00, Congress Hall

Joint presentation on the development of service in the context of operation, incident resolution and investigation. In the Mail.Ru Cloud project, where we have a thousand and a half machines, a multimillion audience and complex business logic, we found answers to the following questions:

1. How to write logs so that the investigation of incidents does not turn into a search for a needle in a haystack?
2. How to collect metrics, what happened to the project?
3. How to add metrics to monitoring in order not to learn about problems with your project from users?

Knowledge in this area can help any high-loaded service.

Dmitry Soloviev , Lead Developer of the Mail.Ru Group Recommendations Department
“Identification of attributes and visual search in UGC-photos of clothes” , 5:00 pm, hall “Nairobi + Casablanca”

One of the most high-tech areas in e-commerce are recommendations. Algorithms based on user behavior or information extracted from texts showed themselves well, but in the field of visual search and visual recommendations, other solutions should be sought.

Dmitry Solovyov will tell you what we encountered in the process of creating a recommender clothing system from photographs from users:

• how to use convolutional neural networks in solving the problem of searching for images;
• how you can quickly build your search for images using pre-trained neural networks;
• how to further train the network to highlight the attributes of clothing helps to improve the quality of recommendations.

The lecture considers the following issues of practical use of neural networks:

• clothing attribute classifications;
• object detection;
• search for similar clothes;
• recommendations of clothing on photographs.

Dmitry Yegorov , head of the database of VKontakte
“How to rewrite VKontakte personal message database from scratch and migrate to it without downtime” , 18:00, Congress Hall

For many years, the private messages section on VKontakte (it has trillions of messages and around a petabyte of data) worked on the same self-written database, which gradually became morally obsolete and required a huge number of servers.

For 10 months, a new message storage system was developed from scratch, removing many architectural limitations. It has much more performance, saving millions of dollars.

At the beginning of 2017, the amount of iron used decreased twice for the user's personal messages section (the old engine required a double increase). Also, a significant part of the logic was transferred from the php-backend to the engine. In the main hall of HighLoad ++, Dmitry Egorov will talk about the architecture of both personal message engines, optimizations made by a small team in a short time, and also highlight in detail the most difficult part - migration without downtime.

November 8

Yuri Nevinitsin, the platform team at Odnoklassniki
"As we forced Druid to work in Odnoklassniki", 10:00, hall "Mumbai"

Once, the MSSQL-based statistics system for 50 terabytes at Odnoklassniki became like this:

• slow (the average response rate was several times less than required);
• unstable (at peak times, statistics lagged behind to half an hour);
• expensive (Microsoft licensing policy has changed, licenses could be worth millions of dollars).

All these problems were solved by introducing Druid: the average response speed increased by 20 times, the statistics ceased to lag behind, instead of an expensive license, open-source appeared. Druid Odnoklassniki is used to store and process operational indicators for the last two years. At the moment, the database processes tens of billions of records per day (hundreds of gigabytes in size).

Yuri Nevinitsin will tell why among all the decisions the choice fell on Druid. Spoiler: high performance, horizontal scalability, fault tolerance.

Alexander Tobol , video platform developer and Odnoklassniki news feed platform
“We write our protocol over UDP or streaming video platform from scratch to a million online” , 11:00, Delhi + Kolkata Hall

Service Video in Odnoklassniki ranks second in Runet by views: over 400 million video views daily. Users not only watch videos, but also generate content themselves - including using the OK Live streaming application.

Odnoklassniki solved the following technical problems of the streams:

• guarantee of delay between the streamer and viewers on a dynamically changing mobile channel;
• continuous streaming even with the loss of equipment;
• Distribution server load during popular broadcasts.

Alexander Tobol will share his experience in building a scalable, fault-tolerant video streaming service, talk about architecture that can withstand thousands of streams online and millions of viewers. On the practical side, the report will answer the following questions:

• how to optimize the broadcast delay to a time comparable to p2p video conferencing;
• how to use hls, dash, rtmp, webrtc protocols;
• How to configure codecs on the client and transcoder to minimize latency;
• TCP delay guarantee issues;
• your UDP protocol: measure MTU, pacer, packet loss encryption, fast retransmite.

The result of work on streaming in Odnoklassniki was the launch of the world's first Android application that can stream FullHD (1080p) in mobile networks.

Denis Linnik , backend-developer in Mail.Ru Group Media Projects
“Stored procedures in NoSQL DBMS on the example of Tarantool ”, 11:00, hall “Nairobi + Casablanca”

Denis Linnik on the example of Tarantool will consider the advantages and disadvantages of the approach of writing stored procedures to increase the speed of processing and receiving data. If we consider Tarantool as a full-fledged application server, then applications running in it can be considered as microservices. The stored procedures on Lua (and also on C) in Tarantool allow you to implement complex logic from large data aggregation to OAuth2 authorization. The report details the features and pitfalls of writing stored procedures, as well as the issue of profiling and code optimization.

Alexander Khristoforov , leading Odnoklassniki developer (platform team)
“Cheaper, more reliable, easier. Storage of petabytes of video and photos in OK " , 12:00, hall" Delhi + Calcutta "

Odnoklassniki store dozens of petabytes of video data and provide access to them at speeds of hundreds of Gb / s - processes require an infrastructure consisting of many thousands of disks and hundreds of servers. Previously, to ensure data reliability and fault tolerance, three replicas of data were stored — one in each data center. The explosive growth of downloadable video content led to the decision to develop a new low-cost and reliable data storage system.

Alexander Khristoforov will tell how it was possible to reduce the redundancy of data storage from 3 to 2.1, and at the same time achieve higher reliability and availability of the system. The report describes a simple and safe way to replace thousands of disks.

Oleg Anastasyev , leading Odnoklassniki developer (platform team)
"One-cloud - data center management system in Odnoklassniki" , 15:00, Kaliningrad Hall

Classmates are more than 8,000 iron servers in several data centers. Each of the machines was specialized for a specific task - to ensure failure isolation and automated infrastructure management. However, it became clear that the introduction of a new control system would allow more efficiently to load equipment, facilitate access management, automate (re) distribution of computing resources, speed up the launch of new services, speed up reactions to large-scale accidents.

Abstracts of Oleg Anastasyev’s speech on the cloud management system of DC:

• basic principles, processes and details of the implementation of the Odnoklassniki cloud;
• ensuring the fault tolerance of the cloud itself and the tasks it performs;
• approach to the isolation of tasks and increase the density of use of technology;
• real exploitation of the cloud and tasks in it.

Andrey Domas , Odnoklassniki Lead System Administrator
"Balancing" at the speed of the wires. " Without ASIC, without restrictions. NFWare Solutions for Classmates , 16:00, Mumbai Hall

The qualitative and quantitative growth of DDoS attacks in recent years makes them one of the top-priority problems. Depending on the attack vector, one or another part of the infrastructure becomes a bottleneck. In particular, with a SYN-flood, the first blow falls on the traffic balancing system. From its performance depends on success in resisting attack.

Andrei will share his experience on how Classmates and NFWare implemented a high-performance load balancer software. He will talk about how the solution provided bandwidth of up to tens of millions of CPS, higher than the maximum capabilities of even many hardware counterparts. And how it works in the network of one of the most heavily loaded projects of the Runet.

In addition, the following topics will be covered:

• increase in speed due to multi-core architecture;
• selecting from scratch the developed stack instead of the usual Linux stack;
• ensuring the fault-tolerance of the solution while maintaining the highest performance;
• increased protection against DDoS using specially developed algorithms.

NFWare Technical Director Vyacheslav Morozov takes part in the report.

Arthur Udalov , maintainer of the library of React-components in Mail.Ru Mail
“How to develop a library of components without breaking it”, 17:00, hall “Beijing + Shanghai”

In the Mail.Ru Mail team, the UI components were separated into a separate library and used in several projects, but they encountered a problem: each developer, changing the library to fit his needs, necessarily broke what worked for others. We managed to cope with the problem - in this report you will learn exactly how.

Arthur Udalov will talk about how to effectively develop and maintain the component library for team work in a real project using various tools: Storybook, BackstopJS, Jest, Webdriver.io, TypeScript.

Master Classes

November 7

Roman Tsisyk , developer at Tarantool team
"Tarantool. How, why and why? ” , 11:00, Hall A1.3

Roman Tsisyk will tell you what the secret of Tarantool's performance is as a solution, and when it is, and when it is not worth using it. Theses of the speech:

• hot and cold data;
• DBMS and cache;
• SQL and NoSQL;
• monolith and microservices;
• CRUD and CQRS;
• Tarantool and Redis.

Konstantin Osipov , developer and founder of the Tarantool project
"The internal structure of the Vinyl storage engine in Tarantool" , 13:00, Hall A1.3

Vinyl - the implementation of the diskstore in Tarantool - released in production in Tarantool 1.7.5. Konstantin Osipov will talk about tuning and tuning Vinyl:

• how Level0 works in LSM;
• what data structures are used from in-memory technology;
• secondary key support effectiveness in LSM Vinyl;
• management of LSM tree layers on a disk to minimize the amount of used disk space and reduce the amount of garbage.

At 15:00 Konstantin will continue to talk about the database with the theme "Synchronization of data from PgSQL to Tarantool" .

The task to have quick access to actual data that is in the database was solved using the Tarantool application server. In the process of the application, we get the actual data from PgSQL and update it immediately after the change in PgSQL, which gives us the opportunity to always have the current data in the cache. As a result, we have a smart cache that monitors its state and stores the actual data.

Nikita Pettik , developer at Tarantool team
“SQL in Tarantool: Principles of the Query Optimizer”, 17:00, Hall A1.3

The SQL query translator built into Tarantool allows you to automatically create a quasi-optimal query execution plan for a given query, comparable in performance to the optimal one. This is done using a set of optimizing transformations.

Nikita Pettik will tell you exactly which optimizations are present in the SQL translator, what information will help the optimizer make the right decision when building a query plan and how to store and collect this data.

November 8

Kirill Yukhin , developer at the Tarantool team
“SQL in Tarantool: Status and Plans” , 14:00, Hall A1.3

Kirill Yukhin will tell you what we took as a basis and what problems we encountered when adding support for the SQL language in Tarantool. It will also show what has been achieved with SQL at the moment.

Aleksey Khatskevich , developer at Tarantool team
“SQL in Tarantool: learning to read EXPLAIN” , 16:00, Hall A1.3

SQL queries can be executed in several different ways, for example:

• the performance of the query plan accepted for execution will be greatly affected by the order of the products of join operations (if any);
• A suitable covering index (covering index) allows you to avoid a full table scan.

EXPLAIN is a SQL command that allows you to output a query plan. By learning to understand what the output of this command means, you can greatly enhance the skills of writing effective SQL queries.

Alexey Khatskevich will tell you what the EXPLAIN output in Tarantool is, how to read it correctly, and how to use it.

The conference will be organized live broadcast (fee), but it does not get master classes. The main hall will broadcast in open access - here you can watch two of our speeches for free: “How to write a service whose support does not turn into hell” from the Mail.Ru Cloud project and “How to rewrite VKontakte personal message database from scratch and migrate to it without downtime. "