“Request has matured”: Alexey Fedorov about the new conference on distributed systems

Recently, two events were announced at once about the development of multi-threaded and distributed systems: the Hydra conference (July 11-12) and the SPTDC school (July 8-12). People close to this subject understand that the arrival to Russia of Leslie Lamport , Maurice Hurlihi and Michael Scott is a major event. But there were other questions:

• What to expect from the conference: "academic" or "production"?
• How do school and conference compare? Who is it for both?
• Why do they intersect by dates?
• Will they be useful to those who have not dedicated their entire life to distributed systems?

All this is well known to the person who made Hydra appear: our director Alexei Fedorov ( 23derevo ). He answered all the questions.

Format

- An introductory question for those who are far from distributed systems: what are both events about?
')
- The global task is that around us there are services with large volumes of operations and complex computational tasks that can not be done on one computer. So, there should be several cars. And then there are questions related to how to correctly synchronize their work and what to do in the conditions of not the highest reliability (because the equipment breaks down, and the network falls off).

The more machines, the more points of failure. What to do if different machines produce different results for the same calculations? What to do if for some time the network disappeared and some of the calculations became isolated, then how to combine all this? In general, a million related problems. New solutions - new problems.

In this area there are completely applied areas, and there are more scientific ones - something that has not yet become mainstream. I would like to talk about what is happening in practice and in science, and most importantly - at their interface. That's about it and will be the first Hydra conference.

- I would like to deal with the fact that there is a conference, and there is a summer school. How do they compare? If participants of the school are given a discount on attending the conference, then why do they intersect according to dates, so you cannot visit everything without loss at once?

- The school is a chamber event for 100–150 people, where leading experts from all over the world come and give lectures for five days. And a situation arises when world-class luminaries gather in St. Petersburg for five days, ready to say something. In this case, the decision suggests itself to organize not only a chamber school, but also a larger conference.

It is possible to conduct such a school only in summer, in July, because among these specialists there are acting professors of higher educational institutions, and at other times they are simply not ready: they have students, diplomas, lectures, and so on. School format is five weekdays. It is known that in the summer on the weekend, people love to go somewhere. This means that we cannot do a conference either at the weekend before school or at the weekend after school.

And if you extend it still for a couple of days before or after the weekend, then magically five days of stay of specialists in St. Petersburg turn into nine. And they are not ready for this.

Therefore, the only solution we have found is simply to hold a conference parallel to the school. Yes, it causes some problems. There are people who want to go to school and to the conference, and they will have to miss part of the lectures there or there. The good news is that all this will take place in the neighboring halls, you can run around here and there. And one more good side in the presence of videos, which later you can safely see what I missed.

- When two events are held in parallel, people have the question "which one is more necessary for me, then." What exactly to expect from each, and what are the differences?

- The school is a purely academic event, a classical scientific school for several days. Who was engaged in science and had something to do with graduate school, is what an academic school.



Typically, such academic activities are not well organized due to the lack of event expertise in people who do this. But we are still quite experienced guys, so we can do everything quite competently. I think that from an organizational point of view, SPTDC will be a cut above any scientific or practical school you have ever seen.

School SPTDC is such a format, where each big lecture reads two pairs: “one and a half hours - a break - one and a half hours”. It should be understood that for the first time the participant may not be easy: when this school was held for the first time two years ago, I myself was not used to it, I turned off several times around the middle of a double lecture, and then it was difficult to understand what was happening. But it is very dependent on the lecturer: a good lecturer tells all very interesting hours all three hours.

The Hydra Conference (Hydra) is a more applied event. There will be several luminaries of science who have come to give lectures at the School: from Leslie Lamport , whose works underlie the very theory of multithreaded and distributed systems, to Maurice Herlihy , one of the authors of the famous textbook on concurrency “The Art of Multiprocessor Programming”. But at the conference we will try to talk about how these or other algorithms are implemented in reality, what problems engineers face in practice, who succeeds and fails, why some algorithms are used in practice and others are not. And of course, let's talk about the future development of multi-threaded and distributed systems. That is, we will give such a cutting edge: what world science is talking about now, what the thoughts of leading engineers are spinning around, and how it all fits together.

- Since the conference is more applied, then there will be not only academic luminaries, but also speakers from “production”?

- Definitely. We are trying to look at all the "big": Google, Netflix, Yandex, Odnoklassniki, Facebook. There are specific fun problems. For example, everyone says: “Netflix is ​​distributed systems, almost half of the US traffic is very cool,” and when you start looking at their real reports, articles and publications, there comes a slight disappointment. Because, although this is certainly the world level and there is cutting egde, it is less there than it seems at first glance.

There is an interesting dilemma: you can call the representatives of large well-known companies, and you can call someone already known to us. In reality, the expertise is both there and there. And we are rather trying to pull not “people from very large brands”, but very large specialists, specific people.

For example, there will be Martin Kleppman, who at one time brought rustling on LinkedIn, and also released a good book - perhaps one of the basic books in the field of distributed systems.

“If a person works not in Netflix, but in a company that is simpler, he may think:“ should I go to such a conference, or are there any netflixes talking to each other, but I have nothing to do? ”

“I’ll say this: when I’ve been working at Oracle for a little over three years, I’ve heard the most amazing and interesting things in the kitchen and in the smoking rooms when colleagues gathered to do certain pieces of the Java platform. These could be people from a virtual machine, or from the testing department, or from performance concurrency - for example, Lyoshcha Shipilev and Seryozha Kuksenko.

When they started to discuss something with each other, I usually just listened, opening my mouth. For me, these were amazing and unexpected things that I did not even think about. Naturally, at first I did not understand 90% of what they say. Then it became incomprehensible 80%. And after I did my homework and read a few books, this number was reduced to 70%. I still do not understand a lot of what they are talking about among themselves. But sitting in the corner with a cup of coffee and eavesdropping, I began to understand a little what was going on.

Therefore, when Google, Netflix, LinkedIn, Classmates and Yandex talk among themselves, this does not mean that this is something incomprehensible and uninteresting. On the contrary, we must listen carefully, because this is our future.

Of course, there are people who do not need all this. If you do not want to develop in this topic, you do not need to go to this conference, you will simply lose time there. But if the topic is interesting, but you don’t understand anything in it or while you are just looking at it, then you have to come, because you will not find anything like this anywhere. And I think that not only in Russia, but also in the world. We are trying to make a conference that will be not just a leader on this topic in Russia, but generally number one in the world.

This is not an easy task, but when we have such an amazing chance to gather strong speakers from all over the world, I’m ready to give a lot to do it. Of course, some of those whom we called to the first Hydra will not be able to come. But I will say this: we have never started a new conference with such a powerful composition. Except, perhaps, the very first JPoint six years ago.

- I would like to develop the words “this is our future”: the topic will later affect those who are not thinking about it today?

- Yes, I am sure of that. Therefore, it seems to me very correct to start discussing it as soon as possible. Say, the theory of multi-threading appeared quite a long time ago (in the 70s, works were already in full swing), but for a long time they were a lot of narrow specialists, until the first user dual-core computer appeared at the beginning of the two thousandth. And now we all have multi-core servers, laptops and even phones, and this is the mainstream. It took about 10 years for it to become widespread, for people to understand that this discourse is not a matter for a narrow circle of specialists.

And about the same thing we are seeing with distributed systems. Because basic decisions like load sharing, fault tolerance, and the like have been made for quite some time, but very few people know what a distributed consensus or Paxos is, for example.

One of the most important tasks that I set for this event is to immerse engineers in this discussion more and more. It should be understood that at the conferences not only some topics and decisions are discussed, but also a thesaurus arises - a single conceptual apparatus.

I see it as my task to create a platform where everyone can discuss all this, share experiences and opinions. So that we have a common understanding with you, what one algorithm does, what the other does, which one is better under what conditions, how they are related to each other, and so on.

A very interesting thing associated with the same multithreading. When our friends from Oracle (first of all, Lyosha Shipilev and Sergey Kuksenko) began to actively talk about performance and, in particular, about multithreading, literally two or three years later, they started asking questions at company interviews, people started to discuss this in smoking rooms. That is the thing that was the lot of narrow specialists, suddenly became mainstream.

And this is very correct. It seems to me that we helped these guys popularize all these problems, really important, useful and interesting. If no one ever thought about how a Java server handles requests in parallel, now people at least at some level have an understanding of how it all works. And that's great.

The task I see now is to do about the same with distributed systems. So that everyone understands what it is, where their legs grow from, what tasks and problems there are, so that it also becomes mainstream.

Companies have a huge demand for people who understand this, and there are few such people. The more we create around this content and the opportunity to learn it, the more we give people the opportunity to ask questions that are in the air, the more chances we have to somehow move in this direction.

Prehistory

- The conference is held for the first time, but the school is not the first time. How did all this originate and develop?

- This is an interesting story. Two years ago, in May 2017, we sat in Kiev with Nikita Koval ( ndkoval ), a specialist in multithreading. And he told me that in St. Petersburg there will be “Summer School in practice and theory of concurrent computing” .

The topic of multi-threaded programming in the last three years of my engineering career was fantastically interesting. And then it turned out that in the summer very-very famous people come to St. Petersburg, the same Maurice Herlihy and Nir Shavit, according to the textbook of which I studied. And many of my friends had to do with it - for example, Roma Elizarov ( elizarov ). I realized that I just can not miss such an event.

When it became clear that the school program in 2017 would be cool, the thought arose that the lectures should be recorded on video. We at JUG.ru Group had a complete understanding of how such lectures should be recorded. And we fit into the SPTCC like the guys who made the video for the school. As a result, all the school lectures are on our YouTube channel.

I began to communicate with Peter Kuznetsov, who was the main ideologue and organizer of this school, and with Vitaly Aksyonov, who helped organizationally do all this in St. Petersburg. I realized that it was fantastically healthy and interesting, and probably very bad that only 100 participants could touch the beautiful.

When Peter thought about the need to do school again (in 2018 there was no time and effort, so he decided to do it in 2019), it became clear that we could help him by simply removing all organizational things from him. That's what's happening now, Peter deals with content, and we do everything else. And this seems to be the right pattern: Peter’s probably a more interesting program than “where and when everyone will have lunch.” And we are well able to work with halls, platforms and so on.

This time, instead of SPTCC, the school is called SPTDC, not “concurrent computing”, but “distributed computing”. Accordingly, this is about the difference: the last time at school they did not talk about distributed systems, and on this one we will actively talk about them.

- Since the school is not held for the first time, it is already possible to draw some conclusions from the past. What happened the previous time?

- When the first school was made two years ago, it was expected that there would be an academic event, primarily interesting to the students. And students from all over the world, because the school is only in English, and it was thought that a significant number of foreign students would arrive.

In fact, it turned out that a lot of engineers came from large Russian companies like Yandex. There was Andrei Pangin ( apangin ) from Odnoklassniki, there were guys from JetBrains who are actively engaged in this topic. In general, there were many familiar faces from our partner companies. I am absolutely not surprised, I understand perfectly why they came there.

Actually, the organizers had expectations that there would be academic people at the School, and suddenly people came from the industry, and then it became clear to me that there was a demand in the industry.

If an event, which hardly ever progressed anywhere, gathered an audience of adults at the first click of a finger, then there really is an interest. It seems to me that a request for this topic has matured.


^{Maurice Herlihi at the JUG.ru meeting}

- In addition to the school, in 2017, Maurice Hurlihi spoke at a JUG.ru meeting in St. Petersburg, talking about transactional memory, and this is a little closer to the conference format. Who then came - the same people who usually come to JUG.ru mitapas, or a different audience?

“It was interesting because we understood that Maurice would have a general report, not Java-specific, and made a slightly wider announcement than we usually do with JUG-news subscribers.

A lot of people from communities that I know are not about Java at all: from a .NET party, from a JavaScript party. Because the topic of transactional memory does not relate to a specific technology development. When a world-class expert arrives to tell about the transactional memory, to miss the opportunity of such a person to listen, to ask him questions is just a crime. It just makes a powerful impression when a person, according to the book of which you study, comes to you and tells you something. Just fantastic.

- And what was the feedback on the results? Has the approach for the people from the industry turned out to be too academic and incomprehensible?

- Reviews of the report Hirli were good. People wrote that he very simply and clearly states what was not expected from an academic professor. But we must understand that we have not just called him in, this is a world-renowned expert with extensive experience in speeches and a background from a pile of books and articles. And, probably, he became famous in many respects due to his ability to convey material to people. Therefore, this is not surprising.

He has normal, understandable English, and he, of course, knows a lot about what he is talking about. That is, you can ask absolutely any questions. Most people complained that we gave Maurice too little time for a report: two hours for such a little, we need at least two more. Well, what we had in two hours, we did.

Motivation

- Usually JUG.ru Group is engaged in large-scale events, and this topic looks more specialized. Why was it decided to take on her? Is there a willingness to hold a smaller event, or can many viewers get together on such a topic?

- Indeed, when you do a certain event and ask a certain level of discussion, the question always arises as to how massive this discussion is. What number of people - ten, a hundred or a thousand - is interesting. There is a trade-off between mass and depth. This is a perfectly normal question, and everyone decides it differently.

In this case, I want to make an event “for myself”. I still understand something in multi-threading (I gave lectures on this topic at conferences, told students something several times), but in distributed systems I am a layman: I read some articles and saw several lectures, but not even a single full-fledged book read.

We have a program committee consisting of experts in this field and able to assess the accuracy of the reports. For my part, I am trying to make this event the kind that I myself would like to attend to with my lack of expertise. Will I get interested in the mass public, I do not know. Probably at this stage it is not the main task of this event. Now it is more important to form the strongest program in a short time.

, « », «, ». - , . .

. . Java- , . , , . , , , , , , — .

, . , , , .

70-80% . , ( , ) ( ). . But not at this time.

, -. . , , , , -. , - . JavaScript, Java .NET , , , , , — , - .

: 70–80% . ? , . ? : , .

SPTDC , Hydra , .