“Half of the scientific work on Concurrency is complete nonsense!” - an interview with Roman Elizarov from Devexperts
Good afternoon, this is “No slides”. I was visited by Roman Elizarov, aka elizarov , Java Champion, an expert on Java and multithreading (and more recently on financial mathematics), speaker of numerous conferences, chairman of the jury of the North-Eastern European Region ACM-ICPC, the most prestigious Olympiad in the world in programming, lecturer in ITMO and, finally, VP in technology at Devexperts. In general, the "man and the ship".
In the conversation we covered the following topics:
what financial mathematics is and how to learn it;
how software is designed for the financial industry;
how a multithreading research laboratory appeared at Devexperts;
where Concurrency is developing, and what will be in vogue in the near future;
as the world programming contest came to Russia.
')
Text version - under the cut.
What is financial mathematics, and how to learn it
- Before the beginning of the interview you said that you started to do financial mathematics.How and why did this topic interest you?
- I came across this topic at work - we write a lot of software for the financial industry. And sometimes clients come and ask: “Do you have this? Or do you have this one? ” And at the same time various clever words are spoken, half of which we do not know. That's how I got interested.
At first I tried to learn myself from textbooks - nothing worked. Then he decided to systematize the process, went systematically to study. And because of this, he did not appear at any conferences last year. Instead, he sat under a big pile of textbooks. Yes, and still I have a bookshelf full of books on financial mathematics, and not on the subject of concurrency, which temporarily receded into the background. Of course, I continue to study and teach her, but the study of everything new by concurrency has faded into the background.
- How much is mathematics in the modern financial industry?
- Indeed a lot. Basically, all this is tied to complex financial instruments. Large organizations, banks constantly trade in something complicated. And to assess the risks of a complex financial instrument, we need a very non-trivial mathematics.
For example, such a simple thing as a Monte Carlo simulation. It is clear to any programmer: he put random numbers on, folded, got. It seems that from the point of view of programming it looks very simple. But from the point of view of mathematics, to understand how to do it correctly, where exactly it is necessary to multiply or divide correctly, so that the result relates to reality, is sometimes a nontrivial task.
- Tell me, what is general financial mathematics?What areas of mathematics that we all studied in universities, it is in contact?
- Basically, it is a mathematical analysis, the theory of probability and such a field of it, which very few people study - stochastic processes, such as the Ito integral. They even seem to be taught a bit of everything, but they give only a base. Recently, I talked with a math member of St. Petersburg State University, he said that he had attended a course in advanced probability theory. But even there they talk very superficially about it. If this is where they are taught, then either in specialized educational institutions, such as HSE, or in very specialized departments. And very few graduates know this. And if you deal with complex financial instruments, then without this knowledge you will not get anywhere.
- I remember what you said that you study in London?
- Yes, this is distance education - in the end I never went there. There, the main thing is that, firstly, you have chosen very good textbooks. By the way, to find a good textbook on any subject is a miracle. When I started to teach students the same concurrency, I had several textbooks that I stumbled upon randomly and liked them so much that I had the thought of teaching students.
And with financial mathematics I was so unlucky while I was trying to study myself. I googled about Ito-calculus, found some books, tried to read them - the bullshit was complete. It is not clear why, why what, there is no structure. And on the course, firstly, the normal textbooks are selected, and their authors give lectures. And all this is presented logically, with an emphasis on practical application. It becomes clear why you need all this. They show you real examples, give a bunch of homework, make you really solve puzzles, write code. And this is all on the one hand mathematics, and on the other hand, applied mathematics with the expectation that all this must be programmed.
Nowadays, in modern financial mathematics, practically nothing is done “for beauty” in a closed form, all models that are actually used, they are not considered in a closed form. This is all numerical methods. And either you on diffur grids are solved using the finite difference method, or you simulate something using the Monte Carlo method. And mathematics is needed only to derive either a finite difference scheme, or to understand what needs to be modeled. And then you need to write code, because without a code you will not get any answer.
- Many people blame modern mathematics for being very far from reality.And then - science is purely practical?
- Math - it is different. I myself also criticize concurrency. I read a bunch of scientific papers on this topic, and half of them are complete rubbish. The people come up with some algorithms, make some comparisons, but no one in practice needs these algorithms.
The same thing here, naturally. Math - she moves herself. There is a separate area, some people who use all this. And that is why it is important to have the right tutorial. It's one thing to take a textbook, where simply abstract mathematics. While you make your way through it, you will spend a lot of time, and there will be little useful, two or three theorems or facts. And it is a completely different matter when the material is structured, when the facts that you need to solve practical problems are selected and chained out of the entire vast base of scientific knowledge.
Own course on multithreaded programming
- Let's talk a little about teaching.You've been teaching for over ten years, right?When did you start - was the textbook of Hurliha and Shavit?
- Almost ten, since 2007. There was no textbook of this then, and that was the whole complexity. When I started teaching, it was very hard for me, because there were several textbooks, of which I mixed material, added various scientific works, material from the articles of the same Herlikhi. That is, the structure of this course I had to create myself.
- How did you do it?
- It all started with one book. I accidentally stumbled upon a textbook that I liked. In fact, the textbook is very strange. The author is Garg, called Concurrent and Distributed Computing in Java . I liked it because, first, everything in it was fairly superficially described - both by Concurrency and Distributed in an applied small textbook; secondly, there were practical examples of how to program it; and thirdly, there were references to real scientific works. That is, you finished the chapter, many theorems were omitted in it and many facts were omitted, but at the end there were references to additional reading.
- To many. This is still a theoretical book, but with a feed for practitioners. With an attempt to select what may be needed in practice. And that she really hooked me. Since this book seemed too superficial to me, I followed the links I found in it, studied the relevant scientific literature, selected something for students and built my course from it. The sequence of presentation of the material, I first had about the same as that of Garg. But I diluted his material with theory, which I took from the scientific works of the same Lamport, Herliha, and so on.
And then the textbook on oncurrency Herlihy and Shavita came out. I then subscribed to many blogs, and as soon as I saw the announcement and reviews, I immediately bought and read. And he experienced a shock: “Here it is! Here is what you really need to read such a course. No more, no less". The desired depth, the desired sequence, the desired latitude. After that, I adjusted the course a little.
“But I haven’t mastered this book entirely.”You know what annoys me most about her?It is written in a rather crooked language, in it English itself is some strange and examples too.Remember, there at the very beginning there are examples with dogs, cans.I sat for two hours trying to understand that the authors even tell me ... And then I suddenly realized that this is a classic example of how to do something on atomic registers.How did you get through this?
- It was easier for me, because I was already prepared. I already had a professional deformation, I read all these examples with dogs 100,500 times in articles by Lamport and other authors. I knew where it came from. That it is not Hirliha and Shavit who invented, but this is a classic, which is already twenty years old. And so for me it was all familiar and dear. And I recommend this book to students as an additional reading, not forcing everyone to read it. From my course I am part of a theory that seems to me of little help, I threw out and I don’t read it to students.
—How do you choose what to tell the students and what not to tell?
- I give the students what I myself thought was very beautiful and non-trivial. For example, I prove to students the Herlihs consensus theorem, because it seems to me that she is very beautiful and allows us to understand the essence of things. It allows you to understand why you need a CAS, that without it, nowhere, that even two streams can not come to a consensus, if there is no higher structure. Showing a versatile design. Although its practice in the form that Herlih, you will never apply. This is a very artificial thing, but it uses techniques that are needed further in real algorithms. And in general, I am talking about those purely theoretical algorithms, in which there are some tricks that are building blocks in other, already practical, algorithms.
- By the way, if you read the work of Lamport, for example, then its construction looks a bit artificial.This is good for proof, but when you open some source code in an attempt to understand the application, you see that everything is different in it.It turns out that in practice these solutions are not effective?
- Yes exactly. The main question that worried researchers 20-30 years ago, when this whole theory appeared, was “Can I do something or not?” Nobody ever thought about efficiency, only about whether it is possible or not. And a lot of beautiful theorems have appeared, what is possible and what is not. And I tell all these theoretical constructions on the registers to students for one lecture, very quickly and along the tops. I do not make them know by heart. Because, on the one hand, I want to show that all this is possible, but on the other hand, all these algorithms have no practical value. And the second part of the course, I concentrate on practical algorithms. For example, on the Concurrent lists, I am telling a completely classical algorithm, which is used a lot where.
- Which of the semi-invariant?
- Yes. And accordingly, I analyze it in detail. Because not only the algorithm itself is practically valuable. Despite the fact that after it there was a car of improvements, some scientific works, it is still on the balance between complexity and performance, it breaks everyone.
But, for example, in HashMap there is no such victory yet. Until now, this is an active research topic, which we are engaged in, among others, in our research laboratory and so on. Because HashMap is generally a data structure of all structures. Used constantly.
There is a bike known, it seems, about the CEO of Amazon. When he spoke to a student audience, students who study algorithms were asked: “Tell us, what data structures will we need? What are the algorithms? What are you using? He replied that the most important thing for them was “hash tables, hash tables and hash tables”. And this is all true.
We have the same. And in large business problems it is always like this: hash tables are the most important data structure. And with them, of course, from the point of view of Concurrency there are a lot of open questions. That is, there is even to see how actively all this is changing in implementation. Although the insides of Java, look at how a regular HashMap is written there.
- In the footsteps of ConcurrentHashMap? .
- Well, it was rewritten not on the trail. Long there was a story. Furor was made by dudes who eight years ago studied how a standard dictionary is arranged in PHP, and without even going into the essence of the website, they made a request where the keys have the same hash. And they began to send it to various sites.
95% of the sites they tried were down. And about this they released a wonderful article. After this article, the struggle for hashes began, including the Sun began to fight for them. Sun had the first iteration - randomize hash codes for strings. Then they realized that it was not just the lines. And in general, it all ended with the current implementation: if the bucket of the hash table overflows, it will all be replaced with a balanced tree. The goal of this is a stable fight against hackers. So that in the worst case the hash table does not degenerate into the O (N 2 ) -algorithm, but so that at least O (N log N) works. Because in a real web site, what scale do you come across with N? Maybe ten thousand, one hundred thousand ... And if you have somewhere N 2 somewhere, then your system is down. Because 100,000 squared is already too much. And if you have a small N - then nothing else, normally, will live.
Multithreading: science or business?
- As far as I understand, the decisions of your company are connected with high loads, with multithreading.Tell me, is it your interest in oncurrency that caused you to do this?Or maybe the other way around - did the work make you plunge into this topic?
- It is rather a coincidence. It is difficult to trace the causal relationship. When we started (and this was in ancient times, in the 2000th year or even a little earlier), we, like a small company, took up everything. In those days, programmers were worth their weight in gold, people were ready to order anything to anyone ... And we somehow accidentally got a couple of projects in the financial field. We began to do them, and we liked it. And two years later we realized that we didn’t want to do anything else, we stopped taking any other projects. We decided that there was no longer any creation of websites, no outsourcing, we would only deal with trading applications, brokers and exchanges.
Everything else is a consequence of this decision. You start to engage in these applications, and you start having problems with the transfer of quotations. Because there are a huge number of them, and the main trade is focused on the American market. Again, it so happened historically that we were engaged in more complex financial products, all sorts of options. And there are millions of tools, tens of millions of quotes per second. All this must somehow be handled, sent to clients. There are also many customers, tens of thousands.
And we had to figure it out. First, you rewrite, rewrite, study, profile the code, find the bottlenecks, figure out how to fix ... For the first few years, everything was done entirely with his own blood and sweat, invented solutions, without any science. And then science somehow came by chance. We were all self-taught in this area, every book was read from cover to cover.
And when Java Concurrency in Practice appeared in 2006, you took it and saw that it was a treasure! But you already know all this, it has already reached all of this, because you had practical tasks. But you understand how cool that someone systematized all this. And you know that if a new programmer comes to you, then you can tell him: “Here, take and read Java Concurrency in Practice. There is a summary of knowledge. ” And you start to appreciate this book for this knowledge. And you are ready to subscribe under each page with your blood! There the author writes, and you already know: “Yes, damn it! I was there! True! So do not do it - otherwise “the snow will fall on the head - you will be completely dead”.
- In general, it is very interesting, because the book is really quite old.And the results of those Performance tests that are there today look pretty wild.Now the reality is completely different.It is interesting to see how this all develops.
- Its value is not in the Performance-results, of course. Its value in design templates. It structures knowledge and makes you think correctly. Because the main problem of Concurrency is that it is very difficult to test and it is very easy to allow bugs. The test can be run at least 10 times, but still it will then fall on a hanging system due to the switch. Therefore, it is necessary to write the code so that both you, and the neighbor, and the one who will read this code in a few years, it was clear that it is correct. Because writing code is somehow confusing, then how can you convince yourself that it is correct? And here teaches Java Concurrency in Practice. She teaches patterns. If you write with them, then it is easy for another person to get to know them, to understand that everything is right there.
About the approach to creating API
I recently spoke with Peter Laurie , who makes OpenHFT .He believes that now the whole struggle between the frameworks is no longer for Performance, but for the solution provided to be a kind of closed box that solves all the tasks of Concurrency.Because when we give some kind of Concurrency pens to people, everything falls apart.How do you struggle with the fact that your code then hits someone who does not understand multithreading?
- We are not fighting. We have a different business. We make decisions for end customers, so we don’t have such a problem. We give the tool to specific people more than programmers.
- That is, you do not give API?
- We give, but we try to simplify it. We solve this multi-layered architecture. We have a low-level API, where all guts are out, and where a step to the left, a step to the right - hit a mine. But only we ourselves use this low level, and in our company there are a very small number of people who know how to walk along these mines.
And there is also a high-level API, where everything is convenient and clear. And we don’t even tell our customers what is inside. It is documented, it is understandable. There, too, occasionally bugs come across, but we find them and fix them.
What does the company Devexperts
- Very interesting, tell me more.Do you have any distributed systems that transmit quotes to customers and back exchange, or what?
- We have a lot of things. We generally write a vertically integrated solution. Working with setup data quotes is just a small part of our business. We have a separate product dxFeed - this is actually a separate company in America, which is engaged in collecting data from all exchanges (where there are courses, quotes), distributes and sells to customers in a packaged form through a single API in a single format. And this is only one small direction. Very small - a dozen people work there.
And the rest of the company - we just don’t write anything ... Take a broker of some kind, he needs a bunch of information systems, he needs to take clients into account, he needs to maintain his back office, he needs to take into account transactions ... We have a huge number of teams that have these systems create
- And the stock exchange is also the most different, right?
- We do not automate exchanges. The exchanges prefer to do everything themselves in-house. But we integrate with them in large quantities. These are mainly American, but there is both Asia and MICEX for a long time as our client. And now a good relationship with them, and the RTS was our good customer. Then, when the RTS and MICEX merged, integration into the main information system did not go in our favor. But in general, we have very few customers in the Russian market. Just because the scale is not the same.
- And how much more American stock exchange than ours?
- Yes, it's not even in the exchange. Take the creation of some professional trading terminal from scratch. Every broker wants him to have not the same garbage as his neighbor, but wants to be different. And if earlier they were chasing a smaller commission, but now this is no longer the main feature. Commissions for all brokers are quite low. And the struggle is for a more convenient client interface. Every broker wants something unique. To be able to trade anything, so that there are graphics, nice, fast, not slow down.
- Do you have this web?
- We have everything. We do both web fronts and desktop. Professionals set themselves special racks iron, hang there five - nine monitors. And all these monitors are filled with information. Then it all starts to grow in breadth and height. There are special companies that make for this machine, special racks for monitors.
- This is a Windows, yes?
- Since we write in Java, we work under Windows and under Mac OS X. But specifically such things under Windows usually do.
- So there aren't even any video cards that so many monitors can support.
- You stick at once several video cards, that's all. No problem.
- A powerful computer is needed for this?As far as I understand, the special load of the comp itself does not go, its task is simply to drive and show data.
- It depends on what you do. Of course, this is basically data persecution, but since you have a large screen space, you need to drive a lot of data simply because you have a lot of screens. Well, it also depends on how effectively all the code is written. Many, on the contrary, are engaged in those. analysis, and all databases of quotes are kept on their local computer. If you build something, you often analyze it here. Although now it's all gradually ...
- And why are they?Is this for low latency done?
- No, low latency is generally a separate topic, it is a completely different business. There you don’t need all these monitors, you write code that works really fast, and the main problem is that in order to defeat others, you have to put your pieces of iron right on the stock exchange, in a special colocation, to pay a lot of money for it. And here the monitors will not save you. HFT is a very niche business. We had a couple of low latency projects, but it didn’t work for various reasons. One of them is that everyone involved in low latency is trying to do everything in-house. And we do not trade, we write software.
And you asked what the difference in scale. You see, any large trading system is a project of a million dollars minimum. If you even look at our largest broker, then he cannot afford to spend this money for objective reasons, because they have no way to earn so much. We have several tens of thousands of these traders all over Russia.
All brokers earn commissions, which are generated from our nose. And our brokers simply do not have the money to invest in the development of some sophisticated software.
In America, a completely different scale. There brokers have millions of customers who hold billions in deposits, and the broker earns both on the percentage of these deposits and on the transaction fees. And this allows the largest American brokers to invest in creating really cool systems.
Multi-Threading Lab
- In the early 2000s you did everything.And what are you doing now, when the company has already grown so much?How many people do you have, by the way?
“Four hundred people or more.” I still continue to do everything, including writing code, because I like coding. And therefore I will never deny myself the pleasure of writing something interesting. But not only. , , dxLab. , .
- Listen, we are already talking for a long time, let's talk about one more thing.If I am mistaken, then you are also the director of the European Programming Olympiad, right?
- Not really. I am the chairman of the ACM-ICPC North Eastern European Jury.
- This is the most famous programming contests?
- Yes, there are the biggest prestigious Olympiads for students, called the International Collegiate Programming Contest. They are conducted by ACM - the largest community of professional programmers, which is also known for giving the well-known Turing Prize to everyone every year. It's like a Nobel Prize, but for programmers. Both Lamport and other big people are all laureates of this prize. So, under the auspices of ACM, the largest, most prestigious programming competition is held - ICPC.
- Where do we occasionally see our ...
- Yes. Where periodically our students are all tearing up like a Tuzik heating pad.
- And what are you doing there?
- ICPC is a global Olympiad that takes place in dozens of regions around the world. And we have our own regional zone. It is called the Northeast European Region (NEERC). It includes Russia, Kazakhstan, Uzbekistan, Belarus, the Baltic states. And the system is very democratic. If a team wants to participate somewhere else, it has the right to be asked to enter another zone. Competition universities, this is an important point. Not specific people, and the university sends a team.
- There is a rule that in the final there can not be more than one team from the university.
- Yes. In the final, no more than one team from the university. And one person cannot participate in the final more than twice. Everything is very hard done to avoid professionalism.
The scale of the event is huge. Around the world, hundreds of thousands of people from nearly hundreds of countries participate in the selection process. Only in our Northeast European zone, out of a thousand teams, we select a few guys from a multi-level system who will go to the final.
- How does the selection work - is it standardized?Or at the discretion of the region?
- There is a general rule, and the details - at the discretion of the region.
- Did you develop these details?
- Yes. I, in fact, was one of the founding fathers. I still participated in our first competition.
- How long have these competitions been held?
- The ICPC itself has been held since the 77th year. And I'm there since the 96th year. When I first participated, we did not yet have a region, and we went to participate in a neighboring region. They wrote a letter, asked for it and took us. We went to Stockholm and took first place there, and local people were deprived of their tickets to the final. And they are: "Oh, how so?". And here the director of the Stockholm finale found an original solution, says: “Guys, make a region in your own home”. Then Anton Sukhanov, the winner of the All-Union more Olympiads in Informatics, was engaged in this. He organized the region here, the country is big.
- And you were 20 years old, probably?
- During that first competition I was in my second year. And my team became the first champion of Russia, we won first place in our region, we went to the final. Not so much in the final: not below, but not to top places. And in the next region, since I had already participated in the final two times, I could not already participate, and therefore I became the organizer of this whole affair. Sukhanov then went to work at Microsoft, left everything to me. And since then I have been doing this.
- That is, 20 years already?
- Yes. We had now the anniversary semi-final, the twentieth. And I have been organizing it for 19 years. When I first worked at ICPC, it was terrible. This happened in the Anichkov Palace, I spent the night there for several days, actually living there.
Now we have a great big team of the organizing committee, a lot of people. Each has its own direction. I am the chairman of the jury, and my task is to select the tasks. And I don’t invent everything myself, I have wonderful colleagues who send ideas, we get together, decide which tasks are the best, we work on them. I then do the final proofreading, that's all.
The last few years we have been holding ITMO-based competitions in St. Petersburg. But we have a big country, so we had to simultaneously hold competitions in St. Petersburg and in Barnaul, because it turned out that from that part of Russia, students simply do not have money to fly to St. Petersburg. It began with two, and now we spend in four places at the same time. This decision was made for financial reasons so that people could participate. The last competitions were held in St. Petersburg, Barnaul, Tashkent and Tbilisi. We are trying to bring the Olympiad to the people, so that they could still participate. And then people go to the final.
- And how many teams go to the final from our region?
“Usually a little over ten.” This year, in my opinion, 13 or 14.
- And how many in the final?
- In the final 120-125 teams.
- It turns out that our region is about a tenth of everything in the world?
- Yes. Our region is big. And we perform well historically.
- You took the final of the whole world of this Olympiad?
- Yes. Two years ago. I was the director of this final.
- Is it because your team won?
- Because we have been doing this for a long time and because we won. There were a lot of factors. It is like the Olympic system: each year the final is held in a different place. Different countries wishing to host the final send applications. We, too, at some point sent a request. Not that it was our idea. Rather, it was the idea of the Directorate of the competition: "Let's guys, Russia has never been, and you have been doing this for a long time, hold the final." We scratched the bottom of the barrel and found finances. And they sent a request. And they organized the final for themselves. It was a great experience.
- You spent in the Jubilee, in my opinion?
- Yes. The final itself was held in the Jubilee. The trick is that you should, by organizing the final, place all the participants. Participants arrive at their own expense to you for the final, and then you must fully ensure them. Settle, feed, entertain. Full board.
- And how many people come?120 teams, each team is probably about six people?
- Three people plus a coach. At least four.
- Spare, probably more?
- There is no spare. They were canceled, just to save money. We decided that instead of taking the spare, it would be better to invite more teams. Therefore, the team officially - four people. As a result, 1200 people came to us somewhere. This is a string of buses. Everything was in the summer. We all settled in Isakievsky.
- It's white nights and stuff.Where are the hotels so much to take?
- Astoria, Petro Palace - all right in the center.
- This is a lot of money in fact.Who financed it?
- It was a federal grant.
- That is, you came to the Government or to the Ministry of Education and said: “Guys, this is the story”?
- Essentially, yes. It is a great merit of ITMO and its rector Vasilyev, that he managed to win a large educational grant for the development of education in general. It was not a separate grant for this Olympiad, it was such a huge grant for the creation of an education system and support for the education of programmers. And a small piece of this grant was the holding of the ICPC final.
- That is, financing was offered, did you not particularly interact with the city there?
- We interacted with the city on issues of support from the outside, so that the police would accompany the whole thing, do buses there. Advertising around the city.
- And why advertising?
- And you know, the city itself suggested. The city also has social advertising space. And since the city participates in this, helps, supports, then they told us: “Here we allocate to you some places”. Together with them we developed advertising layouts. And she was hung there. It's nice, you come to the event - you have advertising hanging.
And this is really a wonderful event. This is the most prestigious programming competition in the world. That is, the fact that it came to our city is actually super. This is a great achievement.