⬆️ ⬇️

And the bear, it seems, is highly loaded

At the end of June, Novosibirsk took HighLoad ++ for the second time. If last year, the effect of the novelty played a significant role in the fact that everyone was happy with everything, but this time we needed to prepare much more carefully to preserve and exceed the impression. First and foremost, with triple strength, we took up the main thing - the program. Now, when the speeches were held and even selectively reviewed in the second round, the feedback from the participants of the conference has been collected and processed, we can confidently say that it has been a success.



The basis of HighLoad ++ Siberia was 38 excellent reports, full of experience and deepest expertise. International giants and regional companies all shared their experience in working with tasks for which standard tools simply do not exist. And people made a special conference. We honestly tried to understand what was the matter and how to make the same spiritual events in other places, but no - because of the inexpressible friendly atmosphere, we will have to go to Siberia again, and we are all for it. In addition, we are already thinking about new products.





')

They brought the bear with them; not a single representative of the local fauna was hurt.



But what we are not going to change is the approach to the selection of reports. Unless we start to approve topics earlier, that we are already using: HighLoad ++ in Moscow in 4 months and some reports have already been announced . So far this is the only high-load conference in Siberia, and the amount of useful information and technical hardcore details was about a third of the big Moscow brother, and if you go to the density on the souls of the participants, it was much higher. Some reports in the opinion of the program committee can be safely included in the top of all reports on high loads over the past decade. This is confirmed by the audience ratings - the average score of reports 4.2.



To give you an impression of the HighLoad ++ Siberia program, here are a few brief notes. This is not a top voting rating, and there is no need to pay attention to the order - it is just a collection of interesting topics that are different enough to be a small representative sample. All videos for the full picture, we gradually put on the youtube channel (subscribe, put likes, press the bell - these are all blogger stuff to see updates).



Video calls: from millions per day to 100 participants in one conference

Alexander Tobol (Odnoklassniki)



Now all popular messengers have the opportunity to call the other party. Of course, it is convenient to use the same tool for any connection. Therefore, if you have a corporate communication tool, but there are no calls yet, they should be added. Where to start, what protocols and technologies to use, Alexander Tobol ( alatobol ) knows. Even if in the near future you do not plan to develop a video call service, Alexander’s report is full of hardcore details about data networks in general. Probably, therefore, his report received, it seems, a record mark of listeners in 4.9 out of 5.







Last year, Alexander talked in detail about the P2P device calls, but this time only recalled the main points and moved on to features such as signaling and coding for one-to-one calls. It is easy to understand that if one of the participants has lost the quality of communication (and most of this traffic goes through mobile networks), then the bit rate will have to be lowered for all participants in the conversation. But in order to decide what to do with it, it makes sense to see what others have achieved in this direction, and - obviously - take the best and correct the disadvantages.







In Odnoklassniki have chosen:





In the final comparison, this solution is comparable to Zoom in terms of delay, battery consumption and quality, but Zoom is not compatible with WebRTC (and we all read the news about it). When you decide to repeat the procedure and also compare competitors - do not forget OK. Or immediately take advantage of Alexander's advice, his report was once again full of important technical details, which seems to be fine for DIY instructions.



How to create a high-loaded system of sending notifications for events

Artyom Gashkin (CFT)



The company CFT is a bright representative of regional IT-specificity - a large Fintech enterprise. This report dealt with the work of the CardStandard processing center, which - just think about it - processes payments on every third card in the country.







As soon as you pay for something, this processing informs you via SMS or push. The bank that issued your card issuer also wants to receive such notification online. This is the goal of the project, which Artyom Gashkin talked about: to implement a module for sending notifications that would cope with double load. Unfortunately, Artyom had no right to give exact data, said only that the load on individual modules reaches 200 transactions per second. At the same time, work was done to reduce the load by changing the system settings. The developers wanted to make such a performance margin so as not to return to this issue as long as possible. Solution requirements are fairly standard, but the main thing is that the time for processing authorizations should not increase.



Traditionally, an enterprise company uses Oracle, which, if possible, is very difficult to scale horizontally. Therefore, in order not to increase the load on the database, i.e. to keep the minimum number of connections to the database, Apache Kafka was chosen.



The choice of the implementation option was approached as it should be for engineers - they measured the time of moving 400,000 records from one topic to another. This data can be interpreted as the time during which the processing will restore its operability after a failure. We stopped at a producer with asynchronous delivery waiting, finding that 20-30 seconds is an acceptable recovery time. Artyom also talked about the concrete implementation - on the one hand, everything is on the surface, because Kafka guarantees that if you send two records to the topic partition consistently, they will be delivered in the same order. On the other hand, the developers had to dive deep into the features of work and documentation. At the moment, notifications about the completed operation are sent to the bank in approximately 0.5 seconds.



Scaling the resulting system is easy - if the load increases, you need to increase the number of specific handlers. The modules for routing and sending notifications work without changes, and the module that generates notifications is responsible for business logic. When one-time passwords, for example, need to be sent quickly, for example, it will be sufficient to implement new handlers. This refinement does not affect the existing system, and all regression tests will pass without problems.



Since we already mentioned CFT, we will tell you how their partnership embellished the conference. They organized a whole lounge area in which competitions and games were held all two days. But StudioCFT, a mobile studio for recording podcasts with speakers and conference gurus, became the highlight of the program. Guests include: Vladislav Blinov and Valeriy Baranov from Tinkoff Bank, Sergey Sporyshev from ITSumma, Victor Eremchenko from Miro, Sergey Polovko from Yandex, and also Oleg Bunin and Alexey Obrovets ( talk about what men talk about in 2019). Interviews are posted on the youtube channel of the company.







Best GEODIST () west of Rio Grande

Andrey Aksenov (Avito, Sphinx)



"Use linear interpolation, boys."


Andrew began in the usual ironic style, they say, if this is understandable, then you can disperse. It is clear, it is understandable, but if you also apply the experience of the developer Sphinx, on which Avito search works, then everything will be better. The theme for HighLoad ++ Siberia Andrew chose the function GEODIST (), which is particularly used for sorting, filters, map search, etc.







Task: find the distance between two points defined by two coordinates. It would seem that the seventh grade, the second quarter. But if the distance is calculated not within the tetrad leaf, but at least on the scale of one region of the Russian Federation, then the distance should be considered on the “sphere”. Or rather, the ellipsoid. And absolutely accurate, geoid. After all, how not to get involved with a geoid, which approximations and ancient optimization techniques work in a large production, we will not retell it - see the report.





Experience modeling from the team ComputerVision Mail.ru

Edward Tiantov Mail.ru Group



The team of computer vision solves problems for Cloud projects, Mail and specialized B2B product Vision. This recognition of persons and attractions for photos, text from photos for mail, etc. The content of his report, Edward Tiantov ( EdT ), began with an assertion suitable for any area, but especially relevant for AI:



“Task setting is a critical stage.”


At this stage, the constant collaboration of the Product Manager and the machine learning specialist is necessary. If you try to save money on it, you either miss something really important, or spend several months learning to recognize drawn faces, and the production model will never meet any such example. By the way, Edward will talk about UseData Conf . In September about other features of project management based on machine learning technologies.







And only when the statement is clear to all participants in the project, can one begin to work on a task that begins with data preparation. Obviously, they need to be cleaned, both from bad markings and debris inside the classes, as well as from excessive or insufficient divisions into classes. And sometimes data has to be generated, and if it is enough to take a PIL and distort the text for the character recognition task, in other cases you need to be much more creative.



Turning directly to learning, in the field of metric learning, Edward, relying on the extensive experience of Mail.ru, unequivocally recommended Angular Softmax for both pattern recognition tasks and classification in principle, and spoke about the tricks that make it more effective.







And the inclusion of a fairly simple knowledge distillation and decomposition with almost nothing gives + 0.5–1% to an AP. Byte worked great for texts Byte Pair Encoding, and training in FP16 with Nvidia's Apex saves 20% (twenty!) Of time for almost nothing.



How to carry models for production is a separate big conversation, because data scientists are even PyTorch, and nobody wants to deploy it at all. A good option, how to deal with it, has appeared recently. The developers of PyTorch realized the pain of their users and released TorchScript, which serializes the Python model into a static graph. With this conversion, everything works exactly as in Python, and the first wave of bugs is already captured - you can use.



Amazon Scaling

Vasily Pantyukhin (Amazon Web Services)



This report is a typical example of how we get global experience from international companies. That's right - we call Russian-speaking children who grew up in our engineering culture and at our conferences. Foreign speakers are good for advertising, but in fact participants usually rate their reports not too high. Our, in the sense of general in Russia, professional conferences are aimed at the exchange of professional experience. And overseas reports about new companies are popular, which no specialized technical conference will take into our program. Why is this so - a separate question, but we try to choose Russian speakers, other things being equal. This in terms of lack of language barrier and difference in mentality is good for understanding the material.



Returning to AWS, millions of users are simultaneously served by their services, achieving the desired results. The system automatically scales, elastically adjusting to each user so that it seems to him that he is alone in this universe. So thought Vasily Pantyukhin, until he became an architect at Amazon. Now, of course, he knows what is under the hood of the AWS cloud services, and shared some interesting aspects with the conference guests. What physical servers are behind the ephemeral concept of the cloud, which optimizations are used for them, how the most scalable AWS Lambda service works, how Amazon builds its own databases, how the network works.



Most of all in the speech, Vasily is captivating inspiration, with which he talks about solutions, and the mentioned fact that having plunged into the internal structure, he only began to trust the company's services more. This report is also useful as a review for those who are just starting to work with AWS, as well as those who need technical details - that is, everyone.





Vasily also conducted a élap on the subject “Brain-friendly diagrams,” which, judging by the discussion in the chat , was very useful. And in the record it will not be, this is a special format. Therefore, life hacking for the future: if at our conference you choose between the report and the mitap, then give preference to the mitap, and then look at the report in the notes, good, we all record and give the video in a few days.



NodeJS backend

Yuri Gavshin (Bolt)



Bolt - from English fast moving - a platform for the provision of transport services: taxis, private carts, motorcycles and rental of electric scooters. Over the past three years, the company has grown more than tenfold in all respects, which is why the main requirements for the backend are: fast withdrawal to production, resiliency and resiliency (dropped service, does not affect the performance of its neighbors), scalability.



The basis of the stack - NodeJS. Its distinguishing feature is non-blocking I / O and asynchronous networking. There is no unequivocal opinion as to whether this is a good idea and how complex services can be made on NodeJS. It seems that there are not so many reasons to choose Node instead of a mature server language, but the short time-to-market is just one of them, so the production experience of developing a high-loaded backend is very interesting. Moreover, Yuriy described in detail and with examples how to effectively use the advantages and level the minuses of NodeJS, for example, recommended using TypeScript and switching to async / await. He paid attention to such features as the inconvenience of building monoliths. NodeJS forces developers to limit the size of services, and this, according to the Bolt team, is a plus. He touched upon the topics of testing and monitoring.







Everything that was left behind the frame of the report, and much more could be learned from the service developers at the company's stand. In addition to the technological stack, they also discussed relocation to Estonia - but this is already within the framework of the mitap, under which traditionally there was a place and time. Spontaneous small meetings and lively discussions are why conferences are needed. Where else can you look at Zabbix live coding tricks or share documentation pains?







In general, this time we were supported by more companies. We think we passed the test last year and proved to the locals that we are working for their benefit.



Needed where was born



So it seems to us. And this is the main idea that we pursue, bringing our conferences outside of Moscow. So that cool developers are not forced to flock to the capital or foreign offices of large companies because only there will be challenges for them, you need to develop professional regional communities. For example, we specifically invited community activists from Novosibirsk and Krasnoyarsk, and at the same time St. Petersburg and Krasnodar to share their experience and more successfully pump their region.



Here, however, it should be noted that we still advise to go to Moscow. Content at our conferences is not repeated, so if you want to be aware of everything that is happening in the world of high loads, we will meet in November in Skolkovo.



Networking and others.



Eat a friendly atmosphere began on the eve of the meeting of the speakers. This is now our tradition - after the runs on the site, but before the conference, to arrange a meeting. In order for the speakers to get to know each other and the program committee, to see that people are the same around, stopped worrying about the performance and came on Monday in a great mood, ready to share their unique experience.







And then already during the conference, we also had room for communication and even entertainment. Not all the same in a hardcore dive, sometimes it is necessary and switch. Stands for this very it.



Sberbank had target shooting as mobile games, plus more familiar video games to IT players.







Sportmaster could not stay away - set table hockey, which is not empty for a minute. In the meantime, waiting for the party to end and your turn will come, you could ask the developers about Sportmaster products, which, of course, are not limited to an online store.







After a continuous coffee break, which is great for chatting with colleagues and new acquaintances, the leading halls tended to warm up and fortify themselves and invited the audience to the lectures.







Those who did not succumb, went to the mitap.







When, after this, the brain has already turned into turbo mode, any puzzles on a piece of paper and puzzles become uneasy and are solved in two accounts.







In the evening, a well-deserved rest. True, again with a strong intellectual component - a quiz quiz, pliz was prepared for the guests.







In short, had a good time. Mentally.







What's next



Let's be honest, we see a lot of growth points, and we are already planning what will change in HighLoad ++ Siberia 2020 and what other topics we need to throw firewood into.



We will experiment with the formats, not only in individual points of the conference, but also in the whole event. Let's take a closer look at other sites, because they say that Novosibirsk loves camps. In any case, by the next summer we will have time to test the ideas and arrive with tested hypotheses.



Write in the comments what was not enough for you, which of the Ontik conferences you want to see in Novosibirsk or your other city. DevOps , TeamLead Conf , KnowledgeConf - once the Novosibirsk community of technical writers are very actively involved in its organization - offer, and we will figure out how to implement.

Source: https://habr.com/ru/post/460479/



All Articles