Review of the best reports from HighLoad ++ 2017
In the next few articles I will talk about the best ( in the opinion of the participants ) HighLoad ++ 2017 reports. The organizers kindly opened up access to videos that you can right here and watch.
For me, this is the opening of the year - at the powerful technology conference, the first place is given to the report, albeit from a techie, but about MANAGEMENT. Of course, you can argue that the humanities are more willing to give marks and, by default, a more loyal audience, but the fact remains.
The story about the features and problems in the transition from the engineer to the head. The report can be divided into two parts. The first one, which reveals the topic of the report - about the problems in the transition from the engineering staff to the management team, and the second - tips for managers on how to grow and care for newly minted managers. In fact, the subject matter that is considered covers all levels of the hierarchical ladder, since “Sores” are characteristic of moving to any next level of control.
')
It is always interesting to listen to a professional who honestly and openly shares experience, and, both successful and unsuccessful.
The first and very cool advice sounds like - “be honest, about what you do well and what you can do.” After all, this is essentially the only support that you will have during the transition to a new level. In fact, Artyom recommends conducting a self-examination session and honestly (this is very important) understand where your weaknesses are, so as not to use them under any circumstances, or at least not to use them first. If you have a good memory, you are a good communicator - use these qualities.
If you still managed to become a leader, then you should be prepared for the following unpleasant moments to be encountered on your way:
The second part of the story is devoted to the issues of cultivation and search for managers within the unit: search, testing competencies, recruitment, and most importantly support in the early stages of the formation of a new leader. You can not throw a person, no matter how cool he seems.
In conclusion, I will give my hit parade of episodes from this story:
Artyom gives an example of how he and his ward tried to create a manager for a year, but in the end they admitted that all this was not the case and the engineer remained an engineer. At the same time, she especially notes that there should be a legend in case of failure, when the team will have to explain why the person returned to the previous position (my personal experience suggests that this is the most difficult task in such a situation).
In the second place is the episode when, to the surprise of a part of the audience, the phrase suddenly sounds - Being an engineer is much more fun in fact, and the transition to managers is not a transition to a country of pink ponies.
Well, the pedestal closes the story that after 10! years, nothing human is alien. Prufpik.
It remains to be noted that 25 minutes of questions and answers after the end of the report once again demonstrated the relevance of the topic of management and career development in IT.
In second place is perhaps one of the most charismatic people in IT Co. today. Super professional in his field, the author of the popular search engine Sphinx, “Voronezh cattle” (do not think, this is a quote from a report at one of the past conferences :)), merry fellow and matershinnik with a “left-cutting” point of view on many questions - Andrey Aksyonov. Whoever has not heard his reports at least once, I strongly recommend.
Recently, Andrei began to talk about general technical topics, in this report the focus on compression is an overview of the methods and approaches for compressing different types of data. We are talking about general approaches and algorithms that are applicable in almost all areas of programming, regardless of what language you write - in C ++ or, I'm sorry, in PHP.
In general, compression may not be necessary to solve your specific problem, but nevertheless, at any moment a situation may arise where compression will save you. Even if you do not store anything (yes, yes, compressing variables — large arrays are sometimes also very useful). From myself I want to give a simple household example - at the moment when a catastrophe occurred with disk space, transcoding of video with a lower bitrate came to the rescue, which is essentially one of the examples of compression, albeit with a slight loss of quality, as well as recompression of images, That does not give a good result, but on a large number of files - the exhaust is very noticeable.
The whole story is filled with excellent examples, in fact, everything is based on the analysis of examples of solving various problems. IMHO one of the best formats for presenting information.
The story has two parts:
So, about coding (not to be confused with writing code).
“Frequent packing, rare inflating.” Everything. Disagree. Inflating under compression sounds seditious, but in fact it is not.
“Intuition is not to be trusted when it comes to large amounts of information. Need to check. Build frequency tables. In tests, the most common symbol is most likely a space and a carriage return. And not always remember about it.
But everything is not always great. An excellent example of how to get a shot in the leg through an unsuccessful compression is given. Encoded frequent (space) in one bit, and inflated rare to 9 bits. As a result, they received savings of 3%, but at the same time they deprived themselves of the possibility of quick access to the elements of the text array, since items are now unclear how long. It turns out that the principle is generally correct, but it was not applied very much.
The conversation goes on about how the coding algorithm evolves step by step - “we are looking for a balance”, where is actually frequent, and where is rare. As a result, we got an algorithm that compresses a little worse, BUT, it eliminates the work with bit-by-bit reading, which, all other things being equal, is rather a gain.
“Don't be shy about inflating the rare very much.” True, the same nuance comes up again - it does not always work. It all depends on the data set. In general, compression is all about analyzing data sets, selecting algorithms and methods. With a good choice, it is your result that can surpass all classical algorithms, both in compression quality and speed of operation.
What is still guilty bits? In fact, nothing is simply more difficult to implement a read operation, which leads to a decrease in speed, albeit to an increase in space savings. Then bytes? BUT, as Andrew rightly points out, if you compress a byte into a byte, you will be surprised to find that the compression ratio is one. In other words, the assertion is once again confirmed that everything has its time and place.
By the way, it is worth noting that almost all the examples that Andrew cites, to one degree or another, can easily be reproduced in most modern programming languages ​​with minimal refinement, and most importantly are simple to understand.
Here everything is much broader, because Each content type has its own characteristics. Consistently over the years, we look at the evolution and birth of algorithms:
Alas, but since then, nothing new has been invented. Inside all the transformations, nothing outstanding, and most importantly, almost all use the same LZ, but in its more modern implementation of the LZ77. Prufpik.
And what if this is not a text? For example, in the case of search engines, when almost all the data are huge arrays of numbers. And here mankind came up with a solution - delta coding and various variations on its subject. But here, Andrew recalls - not everything is so cool, there will always be situations where it does not work. And back to the idea of ​​selecting an algorithm for data and tasks. For those that want to expand their horizons, here is a list of key words (not all) that sounded in the report.
In general, if we summarize the report, we can draw the following conclusions:
In the meantime, we invite you to continue the discussion on the topic of timblids at our conference TeamLead Conf , and compression and programming in general with the help of properly growing hands - at the Backend Conf conference on server programming.
Goth2Boss: Breaking and Waste Potential when Moving from an Engineer to a Timlide / Artem Kalichkin
For me, this is the opening of the year - at the powerful technology conference, the first place is given to the report, albeit from a techie, but about MANAGEMENT. Of course, you can argue that the humanities are more willing to give marks and, by default, a more loyal audience, but the fact remains.
The story about the features and problems in the transition from the engineer to the head. The report can be divided into two parts. The first one, which reveals the topic of the report - about the problems in the transition from the engineering staff to the management team, and the second - tips for managers on how to grow and care for newly minted managers. In fact, the subject matter that is considered covers all levels of the hierarchical ladder, since “Sores” are characteristic of moving to any next level of control.
')
It is always interesting to listen to a professional who honestly and openly shares experience, and, both successful and unsuccessful.
The first and very cool advice sounds like - “be honest, about what you do well and what you can do.” After all, this is essentially the only support that you will have during the transition to a new level. In fact, Artyom recommends conducting a self-examination session and honestly (this is very important) understand where your weaknesses are, so as not to use them under any circumstances, or at least not to use them first. If you have a good memory, you are a good communicator - use these qualities.
If you still managed to become a leader, then you should be prepared for the following unpleasant moments to be encountered on your way:
- A burning desire to shield their employees from external threats (in the Corporation of Good there is a special term Shit Umbrella, which very vividly illustrates this behavioral pattern). From the point of view of Artem, this is one of the greatest evils for the team and for the manager, since the unit becomes a black box for the outside world, problems are being masked, the level of self-criticality and its errors is reduced. All this leads to a decrease in quality and interest in the work. Any situation that takes a person out of the comfort zone stimulates him to connect additional resources and work as efficiently and productively as possible. Want to get it - close the umbrella, though not immediately. By the way, about how to close an umbrella correctly, there is a question and an answer after the main report. As an illustration board, Artyom cites an episode from the life of his colleague, who, in the case of the facie, joined the team and said, “We screwed up”. What is important here is that there is no one who is guilty. As a result, the team rallies, mobilizes and corrects what caused this arrival.
- The complexity of the perception of pending results. Here we are talking about the fact that the engineer has quite a concrete result of the work, which can be achieved in the foreseeable time and realized. As for the head, then everything is more complicated, because the very essence of leadership does not imply an objective assessment of the actions of the head. Hardly anyone can say unequivocally - “yes, we have implemented the processes successfully.” More often you can hear the epithets "in general", "enough", "almost", etc. Those. the final result can be achieved and will be achieved, but not immediately, and most importantly, all the time, until there is no result, the manager is under the impression that nothing is happening and some uncertainty, that what is being done is being done in the direction of achieving the goal.
- The desire to develop ideal solutions. It is important to remember that “the best is the enemy of the good.” Unfortunately, the current realities are such that, in almost 100% of cases, the manager is forced to make a decision in the context of lack of information, time for analysis, or lack of resources to develop that ideal solution. It sounds a bit strange, but Artyom claims that if you made an ideally-correct decision, then you must have missed something somewhere. And at least spent too much time on its development. From myself I will add, such decisions are made often enough, but it’s more like luck and much experience.
- Relevant slide
The scourge of all newly-baked managers is the desire to do everything on their own and generally do something with their hands. A very important point - it is important to distinguish the problem of delegation from the desire to work with your hands. Not always the desire to work with hands is connected with the lack of ability to delegate. Most just like to do engineering work and slightly reduce the very “professional breaking”. Extremely useful advice - work with your hands, but shift the direction of your activity from tactical tasks to strategic ones. Quote: "Former engineers do not exist."
The second part of the story is devoted to the issues of cultivation and search for managers within the unit: search, testing competencies, recruitment, and most importantly support in the early stages of the formation of a new leader. You can not throw a person, no matter how cool he seems.
In conclusion, I will give my hit parade of episodes from this story:
1 place.
Artyom gives an example of how he and his ward tried to create a manager for a year, but in the end they admitted that all this was not the case and the engineer remained an engineer. At the same time, she especially notes that there should be a legend in case of failure, when the team will have to explain why the person returned to the previous position (my personal experience suggests that this is the most difficult task in such a situation).
2nd place.
In the second place is the episode when, to the surprise of a part of the audience, the phrase suddenly sounds - Being an engineer is much more fun in fact, and the transition to managers is not a transition to a country of pink ponies.
3rd place.
Well, the pedestal closes the story that after 10! years, nothing human is alien. Prufpik.
It remains to be noted that 25 minutes of questions and answers after the end of the report once again demonstrated the relevance of the topic of management and career development in IT.
I want to compress everything / Andrey Aksyonov
In second place is perhaps one of the most charismatic people in IT Co. today. Super professional in his field, the author of the popular search engine Sphinx, “Voronezh cattle” (do not think, this is a quote from a report at one of the past conferences :)), merry fellow and matershinnik with a “left-cutting” point of view on many questions - Andrey Aksyonov. Whoever has not heard his reports at least once, I strongly recommend.
Recently, Andrei began to talk about general technical topics, in this report the focus on compression is an overview of the methods and approaches for compressing different types of data. We are talking about general approaches and algorithms that are applicable in almost all areas of programming, regardless of what language you write - in C ++ or, I'm sorry, in PHP.
In general, compression may not be necessary to solve your specific problem, but nevertheless, at any moment a situation may arise where compression will save you. Even if you do not store anything (yes, yes, compressing variables — large arrays are sometimes also very useful). From myself I want to give a simple household example - at the moment when a catastrophe occurred with disk space, transcoding of video with a lower bitrate came to the rescue, which is essentially one of the examples of compression, albeit with a slight loss of quality, as well as recompression of images, That does not give a good result, but on a large number of files - the exhaust is very noticeable.
The whole story is filled with excellent examples, in fact, everything is based on the analysis of examples of solving various problems. IMHO one of the best formats for presenting information.
The story has two parts:
- Coding with code examples and a clear demonstration of the pros and cons of different algorithms.
- Transformation, overview, because the variability is too high - we transform the pictures in one way, the video in another. But there are examples too, and they are close to many.
So, about coding (not to be confused with writing code).
Tip 1.
“Frequent packing, rare inflating.” Everything. Disagree. Inflating under compression sounds seditious, but in fact it is not.
Tip 2.
“Intuition is not to be trusted when it comes to large amounts of information. Need to check. Build frequency tables. In tests, the most common symbol is most likely a space and a carriage return. And not always remember about it.
But everything is not always great. An excellent example of how to get a shot in the leg through an unsuccessful compression is given. Encoded frequent (space) in one bit, and inflated rare to 9 bits. As a result, they received savings of 3%, but at the same time they deprived themselves of the possibility of quick access to the elements of the text array, since items are now unclear how long. It turns out that the principle is generally correct, but it was not applied very much.
The conversation goes on about how the coding algorithm evolves step by step - “we are looking for a balance”, where is actually frequent, and where is rare. As a result, we got an algorithm that compresses a little worse, BUT, it eliminates the work with bit-by-bit reading, which, all other things being equal, is rather a gain.
Tip 3.
“Don't be shy about inflating the rare very much.” True, the same nuance comes up again - it does not always work. It all depends on the data set. In general, compression is all about analyzing data sets, selecting algorithms and methods. With a good choice, it is your result that can surpass all classical algorithms, both in compression quality and speed of operation.
What is still guilty bits? In fact, nothing is simply more difficult to implement a read operation, which leads to a decrease in speed, albeit to an increase in space savings. Then bytes? BUT, as Andrew rightly points out, if you compress a byte into a byte, you will be surprised to find that the compression ratio is one. In other words, the assertion is once again confirmed that everything has its time and place.
By the way, it is worth noting that almost all the examples that Andrew cites, to one degree or another, can easily be reproduced in most modern programming languages ​​with minimal refinement, and most importantly are simple to understand.
Now about the transformation.
Here everything is much broader, because Each content type has its own characteristics. Consistently over the years, we look at the evolution and birth of algorithms:
- 70s - Lempel-Ziv (that same LZW, https://ru.wikipedia.org/wiki/Algorithm_Lempelya_—_Ziva_—_Velcha , which is used to this day in specifications, for example, in pdf and gif)
- The 80s - PPM https://ru.wikipedia.org/wiki/%D0%90%D0%BB%D0%B3%D0%BE%D1%80% D0%B8%D1%82% D0%BC_%D1% 81% D0% B6% D0% B0% D1% 82% D0% B8% D1% 8F_PPM (7zip, RAR, WinZip and a few more)
- The 90s and 2000s passed under the motto of productivity growth and additional tuning of what they invented earlier.
- 2010e - Brotli - https://ru.wikipedia.org/wiki/Brotli (used mainly by browsers)
Alas, but since then, nothing new has been invented. Inside all the transformations, nothing outstanding, and most importantly, almost all use the same LZ, but in its more modern implementation of the LZ77. Prufpik.
And what if this is not a text? For example, in the case of search engines, when almost all the data are huge arrays of numbers. And here mankind came up with a solution - delta coding and various variations on its subject. But here, Andrew recalls - not everything is so cool, there will always be situations where it does not work. And back to the idea of ​​selecting an algorithm for data and tasks. For those that want to expand their horizons, here is a list of key words (not all) that sounded in the report.
In general, if we summarize the report, we can draw the following conclusions:
- The approaches are all quite simple, the main thing is to think what best methods of compression and transformation it makes sense to apply.
- Nothing new has been invented in 50 years, although they have tried diligently. Everything new is a variation on the theme of the old. Therefore, most likely for your task is suitable for something from the above, especially if you carefully follow the advice. And if you do not believe - look and listen to how cool search engines and databases are arranged. Under the hood is all that was said in the report.
- Do not hesitate to drive benchmarks and compare the results. By results, you can easily abandon gzip'a in favor of zstd.
- Aksyonov is very cool, although from unaccustomed to many it will seem that he slanders too much, but he is. I assure you, all his stories are worthy of attention.
What will happen next?
3. Fraud in mobile applications: Vodim Antonyuk ;
4. We make our firmware for the IP camera on Rust / Max Lapshin (Max Lapshin) ;
5. Big Mail: how we built DataLake in the Russian Post / Alexey Vovchenko;
6. Cheaper, more reliable, easier. Storage of petabytes videos and photos in OK / Alexander Khristoforov;
9. Java and Linux - operating features / Alexey Ragozin;
10. Details on how Causal Consistency is implemented in MongoDB / Mikhail Tyulenev (Misha Tyulenev) ;
In the meantime, we invite you to continue the discussion on the topic of timblids at our conference TeamLead Conf , and compression and programming in general with the help of properly growing hands - at the Backend Conf conference on server programming.
Source: https://habr.com/ru/post/346934/
All Articles