“It seems to me that we are approaching the time when everyone will be able to code. Code is freedom. ”
Habr, hello! We are continuing a series of interviews with Newprolab alumni, in which they tell about their history of transition to the field of work with big data. The stories are all different and will be interesting to those who are thinking about changing their career path or how new knowledge can help solve current problems. Today we have, most likely, the second option. Meet Dmitry Rodin, a developer and contributor to the open source analyst platform Rockstat ; co-founder, co-author and teacher Digital God - educational programs for digital professionals; CEO Xeteq .
We talked to Dima about programming languages and what solutions he writes on them, about his open source project and why he does it, about the new specialty that they teach, etc.
')
- Dima, tell us a little about yourself.
- My caramel craft immediately comes to mind, I have long been joking with it, but seriously, I’m a programmer in my heart, or an architect, I like to invent and create new solutions, to involve other people in them. For the last 10 years I have been involved in marketing technologies, and I got so carried away with them that it turned into a hobby. Developed almost everything: from the advertising management system and DSP to the anti-fraud with hardware monitoring modules. I think I have known Zen, but this is not certain.
- You have been programming for more than 15 years, tell me how many programming languages you know and use in your work and your personal projects?
- I think that counting the number of languages is wrong, the most important skill is to understand several concepts of languages, and then there will be almost no difference on what to write. Yes, of course, each language has its advantages and pitfalls, we should not forget about it.
- And from what languages did you start and what do you use now.
- Now for me there is no particularly big difference on what to write. I started with the development of sites for PHP, it was more than 15 years ago. True, for many years I have not used it.
After PHP was Ruby, then it was the HYIP language, many conferences were held, and its framework, Ruby on Rails, became a model for many others. In general, it was in Ruby that many concepts were born, which were later adopted and reborn in other programming languages and their ecosystems.
Then I began to actively develop in JavaScript (node.js). Browsers once competed in speed and security, and in the course of that race, a very productive JavaScript V8 engine was born, located inside the Chromium browser (note: an open browser on which Chrome is based). Some guys came up with taking this engine and screwing it to libevent, which resulted in a wildly productive, asynchronous thing that did other server-side scripting programming languages, albeit with a disgusting syntax and JavaScript ecosystem of that time. Just in this server language I developed.
As for me, so JavaScript was the most terrible language, it was almost impossible to do well, i.e. write compatible code that is convenient to debug, test and develop. But in recent years, JS has greatly improved: the language itself has evolved, many assembly tools have appeared, and most importantly, add-ins that allow us to describe data types, check them for compliance, and find potential errors that a developer can make, thereby eliminating most of them. One of these add-ons is TypeScript, for several years I have been using it instead of pure JS.
From Python, of course, there’s no way to go, everything is there, it’s convenient and fast enough, you don’t need to invent anything, you can do almost anything you want. It is well suited for working with data: a lot of mathematical packages, handy tools like Jupyter, take at least the absence of restrictions on the size of the number in mathematical calculations.
My favorite, on which, however, I least work - Elixir. It is based on another functional language Erlang, created by Ericsson for programming telecommunications equipment. Its main feature is parallelism and fault tolerance. By the way, he appeared a long time ago, practically my age. Elixir is just as wildly parallel and fault tolerant. Erlang and Elixir have interesting architectural features, for example, one server can handle several million simultaneous connections by using its own implementation of lightweight processes and exchanging messages through lightweight connections. True, Erlang has an extremely inconvenient and inexpressive syntax.
So, Elixir, in fact, is the same Erlang with all its power, but with a much more pleasant and convenient syntax from Ruby. In Xeteq, we built a productive MTA on it (approx: mail transfer agent, i.e., a mail server). However, I am still afraid to do large-scale projects on Elixir. When using it, you have to write too much yourself: what has been implemented in other languages in the form of ready-made libraries for a long time, you have to do it yourself. Although sometimes it is useful to “reinvent the wheel”, because you can invent something new and worthwhile. And the community is kind of sluggish. All projects have a very small community, for example, let's take the basic web framework plug - it has less than 2 thousand stars, its analog is python flask, which has almost 40 thousand, and ruby sinatra has a little more than 10 thousand.
- Let's talk about the projects. Tell us about your most interesting, significant and important projects, and what is their fate.
- My first serious project was Blonde.ru. In 2009 it was a very ambitious project, the essence of which was that each person, not having special knowledge in contextual advertising, can independently launch an advertising campaign, and simultaneously in all advertising networks and from one simple interface. It was there that I grew very professionally and met people who greatly influenced my life. It so happened that after a few months, the technical director’s chair was vacant, and while they were looking for it, I was appointed by the IO. Some time passed, the chair was empty. The head of the company, Yura Yeliseyev, took and made me a full-fledged technical head, I don’t know what he was guided by, I was pretty green for this position, although maybe I saw something in me ( laughs - author’s comment ). By the way, it was there that we met with Alexander Shvets (we founded Digital God with him). He came to us to the position of a programmer while still a student of Baumanka and turned out to be the best in this. In general, we then became very good friends. The project is alive, it was renamed and transformed several times, it entered foreign markets, now it is known as Aori.
- And after Blonde.ru you started doing your first project, right?
- Yes, my friend and I decided to make a commodity widget, in fact, the Yandex.Market advertising network. To make it clearer: there is Yandex.Direct, and there is YAN (Yandex advertising network) - different sites where their blocks are located, and they either catch up with the person through a search, or thematically the show takes place. Yandex.Market didn’t have such a thing, but it had a special content API through which it was possible to receive hot offers from stores and a link where to send those interested. And if the purchase was made, then we were given 50% of the cost that the advertiser paid. Quickly enough, I wrote down the engine - a widget that is placed on the site, it determines the subject, looks what people were interested in, to make an automatic selection of goods from the Market. It worked great. We have learned how to arbitrate well (“arbitrage” - buy an advertising space and make it profitable at the expense of another channel), identify good / bad websites, began to actively buy advertising space on many websites and place your widget there. Among the sites was a major automotive. As a result, we had a very fast growth, but, for some strange reason, Yandex unilaterally broke the contract with us. We tried in every way to find out the reason, but this was never possible, we had to close the project. It was a great sadness, because a lot of energy was invested in this project, and the indicators were very good. Now I’m sure that it was a good experience that taught me how to correctly assess risks. In general, the experience is only positive, and if something does not succeed, it will only help you in the future to avoid even greater difficulties.
- And yet, gaining all sorts of experience, you continued to invent new projects. What happened next?
- Somehow I set myself the goal of making my own counter to collect data about the behavior of people on websites and calculate statistics based on this - like how Google Analytics and Yandex.Metrica collect their data. The code is put on the site, it keeps track of all visitors' actions on the site, transmits to the server, the server side receives this data and stores it in a distributed database. This project is called Rockstat. Later, I thought about making end-to-end analytics service on its basis, which, by the way, is now the pain of many companies. But I didn’t have the energy or time to develop, and I didn’t even begin to start. And it’s good that I didn’t, now I have the opinion that the end-to-end analytics services are doing their job. They are suitable for those who do not understand analytics and marketing, who do not have an understanding of how everything should work, or those who have low needs. In other cases, they run into a lack of integration or incompatibility of work logic. But I found an interesting use for raw data collected by the meter.
- What?
- For example, consider multi-channel attribution based on the behavior of the site visitor, and not on the number of the visit. Attribution is a way to distribute the value of the sale between the advertising channels that participated in it. Earlier, the attribution model based on the probabilistic passing of a funnel has become popular, but it is suitable only for sites with a funnel and a not too long sales cycle - these are most online stores. But there is another side: nedvizhka, avtoshka, wedding dresses, where the cycle of decision making is much longer, and there is simply no adequate funnel on the site. For them, I thought up to calculate the index of activity / involvement behavior. At that time, my counter already had several users, this allowed to test the hypothesis, calculate attribution based on the involvement of users in interaction with the site. I took two sites where there is a funnel and not a very short sales cycle. Compared the results of the model based on the probabilistic passage of the funnel and the model based on involvement, they turned out to be significantly correlated. After all, we live in such a time: if the content of the page does not cling or does not give hope, then it will immediately be closed.
I thought that this could be useful and in demand. He walked through several companies, told how they could improve the method of evaluating the effectiveness of advertising channels, but it turned out to be too difficult ... It is difficult to understand and adds difficulty in justifying costs. In general, there are very few people who want to do their work really cool, who are driven by what they are doing and who are eager to take new peaks. More often it turns out that, encountering a difficult and incomprehensible decision, they would rather score on it, they will not complicate their lives. He spoke at Data Science Week, Moscow Digital with a story about this model. There were many interested, but no one has matured.
And then there was another turn, some guys came to me to talk about the analyst (I thought so), but it turned out they came to offer me a job, and for a very unusual position of marketing director for me; and not somewhere there, but to the promising daughter of Sberbank Plazius. This experience deserves a separate story, so perhaps I will miss it.
While still working at Plazius, I began to teach. Together with a friend, together with the Tceh platform, we launched a full-time educational course “Programmatic Advertising Specialist”. The course helped people to increase their competence in the difficult field of high-tech advertising, or even get a new profession. And then I ended up at Linkprofit, where I started creating marketing technologies that were primarily focused on the market, and not on meeting the needs of the company. Then the company was in the process of restructuring, without even noticing it, I was appointed head of the new IT company Xeteq, which was included in the LT Digital group of companies.
- Wait, but what about the Rockstat? Abandoned him, since no one understood?
- Once I thought: “Why does Rockstat go dead?” Maybe someone will benefit. " Redid the service in the box, added data processing tools and laid out in open source called Alcolytics. And then it started ... A community of very professional guys, whom I used to meet quite rarely, began to gather around the project. They began to implement it at home. I was so amazed! And very competently approached how to use it, I didn’t even have such thoughts. Then I decided for myself that I would only work with enthusiastic and professional people, and let the rest go with the forest, I would not even get involved with those who do not know what they want, who should “sell” the idea and then prove its effectiveness.
Almost all my free time I began to spend on the development of this project, I was very motivated by the apparent return. “What the hell, I used to want to commercialize everything, if I'm preoccupied with the fact that they use it, and people like it,” I thought. In general, at first it was somehow scary, all of a sudden a bug, but I can’t even fix it or will install it wrong, they will spoil everything, and then they will say that the software does not work (* laughs - author’s comment *). I knew about a dozen servers with Alcolytics, probably there were more of them, but this knowledge was enough for high motivation, I constantly modified the project, and it got better and better.
At one user in Alcolytics data from crm and about calls was added, a through analytics was built. These data were needed to build clever cross-site attribution. This had to be done outside, because There was no way to run the code inside. Then I decided to make it possible to create small scripts to develop my API-NIS, which will allow to take this data into crm or else where, where they should be used. He worked many options, but everywhere there was some flaw. I decided to make a mock service for Docker, but I got so carried away with this process that after two months I realized that I had made my microservice platform, and in fact I wanted to do a simple thing for running scripts. I was very fascinated by the thought: “Why do we need analytics if there is no possibility to automate decision making?” It is enough to look several times at the numbers, and it is clear in which case what to do. This allows you to do either commercial software, usually having a decent price, or difficult, in terms of implementation and support, open source.
- Yes, simple solutions are not for you, if you do something, then in a big way.
- I imagined how many different automations can be implemented. They will perform various tasks: manage the advertising budget and rates, or, if we talk about RTB advertising, it will be possible to shift users from one segment to another if they change the status in the system. You can do whatever you want. In the process of working on this platform, I invented many “bicycles”, I had never invented so much before. Nobody will praise me for it, but I like them and they turned out very cool. There is another very good argument - it all works on one server for $ 10, keeping the site with 1-2 million visitors per month, without straining at all. Developed frameworks for 2 programming languages that implement all the necessary functionality, organizing interaction with other microservices. By the way, now huge companies are built around microservice technologies.
And still, when you make an open source solution, you are not ashamed to take a piece from another open source, for example, mobile clients, for the implementation of which a lot of time would be spent. In commerce, of course, this is impossible - these are huge reputational risks, and indeed a disgrace. I learned so much cool about open source when I started working with it.
Now this is not a project that solves specific problems, it is a platform that helps to realize your desires, and it is easy to do it, do not bother with complex and boring tech-nar things.
Recently released in alpha testing the 3rd version , returning the original name Rockstat . There is little documentation, many bugs, but users are happy. There are big companies among them: one airline, and an advertiser from the top-30 of Russia on the queue. Of course, there are a lot more users. We have a small community in TG (@rockstats), the guys will report on problems and achievements.
- Why and why so much time and personal money you spend on the project, which then spread in open source. What drives you?
- It seems to me that the Rockstat project can greatly change the sphere of digital. It is clear that a standard is suitable for small companies, and giants are building complex solutions, but there are companies with talented specialists and complex needs somewhere in the middle, they are becoming more and more. They create bold and effective solutions with their own hands. The market will follow them or not, it is not known, because we know what Russia is famous for. For some time I will develop the project here, but only he will mature a little, I will try to announce it in other markets.
It was a long way, but now I can honestly say: “I don’t know why I need money, where to put it.” I don’t know why I once thought about money, probably it was imposed by society. Even before open source, I began to teach, I liked to share my knowledge, although at first I thought: “Shouldn’t I have to keep some knowledge with me?” But no, you need to share with the community, there are many worthy people that your knowledge can be helpful. And the most important thing is that it comes back.
- And what interesting and cool projects did you do as part of your work?
- Oh, I have all such ( laughs - author's note ). Now there are several projects in parallel. I will tell, for example, about email service. Mailing lists are already quite an adult direction, there are a lot of players, both local and foreign. However, this niche is not occupied: the players do not have enough opportunities for segmentation, the very high cost of mailings, they are all from the 00s, ineffective. Honestly, I did not want to take on this task for a long time. As a result, I decided to start by creating my own mail server - MTA (Message Transfer Agent) - with the minimum necessary functionality. This is still a great job. For 2-3 months of work, we received an MPV (working prototype). In a single user action, multiple servers may start, and a sending system will roll over them. Letters flow like water - it can scale to huge volumes. Resource consumption is minimal, you can send at least several billion letters a day, and the system will not even notice it. But it’s not so easy to send a billion letters - you need to “warm up” the servers, gradually raising the number of letters sent. I planned to make this project on Elixir, though, personally, I didn’t start implementing, but the guys supported the idea, mastered Elixir and did.
- You have such a great experience, you know a lot of things and know how, tell me why you need the program “Big Data Specialist” ? What was the need?
- At that time, the largest projects were limited to a cluster of several servers from a relational database. About Hadoop ecosystem and machine learning did not know much. The course gave me useful knowledge, I remember the whole theory, I know where to dig if anything is required; Immediately I can answer, than models on trees differ from linear and gradients. In general, there were so many things there: both the use of Hadoop, and recommender systems, I still scare colleagues with the phrase “collaborative filtering”, and about building DMP, which was especially important for me - at that time I was working on a similar project. Now we have to work with data a lot: using machine learning to find patterns in the data or to calculate the effectiveness of advertising; determine the values of microconversions on sites. In general, the knowledge gained has found many applications and I am sure there will be even more. This is already real, without this in any way.
- Let's talk about Digital God - your educational project. How did it come about and what are you teaching?
- This project is out of work, of course, but the need arose just from work: I decided to find programmers in a cunning way (this is a big problem), make courses for programmers and teach them to Ad- / Mar-Tech. To test the hypothesis, I made a post on Facebook, but the programmers did not respond. Digital professionals responded. And it was decided to go this route: to teach digital programming specialists. It turned out that it is much more interesting and useful because programmers are given marketing and analytical specifics, but they are not bad at all, and these guys have suffered: they write TK for a week, then wait for weeks for them to do everything.
My friend (Alexander Shvets) worked the course, launched it, and from the very first it worked - we have a third of the participants survived ( laughs - author's note ). They were very tired and tortured. They had a hard time, they had to sit all weekend long, but they got a huge professional growth and opportunities. Now the 2nd stream is coming to an end, and the 3rd is approaching the equator. More than half of the people who come to us know that they need to learn programming. They are already experiencing severe difficulties due to the lack of the possibility of free work with data.
Code is freedom, it is, in essence, a speech for computers. And ready services, applications are restrictions of your opportunities by the interface and the logic developed under the majority. You can do almost anything with code. You need to be thoroughly remarked or think about religion in order to understand what is not decided by programming. And in general, coding is not difficult, because the code is very similar to ordinary English speech. You do not need to be afraid, you have to start learning how to write code, you cannot get away from it.
Rockstat turned out to be very useful here: we do not start from the basics, but immediately proceed to solving problems that we can face every day. In the second lesson, for example, we solve the problem of classifying visitors to a site according to their behavior, understanding which visitor they wanted, having their history. It is used dataset from sites of strip clubs (they are also users of Rockstat and kindly provided a cut of anonymized data). And they found who visited the site for the purpose of employment, and who is interested only in a visit to the club. By the end of the program, everyone should make a course project, according to which we will make a decision on the success of the course. Someone creates his own API, someone considers cross-cutting analytics, one comrade implemented scraper-s for collecting data that cannot be obtained by the API.
Rockstat and Digital God are two major serious projects outside of work. But there is still the most beloved - "Your sweet" . Somehow a friend brought a small caramel in the form of manhood. “Pff, nonsense,” I thought, and threw her in the kitchen, where she lain for about a week. At some point I was bored and I decided to make a candy as large as it should be. Once in that period I was not working, there was a lot of time, and I was confused. In the meantime, I did, I realized that there was no reason for me to keep this valuable knowledge with me, I had to share it with others. I had a lot of fun doing it, and after a lot of trial and error, a box appeared with all the necessary components and instructions on how to make everything yours.
Issue 1 of the Digital God stream course Digital Rockstar
- Let's go back to the educational project. Tell me, what skills does a graduate of Digital God have? What can he do?
- The most valuable thing is that the participants got their freedom, they can now describe their tasks in the computer language, and he will take care of the decision, and will do it on a schedule that will lay down or even process everything in real time. I think the most interesting conclusions will be in a few months, when graduates will gain more experience.
In addition to digital specialists, we also had owners of small businesses that were involved in digital, because in the modern world everything is very tied to the indicators of the effectiveness of advertising and websites, and it’s very hard to find an advertising agency that can adequately take on it, and no automation they will not do.
- Why do you not do in principle corporate programs, despite the incoming proposals?
- Knowledge should not be given exclusively to any one big company, besides, according to my observations, the interest of corporate participants is usually lower. Therefore, I don’t want to come to a course for people who are forcibly sent to study. Maybe I'm wrong. If you really need someone, they can take a course with employees of other companies, but, most importantly, they will be in a group of interested people, the atmosphere and impact will be completely different.
- Starting from Blonde.ru, where you quickly grew to a technical director, you always had commands in submission. Please tell us about your approach to work and people in the team.
- Yes, it turns out that most of my professional life I occupy leadership positions, but in my heart I am still a free artist: I need freedom from routine and bureaucracy, this allows me to think freely and find good solutions. I do not like management, for me it is depressing, but there is no way to go. I am not a team leader in the team, now Sasha (Alexander Shvets, a longtime friend and co-founder of DigitalGod) manages the team and so it’s all better. , : « , . \, ». , , , . , , , .
, IT : - , , , , . , , , , , , . , , .
— , ? , / ?
— , , , , . , - , .
, , . , , . - , , , . , . , , , - , , , .
— , ? , .
— ( – . ). , , , . , . , , - .
— , ?
— - , . — , , , . open source, , , Digital God digital . , . , , – , , , …
We talked to Dima about programming languages and what solutions he writes on them, about his open source project and why he does it, about the new specialty that they teach, etc.
')
- Dima, tell us a little about yourself.
- My caramel craft immediately comes to mind, I have long been joking with it, but seriously, I’m a programmer in my heart, or an architect, I like to invent and create new solutions, to involve other people in them. For the last 10 years I have been involved in marketing technologies, and I got so carried away with them that it turned into a hobby. Developed almost everything: from the advertising management system and DSP to the anti-fraud with hardware monitoring modules. I think I have known Zen, but this is not certain.
- You have been programming for more than 15 years, tell me how many programming languages you know and use in your work and your personal projects?
- I think that counting the number of languages is wrong, the most important skill is to understand several concepts of languages, and then there will be almost no difference on what to write. Yes, of course, each language has its advantages and pitfalls, we should not forget about it.
- And from what languages did you start and what do you use now.
- Now for me there is no particularly big difference on what to write. I started with the development of sites for PHP, it was more than 15 years ago. True, for many years I have not used it.
After PHP was Ruby, then it was the HYIP language, many conferences were held, and its framework, Ruby on Rails, became a model for many others. In general, it was in Ruby that many concepts were born, which were later adopted and reborn in other programming languages and their ecosystems.
Then I began to actively develop in JavaScript (node.js). Browsers once competed in speed and security, and in the course of that race, a very productive JavaScript V8 engine was born, located inside the Chromium browser (note: an open browser on which Chrome is based). Some guys came up with taking this engine and screwing it to libevent, which resulted in a wildly productive, asynchronous thing that did other server-side scripting programming languages, albeit with a disgusting syntax and JavaScript ecosystem of that time. Just in this server language I developed.
As for me, so JavaScript was the most terrible language, it was almost impossible to do well, i.e. write compatible code that is convenient to debug, test and develop. But in recent years, JS has greatly improved: the language itself has evolved, many assembly tools have appeared, and most importantly, add-ins that allow us to describe data types, check them for compliance, and find potential errors that a developer can make, thereby eliminating most of them. One of these add-ons is TypeScript, for several years I have been using it instead of pure JS.
From Python, of course, there’s no way to go, everything is there, it’s convenient and fast enough, you don’t need to invent anything, you can do almost anything you want. It is well suited for working with data: a lot of mathematical packages, handy tools like Jupyter, take at least the absence of restrictions on the size of the number in mathematical calculations.
My favorite, on which, however, I least work - Elixir. It is based on another functional language Erlang, created by Ericsson for programming telecommunications equipment. Its main feature is parallelism and fault tolerance. By the way, he appeared a long time ago, practically my age. Elixir is just as wildly parallel and fault tolerant. Erlang and Elixir have interesting architectural features, for example, one server can handle several million simultaneous connections by using its own implementation of lightweight processes and exchanging messages through lightweight connections. True, Erlang has an extremely inconvenient and inexpressive syntax.
So, Elixir, in fact, is the same Erlang with all its power, but with a much more pleasant and convenient syntax from Ruby. In Xeteq, we built a productive MTA on it (approx: mail transfer agent, i.e., a mail server). However, I am still afraid to do large-scale projects on Elixir. When using it, you have to write too much yourself: what has been implemented in other languages in the form of ready-made libraries for a long time, you have to do it yourself. Although sometimes it is useful to “reinvent the wheel”, because you can invent something new and worthwhile. And the community is kind of sluggish. All projects have a very small community, for example, let's take the basic web framework plug - it has less than 2 thousand stars, its analog is python flask, which has almost 40 thousand, and ruby sinatra has a little more than 10 thousand.
- Let's talk about the projects. Tell us about your most interesting, significant and important projects, and what is their fate.
- My first serious project was Blonde.ru. In 2009 it was a very ambitious project, the essence of which was that each person, not having special knowledge in contextual advertising, can independently launch an advertising campaign, and simultaneously in all advertising networks and from one simple interface. It was there that I grew very professionally and met people who greatly influenced my life. It so happened that after a few months, the technical director’s chair was vacant, and while they were looking for it, I was appointed by the IO. Some time passed, the chair was empty. The head of the company, Yura Yeliseyev, took and made me a full-fledged technical head, I don’t know what he was guided by, I was pretty green for this position, although maybe I saw something in me ( laughs - author’s comment ). By the way, it was there that we met with Alexander Shvets (we founded Digital God with him). He came to us to the position of a programmer while still a student of Baumanka and turned out to be the best in this. In general, we then became very good friends. The project is alive, it was renamed and transformed several times, it entered foreign markets, now it is known as Aori.
- And after Blonde.ru you started doing your first project, right?
- Yes, my friend and I decided to make a commodity widget, in fact, the Yandex.Market advertising network. To make it clearer: there is Yandex.Direct, and there is YAN (Yandex advertising network) - different sites where their blocks are located, and they either catch up with the person through a search, or thematically the show takes place. Yandex.Market didn’t have such a thing, but it had a special content API through which it was possible to receive hot offers from stores and a link where to send those interested. And if the purchase was made, then we were given 50% of the cost that the advertiser paid. Quickly enough, I wrote down the engine - a widget that is placed on the site, it determines the subject, looks what people were interested in, to make an automatic selection of goods from the Market. It worked great. We have learned how to arbitrate well (“arbitrage” - buy an advertising space and make it profitable at the expense of another channel), identify good / bad websites, began to actively buy advertising space on many websites and place your widget there. Among the sites was a major automotive. As a result, we had a very fast growth, but, for some strange reason, Yandex unilaterally broke the contract with us. We tried in every way to find out the reason, but this was never possible, we had to close the project. It was a great sadness, because a lot of energy was invested in this project, and the indicators were very good. Now I’m sure that it was a good experience that taught me how to correctly assess risks. In general, the experience is only positive, and if something does not succeed, it will only help you in the future to avoid even greater difficulties.
- And yet, gaining all sorts of experience, you continued to invent new projects. What happened next?
- Somehow I set myself the goal of making my own counter to collect data about the behavior of people on websites and calculate statistics based on this - like how Google Analytics and Yandex.Metrica collect their data. The code is put on the site, it keeps track of all visitors' actions on the site, transmits to the server, the server side receives this data and stores it in a distributed database. This project is called Rockstat. Later, I thought about making end-to-end analytics service on its basis, which, by the way, is now the pain of many companies. But I didn’t have the energy or time to develop, and I didn’t even begin to start. And it’s good that I didn’t, now I have the opinion that the end-to-end analytics services are doing their job. They are suitable for those who do not understand analytics and marketing, who do not have an understanding of how everything should work, or those who have low needs. In other cases, they run into a lack of integration or incompatibility of work logic. But I found an interesting use for raw data collected by the meter.
- What?
- For example, consider multi-channel attribution based on the behavior of the site visitor, and not on the number of the visit. Attribution is a way to distribute the value of the sale between the advertising channels that participated in it. Earlier, the attribution model based on the probabilistic passing of a funnel has become popular, but it is suitable only for sites with a funnel and a not too long sales cycle - these are most online stores. But there is another side: nedvizhka, avtoshka, wedding dresses, where the cycle of decision making is much longer, and there is simply no adequate funnel on the site. For them, I thought up to calculate the index of activity / involvement behavior. At that time, my counter already had several users, this allowed to test the hypothesis, calculate attribution based on the involvement of users in interaction with the site. I took two sites where there is a funnel and not a very short sales cycle. Compared the results of the model based on the probabilistic passage of the funnel and the model based on involvement, they turned out to be significantly correlated. After all, we live in such a time: if the content of the page does not cling or does not give hope, then it will immediately be closed.
I thought that this could be useful and in demand. He walked through several companies, told how they could improve the method of evaluating the effectiveness of advertising channels, but it turned out to be too difficult ... It is difficult to understand and adds difficulty in justifying costs. In general, there are very few people who want to do their work really cool, who are driven by what they are doing and who are eager to take new peaks. More often it turns out that, encountering a difficult and incomprehensible decision, they would rather score on it, they will not complicate their lives. He spoke at Data Science Week, Moscow Digital with a story about this model. There were many interested, but no one has matured.
And then there was another turn, some guys came to me to talk about the analyst (I thought so), but it turned out they came to offer me a job, and for a very unusual position of marketing director for me; and not somewhere there, but to the promising daughter of Sberbank Plazius. This experience deserves a separate story, so perhaps I will miss it.
While still working at Plazius, I began to teach. Together with a friend, together with the Tceh platform, we launched a full-time educational course “Programmatic Advertising Specialist”. The course helped people to increase their competence in the difficult field of high-tech advertising, or even get a new profession. And then I ended up at Linkprofit, where I started creating marketing technologies that were primarily focused on the market, and not on meeting the needs of the company. Then the company was in the process of restructuring, without even noticing it, I was appointed head of the new IT company Xeteq, which was included in the LT Digital group of companies.
- Wait, but what about the Rockstat? Abandoned him, since no one understood?
- Once I thought: “Why does Rockstat go dead?” Maybe someone will benefit. " Redid the service in the box, added data processing tools and laid out in open source called Alcolytics. And then it started ... A community of very professional guys, whom I used to meet quite rarely, began to gather around the project. They began to implement it at home. I was so amazed! And very competently approached how to use it, I didn’t even have such thoughts. Then I decided for myself that I would only work with enthusiastic and professional people, and let the rest go with the forest, I would not even get involved with those who do not know what they want, who should “sell” the idea and then prove its effectiveness.
Almost all my free time I began to spend on the development of this project, I was very motivated by the apparent return. “What the hell, I used to want to commercialize everything, if I'm preoccupied with the fact that they use it, and people like it,” I thought. In general, at first it was somehow scary, all of a sudden a bug, but I can’t even fix it or will install it wrong, they will spoil everything, and then they will say that the software does not work (* laughs - author’s comment *). I knew about a dozen servers with Alcolytics, probably there were more of them, but this knowledge was enough for high motivation, I constantly modified the project, and it got better and better.
At one user in Alcolytics data from crm and about calls was added, a through analytics was built. These data were needed to build clever cross-site attribution. This had to be done outside, because There was no way to run the code inside. Then I decided to make it possible to create small scripts to develop my API-NIS, which will allow to take this data into crm or else where, where they should be used. He worked many options, but everywhere there was some flaw. I decided to make a mock service for Docker, but I got so carried away with this process that after two months I realized that I had made my microservice platform, and in fact I wanted to do a simple thing for running scripts. I was very fascinated by the thought: “Why do we need analytics if there is no possibility to automate decision making?” It is enough to look several times at the numbers, and it is clear in which case what to do. This allows you to do either commercial software, usually having a decent price, or difficult, in terms of implementation and support, open source.
- Yes, simple solutions are not for you, if you do something, then in a big way.
- I imagined how many different automations can be implemented. They will perform various tasks: manage the advertising budget and rates, or, if we talk about RTB advertising, it will be possible to shift users from one segment to another if they change the status in the system. You can do whatever you want. In the process of working on this platform, I invented many “bicycles”, I had never invented so much before. Nobody will praise me for it, but I like them and they turned out very cool. There is another very good argument - it all works on one server for $ 10, keeping the site with 1-2 million visitors per month, without straining at all. Developed frameworks for 2 programming languages that implement all the necessary functionality, organizing interaction with other microservices. By the way, now huge companies are built around microservice technologies.
And still, when you make an open source solution, you are not ashamed to take a piece from another open source, for example, mobile clients, for the implementation of which a lot of time would be spent. In commerce, of course, this is impossible - these are huge reputational risks, and indeed a disgrace. I learned so much cool about open source when I started working with it.
Now this is not a project that solves specific problems, it is a platform that helps to realize your desires, and it is easy to do it, do not bother with complex and boring tech-nar things.
Recently released in alpha testing the 3rd version , returning the original name Rockstat . There is little documentation, many bugs, but users are happy. There are big companies among them: one airline, and an advertiser from the top-30 of Russia on the queue. Of course, there are a lot more users. We have a small community in TG (@rockstats), the guys will report on problems and achievements.
- Why and why so much time and personal money you spend on the project, which then spread in open source. What drives you?
- It seems to me that the Rockstat project can greatly change the sphere of digital. It is clear that a standard is suitable for small companies, and giants are building complex solutions, but there are companies with talented specialists and complex needs somewhere in the middle, they are becoming more and more. They create bold and effective solutions with their own hands. The market will follow them or not, it is not known, because we know what Russia is famous for. For some time I will develop the project here, but only he will mature a little, I will try to announce it in other markets.
It was a long way, but now I can honestly say: “I don’t know why I need money, where to put it.” I don’t know why I once thought about money, probably it was imposed by society. Even before open source, I began to teach, I liked to share my knowledge, although at first I thought: “Shouldn’t I have to keep some knowledge with me?” But no, you need to share with the community, there are many worthy people that your knowledge can be helpful. And the most important thing is that it comes back.
- And what interesting and cool projects did you do as part of your work?
- Oh, I have all such ( laughs - author's note ). Now there are several projects in parallel. I will tell, for example, about email service. Mailing lists are already quite an adult direction, there are a lot of players, both local and foreign. However, this niche is not occupied: the players do not have enough opportunities for segmentation, the very high cost of mailings, they are all from the 00s, ineffective. Honestly, I did not want to take on this task for a long time. As a result, I decided to start by creating my own mail server - MTA (Message Transfer Agent) - with the minimum necessary functionality. This is still a great job. For 2-3 months of work, we received an MPV (working prototype). In a single user action, multiple servers may start, and a sending system will roll over them. Letters flow like water - it can scale to huge volumes. Resource consumption is minimal, you can send at least several billion letters a day, and the system will not even notice it. But it’s not so easy to send a billion letters - you need to “warm up” the servers, gradually raising the number of letters sent. I planned to make this project on Elixir, though, personally, I didn’t start implementing, but the guys supported the idea, mastered Elixir and did.
- You have such a great experience, you know a lot of things and know how, tell me why you need the program “Big Data Specialist” ? What was the need?
- At that time, the largest projects were limited to a cluster of several servers from a relational database. About Hadoop ecosystem and machine learning did not know much. The course gave me useful knowledge, I remember the whole theory, I know where to dig if anything is required; Immediately I can answer, than models on trees differ from linear and gradients. In general, there were so many things there: both the use of Hadoop, and recommender systems, I still scare colleagues with the phrase “collaborative filtering”, and about building DMP, which was especially important for me - at that time I was working on a similar project. Now we have to work with data a lot: using machine learning to find patterns in the data or to calculate the effectiveness of advertising; determine the values of microconversions on sites. In general, the knowledge gained has found many applications and I am sure there will be even more. This is already real, without this in any way.
- Let's talk about Digital God - your educational project. How did it come about and what are you teaching?
- This project is out of work, of course, but the need arose just from work: I decided to find programmers in a cunning way (this is a big problem), make courses for programmers and teach them to Ad- / Mar-Tech. To test the hypothesis, I made a post on Facebook, but the programmers did not respond. Digital professionals responded. And it was decided to go this route: to teach digital programming specialists. It turned out that it is much more interesting and useful because programmers are given marketing and analytical specifics, but they are not bad at all, and these guys have suffered: they write TK for a week, then wait for weeks for them to do everything.
My friend (Alexander Shvets) worked the course, launched it, and from the very first it worked - we have a third of the participants survived ( laughs - author's note ). They were very tired and tortured. They had a hard time, they had to sit all weekend long, but they got a huge professional growth and opportunities. Now the 2nd stream is coming to an end, and the 3rd is approaching the equator. More than half of the people who come to us know that they need to learn programming. They are already experiencing severe difficulties due to the lack of the possibility of free work with data.
Code is freedom, it is, in essence, a speech for computers. And ready services, applications are restrictions of your opportunities by the interface and the logic developed under the majority. You can do almost anything with code. You need to be thoroughly remarked or think about religion in order to understand what is not decided by programming. And in general, coding is not difficult, because the code is very similar to ordinary English speech. You do not need to be afraid, you have to start learning how to write code, you cannot get away from it.
Rockstat turned out to be very useful here: we do not start from the basics, but immediately proceed to solving problems that we can face every day. In the second lesson, for example, we solve the problem of classifying visitors to a site according to their behavior, understanding which visitor they wanted, having their history. It is used dataset from sites of strip clubs (they are also users of Rockstat and kindly provided a cut of anonymized data). And they found who visited the site for the purpose of employment, and who is interested only in a visit to the club. By the end of the program, everyone should make a course project, according to which we will make a decision on the success of the course. Someone creates his own API, someone considers cross-cutting analytics, one comrade implemented scraper-s for collecting data that cannot be obtained by the API.
Rockstat and Digital God are two major serious projects outside of work. But there is still the most beloved - "Your sweet" . Somehow a friend brought a small caramel in the form of manhood. “Pff, nonsense,” I thought, and threw her in the kitchen, where she lain for about a week. At some point I was bored and I decided to make a candy as large as it should be. Once in that period I was not working, there was a lot of time, and I was confused. In the meantime, I did, I realized that there was no reason for me to keep this valuable knowledge with me, I had to share it with others. I had a lot of fun doing it, and after a lot of trial and error, a box appeared with all the necessary components and instructions on how to make everything yours.
Issue 1 of the Digital God stream course Digital Rockstar
- Let's go back to the educational project. Tell me, what skills does a graduate of Digital God have? What can he do?
- The most valuable thing is that the participants got their freedom, they can now describe their tasks in the computer language, and he will take care of the decision, and will do it on a schedule that will lay down or even process everything in real time. I think the most interesting conclusions will be in a few months, when graduates will gain more experience.
In addition to digital specialists, we also had owners of small businesses that were involved in digital, because in the modern world everything is very tied to the indicators of the effectiveness of advertising and websites, and it’s very hard to find an advertising agency that can adequately take on it, and no automation they will not do.
- Why do you not do in principle corporate programs, despite the incoming proposals?
- Knowledge should not be given exclusively to any one big company, besides, according to my observations, the interest of corporate participants is usually lower. Therefore, I don’t want to come to a course for people who are forcibly sent to study. Maybe I'm wrong. If you really need someone, they can take a course with employees of other companies, but, most importantly, they will be in a group of interested people, the atmosphere and impact will be completely different.
- Starting from Blonde.ru, where you quickly grew to a technical director, you always had commands in submission. Please tell us about your approach to work and people in the team.
- Yes, it turns out that most of my professional life I occupy leadership positions, but in my heart I am still a free artist: I need freedom from routine and bureaucracy, this allows me to think freely and find good solutions. I do not like management, for me it is depressing, but there is no way to go. I am not a team leader in the team, now Sasha (Alexander Shvets, a longtime friend and co-founder of DigitalGod) manages the team and so it’s all better. , : « , . \, ». , , , . , , , .
, IT : - , , , , . , , , , , , . , , .
— , ? , / ?
— , , , , . , - , .
, , . , , . - , , , . , . , , , - , , , .
— , ? , .
— ( – . ). , , , . , . , , - .
— , ?
— - , . — , , , . open source, , , Digital God digital . , . , , – , , , …
Source: https://habr.com/ru/post/430364/
All Articles