Visiting Odnoklassniki
Recently, I managed to visit the Riga office of the Odnoklassniki company - I will tell about the office itself a little later, and in this post I want to retell conversations with a number of company employees. I'll start with Elina Parkhomenko , she is the director of product development.
Elina and I met in the videoconference hall. At the very beginning of the conversation, she focused on the fact that she does not participate in deeply systemic things - she is committed to setting the vectors for the development of the project, determining the work of those or other services, and managing development. However, Habr's audience didn't ask any technical questions anyway :)
')
Honestly, I almost don’t have time for “meetings” on social networks, but before flying out, I visited the site and was very surprised at how it has changed since I used it before. The site has become more sophisticated, “overgrown” with all sorts of chips - what was the path to such solutions?
Elina : I came to work at the company just when everything was “simple” on the site, but we must understand that without complicating the interface, the emergence of innovations is often simply impossible. In general, we try to maintain the course of the project and the overall concept - there are several aspects.
The first is the social graph. Everything that is not done is done for the sake of users, for their interaction with each other, for strengthening ties within the graph. Here it is important not only the development of the graph, but also its growth - the priority is to attract new people from outside. People are the most important thing.
The second is that these people need to be fun and interesting, so that they have an incentive to enter the project.
Accordingly, all services are developed from these two considerations.
What do you think keeps people on the site?
Elina : As practice shows, users are most in demand at one-on-one conversations with other people, less often - communication in groups. And in this regard, we are trying to make it so convenient for users to communicate in one form or another - here you will receive both private messages and discussions, “guests” and even proprietary “ratings” (a favorite function of users), which often lay the foundation for new dating These are the functions of the site that we are trying to improve and make more convenient - recently, for example, they made the messages “instantaneous” so that users receive messages immediately (and, accordingly, remain on the site) rather than after the fact. From the same "speed" considerations, we made a toolbar, which is always shown in the upper part of the browser - a kind of user dashboard.
The same with the content part of the project - people upload photos for the sake of feedback (for example, comments and ratings), so it was logical to make functional uploading of photos and the possibility for their subsequent discussion, without any restrictions.
That is, by and large, we help to realize the natural need for communication. And when users are comfortable and interesting, they remain on the site.
How do you imagine the audience of the project when developing this or that service?
Elina : It is very important for us that the service is clear even to the simplest person, not to mention the advanced users. Therefore, all the services that we launch always have basic functionality that is understandable to absolutely everyone. This is already in the future, he "acquires ryushki" for narrower audiences, but initially - all for all.
Although, as a rule, we do not make radical changes and we haven’t yet had stories like “return the wall”. Therefore, users are pretty calm about all innovations.
By the way, when we launch a new service, at the testing stage we have a special button “share sensations” - any user can use it to express their opinion on any aspect of innovation. All calls are tracked and taken into account.
And they write a lot?
Elina : If you don't like something, then a lot. But on the contrary, this is good - constructive criticism is not superfluous and we always listen to it. There are those who are dissatisfied with absolutely everything, and sometimes vice versa - they love us whatever we are.
I heard that over the summer you had more than 50 various releases - how does the process of creating new functions of the site occur? If someone came up with something interesting (going beyond the current course of development of the project) - what will be the fate of his idea?
Elina : We have several managers, each of which, as a rule, has 4-5 interconnected (ideologically or functionally) projects. Under the creation of each innovation is going to group - taken by programmers, designers, testers. When the team is formed, the teamwork begins: the idea is discussed, finalized, visualized and begins to be realized.
If the innovation is really interesting, we always find him a place in the development plan of the project. Anyone can offer a function - a user, a technical support employee, or even an administrator.
How is testing new services? Do you collect any statistics?
Elina : At first, everything starts on a virtual machine - by common efforts of the company, we test the strength of the service, look for bugs, fix them, share our impressions. Make the appropriate amendments.
Closer to the release joins the test group to test the load and perception of the product. And only then - on the production. Sometimes innovations are launched as an experiment, that is, for a small part of the audience or for the whole, but for a short time. After that, we carefully study the statistics, loads, reactions in the support service. If there are no problems, then we launch it definitively - the system is very “alive”, so it is important not to break its components or the connections between them. Later we monitor the effectiveness of services.
What kind of violations you have to face most often with spam? How do you fight?
Elina : Now we have very little spam now - either we really won it, or THEY are still on vacation;) But unfortunately, we cannot tell you about the methods of dealing with spam.
The most frequent violation is all sorts of groups that spread something harmful. But we managed to defeat them with our response mechanism - a symbiosis of user moderation and the developed algorithms. Now such groups simply do not have time to "bloom."
That is, you have no army moderators?
Elina : Right. Social responsibility for Odnoklassniki means a lot - we have, of course, moderators, but they simply wouldn’t pull the whole amount of work. In this issue, the users themselves, who possess simply unique responsiveness, come to the rescue - apparently, people want their “home” to be cleaner. So, for example, all uploaded photos are sent to user moderation for the Nth number of people who are very likely to sift out all “superfluous” - now all of our (public) photos are moderated. Popular moderation is a very important component on our project and in the photos section it works like a clock.
And how many users are involved in this?
Elina : More than 5 million. Imagine if this job was done by several moderators ...
Surely on your way there were “fakapy” or vice versa, “surprises of good luck” - share some story?
Elina : The story about cancellation of paid registration comes to mind when we compensated all the users who paid for it for free with the amount spent by the opportunity to rate “5+”.
Then the "wave" covered even undistributed questionnaires - users began to frantically rush through the site, visiting each other and putting the highest score right and left, literally everyone. Of course, we expected to increase activity, but after an hour due to high load, we had to withdraw a “freebie” from a number of users and began to distribute it in portions, with a certain interval. What, as you understand, caused an explosion of applications in the support service: “Why did they give him fives, but not me ?!”.
And by the way, what was the main reason for entering paid registration? Karma fell victim to commonplace monetization or was it a fight against spam?
Elina : Before that, there was a lot of spam, fake profiles, bots and other fraud - no one knew how to effectively deal with it. After the registration was closed, we invented and implemented a number of algorithms that allowed us to get rid of unscrupulous accounts. And, of course, they came up with a lot of innovations so that the newcomers would be interested in us - all this after the opening of the registration ensured the intensive growth of the project. Now we have more than 102 million "confirmed" (by phone) grateful users who can safely communicate with each other.
Statistics somewhere you can see?
Elina : Yes, we are one of the few who have statistics open on the same LiveInternet.ru. An average of 20-23 million visitors per day, most of which come from the "bookmarks".
Tell us a few words about the life of your employees?
Elina : Everybody’s working day starts at different times - usually the people working in a group agree on what time they arrive at the office. As a rule, it is 10 am, although UI-Schnick come at 8.
About 100 people work in the Riga office, 25 of whom are programmers. The youngest employee (he is a developer) recently turned 20 years old, the most adult (manager and designer) almost 40; the average age is up to 30. Everyone speaks Russian.
Internal culture is not much different from other companies, although there are some features. So, for example, an initiative comes from our employees to celebrate any events - whether it be a car purchase, a birthday, or the end of a trial period. The trial period, by the way, is 3 months, but if we see that a person is good and does his job, then we don’t wait.
Are there any female developers?
Elina : Oh, where, where, and we have, not one yet!
What, directly write the code? What language do they write in?
Elina : Like everything in Java! By the way, there are a lot of female testers, and they very well cope with the performance of their task.
And how is the integration of new people into the company?
Elina : Regardless of which office (Riga, Moscow or St. Petersburg) is meant - the newcomers undergo a kind of training. Various specialists (as a rule - leading developers, backend programmers and UI-designers) tell the important features of the project - about the basic principles, the architect, the design - we don’t throw anyone under the tanks. Moreover, such training can be carried out between offices - we constantly fly to visit each other.
Interesting. And finally - can you tell us something of your plans for the future?
Elina : I can’t tell you anything concrete - while we are working on optimizing the project so that everything works for everyone and quickly. We are developing a new mobile application. Well, in a relatively short time we plan to make a complete redesign.
After the conversation with Elina, we met with Andrey, the system administrator of the project - he told us about the changes that had occurred since the March publication. If you missed - I advise you to read it, if you have forgotten - refresh it in your memory; most interesting article - Architecture and Odnoklassniki project platform
Major changes (>> was / - become):
>> Up to 2.8 million users online during peak hours
- Now this figure has increased and is about 3.4 million.
>> 7.5 billion requests per day (150,000 requests per second during peak hours)
In billions from memory I will not say, but now the number of requests per second has increased to about 250 thousand.
>> Network traffic during peak hours: 32 Gb / s
At the moment, this figure has grown to 52 Gb / s
>> 2,400 servers, storage systems
Now the number of servers (and storage devices) has exceeded 3,100. Moreover, if there were three data centers, now there are four - all are located in Moscow. The channels between the data centers have also been expanded.
Have the server configurations been updated?
Andrew : The servers themselves did not update, increased their number. The average configuration is a dual-socket system with quad-core processors, 1-2 disks and a memory capacity of 8GB. The concept of backup has not changed.
Began to use more memory and more cheap iron. Especially for storing binary data, since they launched a new software solution (they were told about it in detail at Highload).
From memory: now a maximum of 96GB on the system that serves the user tape. We install and test 128-192GB on the same system.
Regarding the question about NO-SQL:
1. Recently launched a cluster on “Cassandra” for photo rating service.
2. Launched " Tarantool " + "Voldemort" for the messaging system, where Tarantool is in memory storage, Voldemort is a transport, replication, cluster.
Caches: as a cache, a Java solution is used. The data is stored outside of the Java Heap using the “sun.misc.Unsafe” tool, which avoids problems with the GC.
The next employee of the company, with whom we managed to talk, was the girl Irena , she is the head of technical support service.
Until recently, the caliper received more than 25 thousand hits per day - a huge amount of work. Moreover, 70-80% of the total number of applications were problems with the password - when someone forgot something and can not remember. For comparison, on Habré now about 150 applications per day, but there almost all questions require an individual approach.
Reorganization of help and writing a detailed FAQ allowed to reduce the number of requests by almost 5 times, while the number of "password recovery" has increased. Now the volume of work is 7-8 thousand applications per day, which is also quite a lot - all the problems of users are solved by about 30 employees of the company, divided into two "lines". The first line (7-8 employees working at the same time) is located in Ivanovo and takes the brunt of it (applications that can be answered with a pattern), the second line is located in Riga and is responsible for all other applications.
It is noteworthy that among the employees there is a person (girl) who deals exclusively with ... pornography. More precisely, her removal from the resource - yes, it's about the very social responsibility that Elina was talking about. As I understood from the animated conversation, only lazy did not joke about this topic ... but agree that this girl (and maybe not even one) of Fridays could hold an interesting presentation about her not very dusty work :)
Tired but satisfied, we flew home. When I climbed on Habr, there was a desire to rewrite all the information on the site, which I would be able to facilitate the work of our support service and help new users to get started - I will try to do this in the near future.
To be continued.
Elina and I met in the videoconference hall. At the very beginning of the conversation, she focused on the fact that she does not participate in deeply systemic things - she is committed to setting the vectors for the development of the project, determining the work of those or other services, and managing development. However, Habr's audience didn't ask any technical questions anyway :)
')
Honestly, I almost don’t have time for “meetings” on social networks, but before flying out, I visited the site and was very surprised at how it has changed since I used it before. The site has become more sophisticated, “overgrown” with all sorts of chips - what was the path to such solutions?Elina : I came to work at the company just when everything was “simple” on the site, but we must understand that without complicating the interface, the emergence of innovations is often simply impossible. In general, we try to maintain the course of the project and the overall concept - there are several aspects.
The first is the social graph. Everything that is not done is done for the sake of users, for their interaction with each other, for strengthening ties within the graph. Here it is important not only the development of the graph, but also its growth - the priority is to attract new people from outside. People are the most important thing.
The second is that these people need to be fun and interesting, so that they have an incentive to enter the project.
Accordingly, all services are developed from these two considerations.
What do you think keeps people on the site?Elina : As practice shows, users are most in demand at one-on-one conversations with other people, less often - communication in groups. And in this regard, we are trying to make it so convenient for users to communicate in one form or another - here you will receive both private messages and discussions, “guests” and even proprietary “ratings” (a favorite function of users), which often lay the foundation for new dating These are the functions of the site that we are trying to improve and make more convenient - recently, for example, they made the messages “instantaneous” so that users receive messages immediately (and, accordingly, remain on the site) rather than after the fact. From the same "speed" considerations, we made a toolbar, which is always shown in the upper part of the browser - a kind of user dashboard.
The same with the content part of the project - people upload photos for the sake of feedback (for example, comments and ratings), so it was logical to make functional uploading of photos and the possibility for their subsequent discussion, without any restrictions.
That is, by and large, we help to realize the natural need for communication. And when users are comfortable and interesting, they remain on the site.
How do you imagine the audience of the project when developing this or that service?Elina : It is very important for us that the service is clear even to the simplest person, not to mention the advanced users. Therefore, all the services that we launch always have basic functionality that is understandable to absolutely everyone. This is already in the future, he "acquires ryushki" for narrower audiences, but initially - all for all.
Although, as a rule, we do not make radical changes and we haven’t yet had stories like “return the wall”. Therefore, users are pretty calm about all innovations.
By the way, when we launch a new service, at the testing stage we have a special button “share sensations” - any user can use it to express their opinion on any aspect of innovation. All calls are tracked and taken into account.
And they write a lot?
Elina : If you don't like something, then a lot. But on the contrary, this is good - constructive criticism is not superfluous and we always listen to it. There are those who are dissatisfied with absolutely everything, and sometimes vice versa - they love us whatever we are.
I heard that over the summer you had more than 50 various releases - how does the process of creating new functions of the site occur? If someone came up with something interesting (going beyond the current course of development of the project) - what will be the fate of his idea?Elina : We have several managers, each of which, as a rule, has 4-5 interconnected (ideologically or functionally) projects. Under the creation of each innovation is going to group - taken by programmers, designers, testers. When the team is formed, the teamwork begins: the idea is discussed, finalized, visualized and begins to be realized.
If the innovation is really interesting, we always find him a place in the development plan of the project. Anyone can offer a function - a user, a technical support employee, or even an administrator.
How is testing new services? Do you collect any statistics?Elina : At first, everything starts on a virtual machine - by common efforts of the company, we test the strength of the service, look for bugs, fix them, share our impressions. Make the appropriate amendments.
Closer to the release joins the test group to test the load and perception of the product. And only then - on the production. Sometimes innovations are launched as an experiment, that is, for a small part of the audience or for the whole, but for a short time. After that, we carefully study the statistics, loads, reactions in the support service. If there are no problems, then we launch it definitively - the system is very “alive”, so it is important not to break its components or the connections between them. Later we monitor the effectiveness of services.
What kind of violations you have to face most often with spam? How do you fight?Elina : Now we have very little spam now - either we really won it, or THEY are still on vacation;) But unfortunately, we cannot tell you about the methods of dealing with spam.
The most frequent violation is all sorts of groups that spread something harmful. But we managed to defeat them with our response mechanism - a symbiosis of user moderation and the developed algorithms. Now such groups simply do not have time to "bloom."
That is, you have no army moderators?Elina : Right. Social responsibility for Odnoklassniki means a lot - we have, of course, moderators, but they simply wouldn’t pull the whole amount of work. In this issue, the users themselves, who possess simply unique responsiveness, come to the rescue - apparently, people want their “home” to be cleaner. So, for example, all uploaded photos are sent to user moderation for the Nth number of people who are very likely to sift out all “superfluous” - now all of our (public) photos are moderated. Popular moderation is a very important component on our project and in the photos section it works like a clock.
And how many users are involved in this?Elina : More than 5 million. Imagine if this job was done by several moderators ...
Surely on your way there were “fakapy” or vice versa, “surprises of good luck” - share some story?Elina : The story about cancellation of paid registration comes to mind when we compensated all the users who paid for it for free with the amount spent by the opportunity to rate “5+”.
Then the "wave" covered even undistributed questionnaires - users began to frantically rush through the site, visiting each other and putting the highest score right and left, literally everyone. Of course, we expected to increase activity, but after an hour due to high load, we had to withdraw a “freebie” from a number of users and began to distribute it in portions, with a certain interval. What, as you understand, caused an explosion of applications in the support service: “Why did they give him fives, but not me ?!”.
And by the way, what was the main reason for entering paid registration? Karma fell victim to commonplace monetization or was it a fight against spam?Elina : Before that, there was a lot of spam, fake profiles, bots and other fraud - no one knew how to effectively deal with it. After the registration was closed, we invented and implemented a number of algorithms that allowed us to get rid of unscrupulous accounts. And, of course, they came up with a lot of innovations so that the newcomers would be interested in us - all this after the opening of the registration ensured the intensive growth of the project. Now we have more than 102 million "confirmed" (by phone) grateful users who can safely communicate with each other.
Statistics somewhere you can see?Elina : Yes, we are one of the few who have statistics open on the same LiveInternet.ru. An average of 20-23 million visitors per day, most of which come from the "bookmarks".
Tell us a few words about the life of your employees?Elina : Everybody’s working day starts at different times - usually the people working in a group agree on what time they arrive at the office. As a rule, it is 10 am, although UI-Schnick come at 8.
About 100 people work in the Riga office, 25 of whom are programmers. The youngest employee (he is a developer) recently turned 20 years old, the most adult (manager and designer) almost 40; the average age is up to 30. Everyone speaks Russian.
Internal culture is not much different from other companies, although there are some features. So, for example, an initiative comes from our employees to celebrate any events - whether it be a car purchase, a birthday, or the end of a trial period. The trial period, by the way, is 3 months, but if we see that a person is good and does his job, then we don’t wait.
Are there any female developers?Elina : Oh, where, where, and we have, not one yet!
What, directly write the code? What language do they write in?Elina : Like everything in Java! By the way, there are a lot of female testers, and they very well cope with the performance of their task.
And how is the integration of new people into the company?Elina : Regardless of which office (Riga, Moscow or St. Petersburg) is meant - the newcomers undergo a kind of training. Various specialists (as a rule - leading developers, backend programmers and UI-designers) tell the important features of the project - about the basic principles, the architect, the design - we don’t throw anyone under the tanks. Moreover, such training can be carried out between offices - we constantly fly to visit each other.
Interesting. And finally - can you tell us something of your plans for the future?Elina : I can’t tell you anything concrete - while we are working on optimizing the project so that everything works for everyone and quickly. We are developing a new mobile application. Well, in a relatively short time we plan to make a complete redesign.
Architecture
After the conversation with Elina, we met with Andrey, the system administrator of the project - he told us about the changes that had occurred since the March publication. If you missed - I advise you to read it, if you have forgotten - refresh it in your memory; most interesting article - Architecture and Odnoklassniki project platform
Major changes (>> was / - become):
>> Up to 2.8 million users online during peak hours
- Now this figure has increased and is about 3.4 million.
>> 7.5 billion requests per day (150,000 requests per second during peak hours)
In billions from memory I will not say, but now the number of requests per second has increased to about 250 thousand.
>> Network traffic during peak hours: 32 Gb / s
At the moment, this figure has grown to 52 Gb / s
>> 2,400 servers, storage systems
Now the number of servers (and storage devices) has exceeded 3,100. Moreover, if there were three data centers, now there are four - all are located in Moscow. The channels between the data centers have also been expanded.
Have the server configurations been updated?Andrew : The servers themselves did not update, increased their number. The average configuration is a dual-socket system with quad-core processors, 1-2 disks and a memory capacity of 8GB. The concept of backup has not changed.
Began to use more memory and more cheap iron. Especially for storing binary data, since they launched a new software solution (they were told about it in detail at Highload).
From memory: now a maximum of 96GB on the system that serves the user tape. We install and test 128-192GB on the same system.
Regarding the question about NO-SQL:
1. Recently launched a cluster on “Cassandra” for photo rating service.
2. Launched " Tarantool " + "Voldemort" for the messaging system, where Tarantool is in memory storage, Voldemort is a transport, replication, cluster.
Caches: as a cache, a Java solution is used. The data is stored outside of the Java Heap using the “sun.misc.Unsafe” tool, which avoids problems with the GC.
Support
The next employee of the company, with whom we managed to talk, was the girl Irena , she is the head of technical support service.
Until recently, the caliper received more than 25 thousand hits per day - a huge amount of work. Moreover, 70-80% of the total number of applications were problems with the password - when someone forgot something and can not remember. For comparison, on Habré now about 150 applications per day, but there almost all questions require an individual approach.
Reorganization of help and writing a detailed FAQ allowed to reduce the number of requests by almost 5 times, while the number of "password recovery" has increased. Now the volume of work is 7-8 thousand applications per day, which is also quite a lot - all the problems of users are solved by about 30 employees of the company, divided into two "lines". The first line (7-8 employees working at the same time) is located in Ivanovo and takes the brunt of it (applications that can be answered with a pattern), the second line is located in Riga and is responsible for all other applications.
It is noteworthy that among the employees there is a person (girl) who deals exclusively with ... pornography. More precisely, her removal from the resource - yes, it's about the very social responsibility that Elina was talking about. As I understood from the animated conversation, only lazy did not joke about this topic ... but agree that this girl (and maybe not even one) of Fridays could hold an interesting presentation about her not very dusty work :)
Tired but satisfied, we flew home. When I climbed on Habr, there was a desire to rewrite all the information on the site, which I would be able to facilitate the work of our support service and help new users to get started - I will try to do this in the near future.
To be continued.
Source: https://habr.com/ru/post/130546/
All Articles