How we test advertising technologies of Yandex, and how to learn it
My name is Konstantin Vlasov, and I supervise Yandex at the quality management service of the banner system. As you can guess, testing a rapidly changing system of such magnitude as our advertising technologies, with all databases, component replication, interaction protocols between subsystems, as well as fairly frequent releases, is not easy. At the same time, it is necessary to test everything very carefully, because any mistake can potentially cause serious damage to the company's business. And here it is not only about reputational costs, but also about the loss of real money.
Today I want to talk about this in a bit more detail, and for those who will be interested, I will invite you to learn how to test such complex systems and try all this in practice.
')
We begin, of course, with the story that we are testing. In Yandex, there are two advertising directions: contextual and display advertising.
Contextual advertising is text ads that you have probably seen in search results and on websites that are part of the Yandex advertising network; media is flash, gif, video advertising.
It would seem nothing complicated: he showed ads, wrote off money from advertisers and that’s all. But the advertising technology market is designed in such a way that it is unprofitable to show users random advertising. Advertisers want to show ads to precisely those users who may really find it useful and interesting. The effectiveness of an ad depends on its relevance.
Increasing relevance is a very difficult task, requiring a large amount of information about both the user’s behavior and many other criteria. Therefore, our banner system includes machine learning clusters, sophisticated ranking algorithms, huge databases and super fast web servers needed for instant data transfer.
We use Crypt technology to customize display ads. This is a machine learning method that analyzes the behavior of each user in the Yandex network, which allows you to find out the socio-demographic characteristics of users: age, gender, education, income level, interests - several hundred parameters, based on which we can show ads that are supposedly useful to a specific user . For contextual advertising, the main role is played by the search query that the user has just entered, as well as the query history. The location of the user and the time of the request are also taken into account.
How is the selection of ads for display in each case - the topic of a separate story. Some idea of this process can be drawn from the lecture of our colleague Mikhail Levin . It is only worth noting that the lecture provides a somewhat simplified model for calculating the probability of displaying an ad. In fact, in addition to the click-through rate (CTR) - the relationship of clicks to impressions - a lot of algorithms take part in the ranking, taking into account a huge number of factors and factors, the generation of which is helped by MatrixNet machine learning technology.
It is important to understand that the whole process of generating ad units must occur very quickly - just a few milliseconds. Naturally, all statistics are calculated in advance. However, all this data needs to be quickly retrieved and transferred between components of our system. For this we use the high-performance web server Phantom . And it all works under load, which can be roughly compared with the load on the main search engine in Yandex.
The system under test is a set of web services and a lot of scripts, various databases and interfaces, so because of the variety of components we place emphasis on testing automation. We build and support test automation frameworks ourselves. The main development languages in our team are Python, Perl.
We are constantly experimenting, introducing new approaches and technologies: Model-based testing , Pairwise testing , Continuous Integration , Continuous Delivery and others. The goal of these experiments is to increase our efficiency by optimizing processes and tools.
In testing the fast-changing system of such scale as our advertising technologies, with all the databases, component replication, interaction protocols between subsystems, as well as quite frequent releases, the complex infrastructure and Continuous Integration (CI) cannot be dispensed with. Now we have implemented a full CI cycle using Jenkins and OpenStack -cloud.
We have also developed several plug-ins for Jenkins, and also used a versatile tool for compiling custom Allure test reports, kindly provided by our colleagues.
We do not have a classic division into test designers, developing test cases, and automators. There is a lot of technical development in our work, so we can grow not only in terms of testing, but also in terms of technology. Everything is done with one sole purpose - to provide the user with a quality product. Work is stressful enough, so we try to create a pleasant and fun atmosphere in the workplace. Everyone has their own preferences: sports, music, video games, desktops, etc.
We are actively growing and developing, and our team needs new personnel, so the idea was born to create a Banner Testing System School - two-month courses that can provide the necessary knowledge and skills to work in our team.
Training includes not only the theoretical part, but also the practice of solving problems similar to combat. Lectures and practical classes will be delivered by specialists from our team. Students who have demonstrated success in the learning process will be invited to undergo an internship at Yandex. But even if you do not set goals to get a job at Yandex, our school is a great chance to learn how to test directly from people who develop this area. Our course can be divided into several topics:
More detailed information is available on the page of our school , you can also apply for admission. The set will last until 19:00 on February 21.
Today I want to talk about this in a bit more detail, and for those who will be interested, I will invite you to learn how to test such complex systems and try all this in practice.
')
We begin, of course, with the story that we are testing. In Yandex, there are two advertising directions: contextual and display advertising.
Contextual advertising is text ads that you have probably seen in search results and on websites that are part of the Yandex advertising network; media is flash, gif, video advertising.
It would seem nothing complicated: he showed ads, wrote off money from advertisers and that’s all. But the advertising technology market is designed in such a way that it is unprofitable to show users random advertising. Advertisers want to show ads to precisely those users who may really find it useful and interesting. The effectiveness of an ad depends on its relevance.
Increasing relevance is a very difficult task, requiring a large amount of information about both the user’s behavior and many other criteria. Therefore, our banner system includes machine learning clusters, sophisticated ranking algorithms, huge databases and super fast web servers needed for instant data transfer.
We use Crypt technology to customize display ads. This is a machine learning method that analyzes the behavior of each user in the Yandex network, which allows you to find out the socio-demographic characteristics of users: age, gender, education, income level, interests - several hundred parameters, based on which we can show ads that are supposedly useful to a specific user . For contextual advertising, the main role is played by the search query that the user has just entered, as well as the query history. The location of the user and the time of the request are also taken into account.
How is the selection of ads for display in each case - the topic of a separate story. Some idea of this process can be drawn from the lecture of our colleague Mikhail Levin . It is only worth noting that the lecture provides a somewhat simplified model for calculating the probability of displaying an ad. In fact, in addition to the click-through rate (CTR) - the relationship of clicks to impressions - a lot of algorithms take part in the ranking, taking into account a huge number of factors and factors, the generation of which is helped by MatrixNet machine learning technology.
It is important to understand that the whole process of generating ad units must occur very quickly - just a few milliseconds. Naturally, all statistics are calculated in advance. However, all this data needs to be quickly retrieved and transferred between components of our system. For this we use the high-performance web server Phantom . And it all works under load, which can be roughly compared with the load on the main search engine in Yandex.
Testing
The system under test is a set of web services and a lot of scripts, various databases and interfaces, so because of the variety of components we place emphasis on testing automation. We build and support test automation frameworks ourselves. The main development languages in our team are Python, Perl.
We are constantly experimenting, introducing new approaches and technologies: Model-based testing , Pairwise testing , Continuous Integration , Continuous Delivery and others. The goal of these experiments is to increase our efficiency by optimizing processes and tools.
In testing the fast-changing system of such scale as our advertising technologies, with all the databases, component replication, interaction protocols between subsystems, as well as quite frequent releases, the complex infrastructure and Continuous Integration (CI) cannot be dispensed with. Now we have implemented a full CI cycle using Jenkins and OpenStack -cloud.
We have also developed several plug-ins for Jenkins, and also used a versatile tool for compiling custom Allure test reports, kindly provided by our colleagues.
All these developments are now publicly available on Github.
github.com/allure-framework/allure-core
github.com/allure-framework/allure-jenkins-plugin
github.com/allure-framework/allure-python
github.com/yandex-qatools/builders
github.com/jenkinsci/jclouds-plugin (this is not our development, but we are actively developing it)
github.com/jenkinsci/debian-package-builder-plugin
github.com/allure-framework/allure-jenkins-plugin
github.com/allure-framework/allure-python
github.com/yandex-qatools/builders
github.com/jenkinsci/jclouds-plugin (this is not our development, but we are actively developing it)
github.com/jenkinsci/debian-package-builder-plugin
Team
We do not have a classic division into test designers, developing test cases, and automators. There is a lot of technical development in our work, so we can grow not only in terms of testing, but also in terms of technology. Everything is done with one sole purpose - to provide the user with a quality product. Work is stressful enough, so we try to create a pleasant and fun atmosphere in the workplace. Everyone has their own preferences: sports, music, video games, desktops, etc.
We are actively growing and developing, and our team needs new personnel, so the idea was born to create a Banner Testing System School - two-month courses that can provide the necessary knowledge and skills to work in our team.
Training includes not only the theoretical part, but also the practice of solving problems similar to combat. Lectures and practical classes will be delivered by specialists from our team. Students who have demonstrated success in the learning process will be invited to undergo an internship at Yandex. But even if you do not set goals to get a job at Yandex, our school is a great chance to learn how to test directly from people who develop this area. Our course can be divided into several topics:
- software development and testing;
- working with version control systems;
- work with databases;
- requirements analysis and test design;
- writing test documentation;
- development of automated tests and the feasibility of automation;
- teamwork and effective interaction.
Course program
- Test automation: where, when, why?
Timur Lukin, Yandex - Test automation with xUnit
Alexander Yarulin, Yandex - CI / CD: getting rid of the concepts of "release" and "deployment"
Denis dchr Chernilevsky, Yandex - Product life cycle
Alexey Lapaev, Yandex - Efficient team and task space
Pavel Novoselov, Yandex - Agile vs. waterfall: which design method to use?
Inna Slizovskaya, Yandex - What are the types of software?
Konstantin Vlasov and Denis Chernilevsky, Yandex - Software Architecture and Design
Timur Lukin, Yandex - Types and approaches of testing
Dmitry Isaev, Yandex - Test management: statistics, documentation and plans
Inna Slizovskaya, Yandex - Test Design
Vasily Mikhailov, Yandex - How to choose a programming language?
Ivan Kalinin, Yandex - SQL: MSSQL, MySQL + ORM
Fedor Strok, Yandex - Modern web as a complex system
Marat Mavlyutov, Yandex - Intercomponent Communication Protocols
Alexey Lapaev, Yandex - Version Control System
Cyril Labetov, Yandex
More detailed information is available on the page of our school , you can also apply for admission. The set will last until 19:00 on February 21.
Source: https://habr.com/ru/post/212875/
All Articles