Experiment in Yandex: how robots help test services

The robot can not harm humans
or by their inaction to allow
so that the man was harmed.
Isaac Asimov

I am engaged in Yandex testing and I want to tell you about one of our experimental project. Within its framework, we have created a fully automatic tool for functional testing of web interfaces - the Robotester. Our robot satisfies the first law of robotics: it does not harm a person and does not remain inactive when he finds a bug that can create problems for people.



We thought that a number of test robots could be used on a par with people, and they would promptly and clearly report his problems. Robots never rest and make it so that as few people as possible collide with the error.
')
We really liked this idea, and we started to implement it.

About functional testing

The easiest way to check a site for any kind of bugs is to give a group of people a task to simulate user behavior and fix the errors found. But this solution has flaws. For example, you have to repeat the process with each release and hire more testers who will perform routine operations.

To put yourself in the place of a highly qualified manual tester, open the Market advanced search page . Start choosing a laptop while checking that the layout blocks do not move anywhere, all the page elements are loaded, and after filling out the form, you are taken to the page with the list of laptops, and you do not receive a 404 error. We call this kind of errors common. Now imagine that you need to check all of the Yandex services for them. It's getting scary.

But 404 and 404 in Africa, and js a mistake, whatever the service, is an error, NaN can neither be the price of a product, nor the distance from point A to point B. This class of errors is sufficiently uniform for all services. The process of testing them can and should be automated, and then scaled up, which is very difficult to do with qualified personnel.

Autotests can partially cope with such problems: once you have written, you can run them whenever you want. But autotests need not only to write, but also to support. And we decided to go further and manual testing, and autotests.

The robot, which should test services itself, had to be taught the following:

1. Crawl sites by both links and forms.
2. Recognize page elements and interact with them.
3. Generate and execute test scripts.
4. Catch errors

Crawling sites

This item is closely related to the recognition of elements: if we do not understand that we need to enter the name of the city in the text field, then we most likely will not be able to go to the next page and fully go around the whole site. Such technologies as WebDriver, PhantomJS, etc. are well known in forms testing.

It should be said that there is only one crawler that can bypass ajax-sites - CrawlJax . It operates with all the elements, but does not understand them at all, so for at least a little complicated site it works insanely long. At first we wanted to make a “brain” for this robot, but we realized that it would have to be significantly reworked for our needs. Therefore, we decided to make your product.

We use both “static” crawling and “dynamic”. In the first case, we set the necessary cookies (region, userid) and follow the links. In the second, we interact with the form, go to another page or to another state, which is obtained as a result of the activation of a new input field.

Recognition of page elements

The robot must simulate user actions, so it needs to interact with elements of web pages as well as people do. If there is a drop-down list in front of him, then there are no problems, he needs to choose one of the options. Also, there will be no problems with the checkbox and radio button. Difficulties arise with input fields of two types:

1. Complex input fields with javascript;
2. Fields of text input.

So far, our robot does not know how to interact with an arbitrary field that is being processed by JavaScript, since quite complicated logic can be “sewn up” in such a field. We needed to teach the robot to fill in the fields for date selection, and we created a fairly universal module for interacting with such fields. Now Robotster can choose dates on all Yandex pages.

I would like to dwell on the second paragraph. Suppose we have a text entry field. It is clear that there are fields where you can enter almost any text (for example, the search string on ya.ru ). But in many cases, entering arbitrary text leads to the fact that a significant part of the functionality is not used. Entering incorrect data, in some cases, the robot will not only be able to go to the next page, but also activate all the input fields on the current one.

For example, consider a flight search form on the Yandex.Schedules page. If for some reason the robot decides that it is necessary to enter some popular search query into the first field, and the name of a certain profession into the second field, then nothing good will come of it.



If in the search form the robot makes a similar error, then we will never see (and never check) the search results. If in the registration form, we will never get to the page of a successfully registered user, etc. Moreover, if the fields are filled in the wrong way, in some forms you will not even be able to click a button, clicking on which people should go to a new page or see the old one in a new state.

How can a robot understand what needs to be entered into a certain input field? Similar tasks also arise when developing search robots that are indexed by deep web . The term “deep web” is used to denote that part of information on the Internet that is “hidden behind forms” - to access it, you must fill in some search form. There are several approaches to solving this problem, you can read about them, for example, here , here or here .

We will talk about how this is solved in our testing. But first, we note that although in our case, and in the case of search robots, it is necessary to determine the type of the text field, there are important differences in our tasks:
Search robots bypass all Internet sites. These sites have very different layouts, and the text is written in dozens of different languages. We are building a robot only for some pages of Yandex.

For the search robot, on the other hand, you just need to maximize field recognition. Already some search results can be obtained if 90% of the fields are correctly recognized in the form. But we are not satisfied with the level of recognition below the "absolute" 100%. In addition, we work not only with search forms. Consequently, in order to create a robot that would qualitatively imitate human behavior on the page and send correct information, we had to develop the most accurate recognition algorithm. Universality, we were ready to sacrifice.

Our algorithm is based on a type dictionary. The dictionary is as follows:



As a result, the recognition of the field type is reduced to the choice of one of the ready-made types contained in the dictionary (at the moment we have 27 of them). To do this, for each type of input, we have formed a list of keywords. The list included those that, being present next to the field in the interface, increase the likelihood that it belongs to this type. Of course, not all keywords are equally useful. Each keyword is associated weight, but more on that later. The algorithm can be divided into several parts.

Segmentation. We divide the interface into parts corresponding to the input fields.



In it, we are interested in the text and its location relative to the input field that we are studying. But, firstly, it is not very clear what a “text” is - only visible text may not be enough, but at once it is bad to take the entire text of the layout, because it can contain a lot of garbage. Therefore, we use both the one and the other text, but with different weights.

The task of extracting the neighboring (visible or invisible) text we solve as follows:

• element.innerHTML + = "@ # @ $ @ #!"
• extract (visible / invisible) text from the entire form
• element.innerHTML- = "@ # @ $ @ #!"

Further we find in the received text the substring "@ # @ $ @ #!". It will symbolize the position of our input field in the received text. We extract the text using jQuery.

Putting down the scales. The resulting text is divided into words, after which each word appears weight. It depends on whether the word is visible or not, and on the distance between it and the input field.

Type selection. The keywords for the type “price” in abundance are located next to the price entry field.



The algorithm for evaluating the relevance of the type is as follows. First, words are ordered by weight in descending order. We will call this list X list. Now we will consider the weights of the field types themselves. Each type has its own set of words with weights - it will be a list of Y. We intersect this set with a list of X and get a list of Z. Weights of words in the list of Z will be obtained by multiplying the weights in lists of X and Y. Next we need to get rid of situations in which weight of the type is influenced by "far" words - located in another part of the page. Therefore, let's divide the word in the list Z with the number i by 2 ⁱ , then add all the weights of the words in the list Z - this will be considered a weight of this type.

As already noted, an important element of this method is a set of keywords with weights for each type. Weights must be chosen so that the maximum total weight is in the "correct" field type. How to choose such weight? There are many ways, we have chosen a simple "fit." That is, we set the weights in some way, and then corrected them in a situation where the robot did not correctly recognize the type of some field that we needed to process. A special interface was created for learning the robot, and if the operator indicated that the robot mistakenly chose the type “Price” based on the word “from” (and it was necessary to choose the type “Address” based also on the word “route”), then the weight the words “from” for the type “Price” decreased, and the weight of the word “route” for the type “Address” increased.

However, the universality of this approach is quite low. An increase in the number of tested services will inevitably lead to a situation where we will not be able to manually select weights so that they work on all sites. Then it’s time to come up with something new: for example, use machine learning based on the “correct” field types that the operator sets for some pages. Actually, initially we did not think that such a simple algorithm would suit us, but, as it turned out, it completely fits our needs - at the moment the robot successfully recognizes all fields of all forms for which we launch it. These are 5-6 different Yandex services.

About the results

Robotster helps testers Yandex.Market for more than five months. 12% of the bugs found during this time in the production environment belong to the robot. He does not sleep, does not eat (hardware time, he eats), and most importantly - he is alone, but he has a lot of it. Thanks to 32 cores! I want to believe that in the testing routine manual labor will be replaced by mechanical.

Of course, such robots will not take away the work of manual testers and auto testers, because it can be difficult to figure out the business logic and specifics of the service even to a person. Just in the work of qualified personnel will be less routine, and in Internet services - fewer errors.

While working on the robot, we had many other interesting tasks. For example, I had to invent my own algorithm for crawling, since in a reasonable time it could not cope with the crawling of the Market’s mobile site containing tens of millions of pages. Or, for example, the robot produced a lot of similar error messages. We have developed a special reporting system in which similar errors are “glued together” into one. There are also interesting tasks that have not yet “gotten their hands” - for example, how to distinguish typos in user comments from functional errors that are noticeable in the text. We will tell about all this in the following series.