Yandex has learned to recognize and combine a series of images.
Yandeks.Kartinki today made an important step in its development and in the development of search for pictures. Search results will no longer be just a collection of pictures for the words you specified. Now in the results some images can be combined together. We call it "series."
"Series" are images that appear together on the Internet and are visually similar. For the selection of images is the hierarchical clustering algorithm. He takes all the images from a particular page and highlights a group of similar ones - those that have a common color, shape, details, and so on. If there are at least four similar (but not identical) pictures on the page, they form a series.
')
Read under the cut why we had the idea to make the Series, how we thought out their design and implemented the algorithm.
The presentation of pictorial search results, which is familiar to everyone, is a page filled with a tumbneal grid. Scrolling it down, you usually see new thumbnail images automatically loaded. But sometimes one or even two pictures are not enough to answer a question - you need more images. And it would be nice if they were related.
What is it for? For example, to learn how to fold origami dragon, you probably want to see the picture with each step. You need a few images in the event that you want to view the car from different sides. And, probably, it will be great if these are pictures from one review.
To understand what other scenarios the series might have, we conducted in-depth interviews. One girl said that the series would have helped her quickly find three pictures for the photo frame in the kitchen. It was important for her that they were all with spices and in the same style, because she has a special frame for three images.
There was also a student who said that the instructions in the pictures would help her learn photoshop. Video instructions are too complicated for a beginner, and pictures with explanations are the most. We also talked with a man who told us that quite recently he was looking for instructions with pictures on how to fix a leaked tap. According to our data, in order to solve such problems, approximately 13% of users look for a page through a picture that has detailed information.
Thanks to quantitative polls, we know that 70% of users periodically need to find instructions that should have illustrations, and 20% need such a need every week. This statistic is also confirmed by queries - 9% of all requests for searching in Yandex images are related to searching for instructions. And they are on completely different topics.
In the process of discussing this problem, we had a project called “Series”. We began to think how to select images in order to get not only a relevant, but also a beautiful answer that would complement the issuance of pictures.
Thanks to interviews with invited users, UX testing, a lot of discussions within the team and beta data in the internal Yandex network, we have built a more coherent understanding of this possibility and formulated some requirements.
For example, images should be from one page, so that the user can go to it and find out more - see the entire report, read the explanations, go to other sections of the site. This improves the navigation scenario in which the user searches the site through a picture.
The series is especially useful when it highlights one object, the steps of one master class, or photos taken in the same style (a specific photo session, clothes from one collection, etc.). But the algorithm does this only for pages without aggressive advertising and viruses. In in-depth interviews and UX testing, we are faced with the fact that users react very negatively to excessive advertising and jumping pop-ups. Therefore, we decided to exclude such pages from the candidates for serial ones.
Group the pictures in a series of similar within the pages on which they are found together. That is, if on the page www.example.com met
www.example.com/1.jpg
www.example.com/2.jpg
www.example.com/3.jpg
www.example.com/4.jpg
try to combine them into a series.A series is a group of pictures that are visually similar to each other in pairs.
We want to cluster these pictures by visual similarity. That is, find a subgroup of pictures that are quite similar to each other. We will cluster using greedy hierarchical clustering algorithm. In English, this algorithm is called complete linkage clustering using nn-chain algorithm. To achieve success, we need a metric of image similarity, clustering with which would give a cluster-group with the properties we need.
What are these properties?
In order for something like that to work out, we selected three kinds of descriptors:
Descriptions can be found, for example, in the mpeg-7 standard. We have our own quick and effective implementation.
On the basis of these descriptors, visual similarity is calculated, as a maximum by visual similarity for individual descriptors. This allows you to satisfy the first three Wishlist. To take into account the size, we subtract from the visual similarity the ratio of image areas (max / min). Based on this metric, we implement clustering.
In the course of work, we realized that serial pictures on the issue should look like a single unit - this is a new view of the Yandex response. The most important thing in the design was to make the series of pictures visible.
From the moment we started working on the project, we tried about a dozen different design options. Three of them have been tested both on external users and on our colleagues.
We have seen that externally, users like more than one design.
But they understand a different idea, and with it they interact more.
In the final version, we took into account the best of both options.
We also noticed that it is important to show people when the series opened and when it ended. So there was a final version with the information and the sharing block.
The series will be very convenient for those people who are looking for step-by-step instructions in which pictures are more important than text: how to make an origami dragon, how to draw a cat, a master class in decoupage, how to change or find exercises for a trapezoid.
With them it is easier to look for images that are similar in style - pictures by one artist or collections of photos from one photo shoot. They will also help to consider a car, a product or a point of interest from different sides.
"Series" are images that appear together on the Internet and are visually similar. For the selection of images is the hierarchical clustering algorithm. He takes all the images from a particular page and highlights a group of similar ones - those that have a common color, shape, details, and so on. If there are at least four similar (but not identical) pictures on the page, they form a series.
')
Read under the cut why we had the idea to make the Series, how we thought out their design and implemented the algorithm.
The presentation of pictorial search results, which is familiar to everyone, is a page filled with a tumbneal grid. Scrolling it down, you usually see new thumbnail images automatically loaded. But sometimes one or even two pictures are not enough to answer a question - you need more images. And it would be nice if they were related.
What is it for? For example, to learn how to fold origami dragon, you probably want to see the picture with each step. You need a few images in the event that you want to view the car from different sides. And, probably, it will be great if these are pictures from one review.
To understand what other scenarios the series might have, we conducted in-depth interviews. One girl said that the series would have helped her quickly find three pictures for the photo frame in the kitchen. It was important for her that they were all with spices and in the same style, because she has a special frame for three images.
There was also a student who said that the instructions in the pictures would help her learn photoshop. Video instructions are too complicated for a beginner, and pictures with explanations are the most. We also talked with a man who told us that quite recently he was looking for instructions with pictures on how to fix a leaked tap. According to our data, in order to solve such problems, approximately 13% of users look for a page through a picture that has detailed information.
Thanks to quantitative polls, we know that 70% of users periodically need to find instructions that should have illustrations, and 20% need such a need every week. This statistic is also confirmed by queries - 9% of all requests for searching in Yandex images are related to searching for instructions. And they are on completely different topics.
In the process of discussing this problem, we had a project called “Series”. We began to think how to select images in order to get not only a relevant, but also a beautiful answer that would complement the issuance of pictures.
Thanks to interviews with invited users, UX testing, a lot of discussions within the team and beta data in the internal Yandex network, we have built a more coherent understanding of this possibility and formulated some requirements.
For example, images should be from one page, so that the user can go to it and find out more - see the entire report, read the explanations, go to other sections of the site. This improves the navigation scenario in which the user searches the site through a picture.
The series is especially useful when it highlights one object, the steps of one master class, or photos taken in the same style (a specific photo session, clothes from one collection, etc.). But the algorithm does this only for pages without aggressive advertising and viruses. In in-depth interviews and UX testing, we are faced with the fact that users react very negatively to excessive advertising and jumping pop-ups. Therefore, we decided to exclude such pages from the candidates for serial ones.
How the series is arranged technically
Group the pictures in a series of similar within the pages on which they are found together. That is, if on the page www.example.com met
www.example.com/1.jpg
www.example.com/2.jpg
www.example.com/3.jpg
www.example.com/4.jpg
try to combine them into a series.
We want to cluster these pictures by visual similarity. That is, find a subgroup of pictures that are quite similar to each other. We will cluster using greedy hierarchical clustering algorithm. In English, this algorithm is called complete linkage clustering using nn-chain algorithm. To achieve success, we need a metric of image similarity, clustering with which would give a cluster-group with the properties we need.
What are these properties?
- The same objects or scenes taken from different angles must be in the same cluster;
- The same objects or scenes made in different colors in the image must be in the same cluster;
- Photos from one photo shoot, in which there are enough common colors and details, should be in the same cluster.
- The cluster should have pictures of approximately the same size, etc.
In order for something like that to work out, we selected three kinds of descriptors:
- descriptor based on color layout descriptor ;
- mpeg dominant color descriptor ;
- mpeg-7 edge histogram descriptor.
Descriptions can be found, for example, in the mpeg-7 standard. We have our own quick and effective implementation.
On the basis of these descriptors, visual similarity is calculated, as a maximum by visual similarity for individual descriptors. This allows you to satisfy the first three Wishlist. To take into account the size, we subtract from the visual similarity the ratio of image areas (max / min). Based on this metric, we implement clustering.
Design
In the course of work, we realized that serial pictures on the issue should look like a single unit - this is a new view of the Yandex response. The most important thing in the design was to make the series of pictures visible.
From the moment we started working on the project, we tried about a dozen different design options. Three of them have been tested both on external users and on our colleagues.
We have seen that externally, users like more than one design.
But they understand a different idea, and with it they interact more.
In the final version, we took into account the best of both options.
We also noticed that it is important to show people when the series opened and when it ended. So there was a final version with the information and the sharing block.
Where the series will help you
The series will be very convenient for those people who are looking for step-by-step instructions in which pictures are more important than text: how to make an origami dragon, how to draw a cat, a master class in decoupage, how to change or find exercises for a trapezoid.
With them it is easier to look for images that are similar in style - pictures by one artist or collections of photos from one photo shoot. They will also help to consider a car, a product or a point of interest from different sides.
Source: https://habr.com/ru/post/198830/
All Articles