Great fight with wrapper programs

It is no secret that in the modern world the problem of unwanted and malicious software is becoming more and more acute. Botnets for various purposes, lockers, adware ...
Of course, many are trying to deal with it. I work for one regional provider, and we also pursue a policy of passive warning of users, promote antiviruses and culture of using the Internet. In the end, we are better off if everything works quickly and adequately for the client.
Recently, search engines have joined this. Under the cut an entertaining story about how we became a victim of this struggle, as well as an occasion to think about what this all can lead to.

The summer has just ended, nothing, as it usually happens, foreshadowed trouble. Suddenly a letter comes from Yandex with the following content:

Links to files or programs to which additional software has been added are found on the *****. Ru pages of the site. Thus, along with the downloaded file, additional programs can be installed on the visitor’s computer — safe or doubtful.
At the moment the site is displayed in the search results marked "Be careful when downloading files from this site."

Please remove links to download such files. If during the new check they are not found, the mark in the search results will be removed.

The domain in question is our small forum. We do not promote it in any way, it’s just a place for subscribers to communicate with each other on various topics.
First thought - broke! The second thought - one of the users laid out something bad. Immediately we start the check - we review the page code, look at the latest forum attachments ... And we don’t find anything. We are writing to the support of Yandex with the question of what is actually happening, the answer comes pretty quickly, which we were a bit dumbfounded and led to the appearance of the next correspondence.
')
Yandex:

Your site receives complaints from users to download unwanted executable files. As soon as complaints stop, the mark will automatically disappear. We do not recommend using affiliate programs that cause you to doubt.

Our attitude to poor-quality files is described on the following pages:

blog.yandex.ru/post/81042
habrahabr.ru/company/yandex/blog/226817
company.yandex.ru/rules/distribution

We:

We do not use any affiliate programs. We also do not distribute any executable files, it’s just a local provider's forum.
Please indicate the links to the files that caused the warning.

Yandex:

We give an example of a poor-quality page: http: //*****.ru/viewtopic.php? F = 15 & t = 3503.

We:

I do not see a single file on this page. What is meant by placing a link to some other forum in the first post of this topic? Then how did “unwanted executables” suddenly become “links to other resources”? And how can we control the content of another resource?

Yandex:

In order to avoid the appearance of a warning near your site *****. Ru, there is no need to control third-party resources, you only need to protect your site visitors from poor-quality content. If the user goes from the search results to your site and after several transitions loads the wrappers, then your site will be displayed with a warning in the search results.

We:

In other words - if a person on the forum (blog, personal site, etc.) places a link to some resource from which you can download "unwanted executable files", then when searching in Yandex, will this site appear with a warning?
If so, then I can place such a link on any popular blog hosting such as blogspot or livejournal, wait until someone passes through it and you will also display this blog hosting with a warning?
If not, then I still do not understand the reason for the warning for our forum.
As it is not clear what, actually, to do. Delete threads? Check each link that the user has posted for whether or not there is something to download with wrappers?

Yandex:

We recommend checking the content of pages from which a large number of transitions to external resources from your site occur.
The situation described by you may affect the site’s verdict on the site’s danger by the algorithm. We do not provide any details of the work of our algorithms, including how, when, and in what way the system recorded attempts to use unacceptable methods of distributing additional software, whether they are fixed now, etc. When developing a site we recommend focusing on the needs of users, and not on search engines.

It takes just a few hours after the last message and we receive a new notification:

The last site check 2015-09-07 did not reveal links to files with added software.
In the search results the site is displayed without marks.

What happened? In 2012, a user created a topic with the following content:



The only external link on the page is a link to the torrent tracker unionpeer.
The link opens a catalog of games, in order to download something you must first go into the distribution itself. That is at least 2 clicks from our forum. We, to put it mildly, were stunned by such a harsh approach of Yandex to the case.
They were also struck by double standards. It is clear that they will not put a warning on any livejournal.com, although there are more than enough links to this tracker . But to shake a small forum, then to loudly declare their struggle with unwanted software - this is yes.

It would seem that the incident is settled. In Yandex, they understood (?) The absurdity of what is happening and removed the mark from the site.

But December came and two new notifications:

Links to files or programs to which additional software has been added are found on the *****. Ru pages of the site. Thus, along with the downloaded file, additional programs can be installed on the visitor’s computer — safe or doubtful.
At the moment the site is displayed in the search results marked "Be careful when downloading files from this site."

On the pages of your site *****. Ru found a code that may be dangerous for visitors. Execution of this code when visiting the site may lead to undesirable consequences for the user: computer infection by malware, unauthorized use of its resources, damage or theft of personal data.
The site is currently displayed in search results labeled "This site may threaten the security of your computer."

And again check, and again nothing. According to virustotal, the only positive response is Yandex Safebrowsing. Immediately, users who use Yandex Browser and / or Yandex DNS begin to contact technical support, when they try to log in to the forum, they are given such a warning without being able to bypass it:



Fortunately, there are not many.

We ask again what is happening. It turns out again to blame the unfortunate topic with a single reference to the unionpeer.

Yandex:

During the last check of your site, our robot found low-quality programs available for download, for example, on the page http: //*****.ru/viewtopic.php? F = 15 & t = 3503. These programs may contain additional components that are installed without the user's knowledge. For this reason, on the search results page a corresponding warning is displayed near your site.

You can read more about this in the following materials:

blog.yandex.ru/post/81042
habrahabr.ru/company/yandex/blog/226817
company.yandex.ru/rules/distribution

We:

The service Yandex Webmaster specifically indicates the page http: //*****.ru/viewtopic.php? P = 109830 with the verdict “Bedep_payload”, not the one you are talking about.
No low-quality programs are stored on our server. Please provide direct links to poor-quality software stored on our server so that we can delete these files.

Yandex:

The verdict “Bedep_payload” in the Yandex.Webmaster service on the “security” tab was specified incorrectly. Please ignore this information.

We checked, in this case, the algorithms worked correctly. We do not provide any detailed information about poor-quality content on user sites. Please use the recommendations from our previous letter.

We:

You already gave me these links and there is still nothing in them that would explain your actions. Please indicate a direct link to our forum, on which the "wrapper program" is downloaded. Under the link given in your first answer, nothing is downloaded.

Yandex:

We do not provide such information, sorry. Please try again carefully to look at the page we indicated earlier and analyze all external links for the presence of poor-quality content.

I don't even know if any comments are needed here. Well, to hell with him with a false positive, for which they did not even apologize. But Yandex, under the pretext of fighting undesirable software, pursues some strange policy of blackmail and censorship, refusing to explain the reasons.
This is all puzzling and makes you seriously think about the direction of movement of the modern RuNet, one might think Roskomnadzor alone is not enough.

For us, the credibility of Yandex is completely undermined. Double standards and tyranny have never brought anyone to good. In any case, we don’t have such a big choice - give instructions to tech support, advise subscribers to delete Yandex Browser and Yandex DNS, if they have any problems accessing the forum, or delete the topic indicated by Yandex and hope that they don’t solve tag anything else in the future.

At the moment, the site opens without forced blocking, the warning is only in the search results. However, calls from users are still ongoing, probably something got into the cache.

PS I intentionally replaced the address of the forum with asterisks in order to exclude a sudden influx of visits. Those interested can freely google it on the text of the topic, in order to check everything yourself.

UPD: mark from the site again removed, we for our part did nothing.

The last site check 2015-12-28 did not reveal links to files with added software.
In the search results the site is displayed without marks.

UPD2: a Yandex representative contacted me, with his permission I publish a full log of correspondence in pdf - link .
TLDR - The site was flagged because a significant number of visitors to the forum fell on the topic with reference to unionpeer. People crossed over it and downloaded wrapper programs there. Yandex will try to be more careful in cases where this is done unintentionally, it may be possible to mark in the issue a specific page instead of the entire site. Technical support, deciding that the site is not specifically created for the distribution of adware, will provide the most complete information on the detection of unwanted links to visitors. We, in turn, will remove this particular link from our forum.