Mechanical Turk: now 40.92% spam
The most popular crowdsourcing service, Mechanical Turk, is increasingly being filled with spam. More precisely, tasks from spammers. There are already dozens of projects in the “Test advertising on my site”, “Create an account on Twitter and download me”, “Send my video on Youtube”, “Download specified application”, “Write positive review "and so on.
NYU students Dan Tamir and Priya Kant decided to find out the scale of the problem and accurately calculate how much spamming orders occupy MTurk. To do this, they used the statistics of Mechanical Turk Tracker and analyzed the market for September and October 2010.
All new orders placed during these two months were selected (it is assumed that regular customers are not spammers).
A total of 1,733 new customers appeared on the market during this time, who placed 5842 orders. Further, the researchers used the same crowdsourcing methods and instructed the public to classify these orders as “spam” or “not spam” with giving detailed instructions, which is considered spam (SEO, fake accounts, fake clicks, and fake advertising are included in the task, specific actions are required on site of the customer, there is a request for personal data of the artist, etc.).
')
That's what happened .
Of the 5841 verified tasks, 40.92% were associated with spam.
Then, the researchers checked how often spamming tasks come from normal customers. It turned out - nothing like that. The vast majority of spamming tasks come from a special category of customers, in which 100% of the tasks are spamming.
According to statistics, 31.83% of new customers place only spam tasks. That is, there is a clear border.
So, it turns out that 31.83% of new customers are spammers, and 40.92% of new tasks are pure spam. Obviously, this is a problem.
The quality characteristics of spam is also different from the normal tasks. First, spammers in each set (HITs) have fewer tasks (HIT).
Secondly, they evaluate the work more expensive than usual (because they do not pay?). On average, 80% of tasks are cheaper than a dollar, and spammers have only 60%.
It is not clear why Amazon is not fighting spam in the service of Mechanical Turk. After all, it is easy enough to recognize it by keywords. One possible explanation is that spammers actually pay for order fulfillment, and Amazon stupidly gets its percentage. Judging by the above statistics, spammers provide about half of all financial income of the service.
Another explanation is that Amazon has abandoned the development of the project and simply does not know what is going on there, or it doesn’t care. The service has been working on its own in almost unchanged form since 2005. Against this option, say the recent discovery of Twitter and the blog MTurk.
NYU students Dan Tamir and Priya Kant decided to find out the scale of the problem and accurately calculate how much spamming orders occupy MTurk. To do this, they used the statistics of Mechanical Turk Tracker and analyzed the market for September and October 2010.
All new orders placed during these two months were selected (it is assumed that regular customers are not spammers).
A total of 1,733 new customers appeared on the market during this time, who placed 5842 orders. Further, the researchers used the same crowdsourcing methods and instructed the public to classify these orders as “spam” or “not spam” with giving detailed instructions, which is considered spam (SEO, fake accounts, fake clicks, and fake advertising are included in the task, specific actions are required on site of the customer, there is a request for personal data of the artist, etc.).
')
That's what happened .
Of the 5841 verified tasks, 40.92% were associated with spam.
Then, the researchers checked how often spamming tasks come from normal customers. It turned out - nothing like that. The vast majority of spamming tasks come from a special category of customers, in which 100% of the tasks are spamming.
According to statistics, 31.83% of new customers place only spam tasks. That is, there is a clear border.
So, it turns out that 31.83% of new customers are spammers, and 40.92% of new tasks are pure spam. Obviously, this is a problem.
The quality characteristics of spam is also different from the normal tasks. First, spammers in each set (HITs) have fewer tasks (HIT).
Secondly, they evaluate the work more expensive than usual (because they do not pay?). On average, 80% of tasks are cheaper than a dollar, and spammers have only 60%.
It is not clear why Amazon is not fighting spam in the service of Mechanical Turk. After all, it is easy enough to recognize it by keywords. One possible explanation is that spammers actually pay for order fulfillment, and Amazon stupidly gets its percentage. Judging by the above statistics, spammers provide about half of all financial income of the service.
Another explanation is that Amazon has abandoned the development of the project and simply does not know what is going on there, or it doesn’t care. The service has been working on its own in almost unchanged form since 2005. Against this option, say the recent discovery of Twitter and the blog MTurk.
Source: https://habr.com/ru/post/110224/
All Articles