On the effectiveness of Rutreker's “teachings”

“Every task has at least one obvious and incredibly easy to understand wrong solution.”

As one of you already knows, Rutreker, a well-known resource, conducted the campaign “Civil Defense Teachings”. Its essence was in the "training" closing access to the forum for all users from Russia for a day. The next day, popular online publications were full of the headlines "Rutreker's Learning Blocking did not affect his attendance." For me, as a regular user of this site, the topic with its blocking is especially painful.

Seeing encouraging headlines, I was glad for the small victory of my favorite resource. However, after reading the articles, I had a few questions. That's why I decided to figure it out: did Rutreker successfully cope with the task and taught all his users to easily and naturally bypass the locks?

Rutreker attendance analysis

Internet publications wrote: “the attendance of the resource not only did not fall, but turned out to be higher than the average for the previous week .” This conclusion caught my attention. In terms of numbers, the second part of the quote is correct. Indeed, if you look at the attendance statistics, there should be no questions. But why the conclusion about the success of the event? Yes, the average weekly attendance increased, but the action took place exactly one day - on Sunday, not the whole week. So it is simply incorrect to use the average attendance for 7 days as an indicator of the effectiveness of the action.

It is worth paying tribute to Internet publications, they also mentioned that “ on average, on Sunday in November, the torrent tracker website was visited by 1.24 million unique users - that is, only 10% more than December 6th .” The last statement partially sheds light on the essence of the question and, no matter how sad it may sound, indicates a drop in attendance by 10%, but the authors of the quoted article closed their eyes to this.
')
So what really happened to attendance? Has it fallen, grown or remained unchanged? Let's look at the average daily attendance of the tracker since mid-November (the most stable and relevant area):


Fig. 1. Dynamics of unique website visitors from 11/15/15 to 12/05/15, million of visitors.

Looking at the chart, we can say the following:

• There is no pronounced trend in attendance values;
• There is a pronounced seasonal component (the maximum attendance always falls on Sunday)

Based on the available data, we will model the behavior of attendance in the period under consideration and build a forecast for December 6 and 7. Taking into account the absence of a trend and the presence of seasonality, we use the method of exponential smoothing with seasonal decomposition as a forecasting model.

The seasonal component, highlighted in the values ​​of the initial time series, indicates a typical increase in the number of tracker visitors by 280 thousand people every Sunday . Logical, when else to download?

The resulting predictive model looks like this:


Fig. 2. Actual and simulated attendance levels with a forecast and fact of December 6 and 7, million visitors.

The model very accurately describes the actual attendance of the resource. The average absolute percentage error (MAPE) is 1%, which is an excellent indicator for predictive models. A 98% hit with attendance on the day after the "exercise" - December 7, Monday - further indicates the quality of the selection of the model.

As you can see, the model could not predict only December 6th. That is, on this day, attendance did not comply with standard laws. Unfortunately, it was precisely on December 6 that the “exercises” were held. And their real effect is the discrepancy between the predicted value of attendance and the actual. Thus, the Sunday event led to a decrease in attendance by 190 thousand people, which amounted to 14% of the expected.

Also note that the lower limit of the confidence interval of the predicted attendance at December 6 is 1.26 million visitors. This is 140 thousand more than the actual attendance. That is, with the most optimistic scenario, we can talk about a significant decrease in the number of visitors - from 140 to 230 thousand people, or 11-17% of the expected Sunday attendance, or 13-22% of the average daily attendance in November. Not too much like the picture being drawn on the Internet, right? I dare say that this is quite an alarming result. He looked at the story for a couple of years and realized that Rutreker’s attendance had declined by more than a third in two years:


Fig. 3. Dynamics of the average monthly attendance from December 2013 to December 2015, million of visitors.

Other indicators of the effectiveness of "exercises"

It seems to me that it was not worth assessing the effectiveness of Rutreker’s “teachings” by the level of attendance, because:

• the action was accompanied by active public relations, which a priori was to increase traffic on that day;
• The attendance counter, presumably installed on the main page, speaks only of the user entering the site. And this can not serve as an unequivocal evidence of circumventing academic lock. In addition, judging by the comments on the forum, not all of the learning lock worked: many users still have full access to the site.

In fact, it is best to evaluate the effectiveness of the action based on the internal statistics of Rutreker himself, reflecting, for example, the number of participants in the distributions. Rutreker hasn’t shared it yet ... Above, I diagnosed a decline in attendance as a whole by 11-17%. But how to understand how many users who came to the site could bypass the block and participate in the distribution?

An indirect criterion is the average number of page views per visitor. Let's say the user went to the site, saw an ad about the need to use workarounds and decided not to twitch at all. Then the number of pages viewed by him will be 1. Look at the numbers:


Fig. 4. Dynamics of the average number of views per visitor on Sundays from November 6 to December 6, 2015.

The decline in the index by 10% on the sixth of December is also the effect of the “exercises”. Every tenth user who came to the site could not look for ways to bypass the lock and left the page. This hypothesis is also supported by the fact that the share of users who looked at only one page increased by 3 percentage points. (compared with November) and amounted to 27%.

Features of the choice of ways to bypass locks

It is curious in this story and how users bypassed the “learning” lock. A survey was conducted on Rutreker, and according to his data, the vast majority of torrent users used browser plugins. Relatively complex tools, such as VPN, Proxy, own servers and others, used only one in five. Particularly pleased that only 3% admitted that they can not bypass the lock.

Summary

In fact, without having access to Rutreker’s internal statistics, we can guess as much about the success of the action. Obviously one thing - most of the audience will be lost if the site is still blocked.

But there are positive moments - only 3% could not bypass the lock, or simply did not admit it.