As WEBO Runet counted

Not so long ago, we finished collecting baseline data for the accessibility rating of popular Runet sites . The data was very unexpected, but the most interesting was the counting of lost visitors for one particular site, and for all popular sites at once.

Running a little ahead, I will say that sites lose about a million visitors every day. And in about half of the cases, these losses are almost imperceptible, because they occur either because of "floating" failures, or because of the large influx of visitors (and exceeding the waiting time for a response). Fortunately, this does not happen so often, but in financial terms, the effect is quite tangible. For example, only avito.ru has lost almost 140 thousand visitors over the past week.

Technology

The network of WEBO Pulsar servers now includes 3 independent points (2 in Moscow and 1 in Amsterdam), additionally every minute a number of external points are checked, which should be “always available”. Only on the basis of this information is a conclusion about the inaccessibility of a particular resource.
')
To compile the rating, several hundreds of sites were visited with over a million people per month, and for each of them once a minute the connection is checked (simply speaking: can the site be accessed from a browser, or the site will respond incorrectly, with an error or not will not answer). According to the data compiled during the week, the rating itself is compiled.

The implementation of the described technology, of course, is not trivial. But the most interesting, in my opinion, is to collect information about site traffic (to include them in the rating and calculate the loss of visitors).

Site Attendance

It is very convenient that about 2/3 of the sites use statistics Liveinternet, in which data on visits per month are open. This greatly simplified the task. But what to do with the rest?

Alexa comes to the rescue, which measures the share of the site in global traffic. She has a distribution by country and subdomain, so in almost all cases you can find out exactly what proportion of global visitors came to the site of interest.

Only here is one catch: the share of global visitors is translated into a specific number only if you know the total number of Internet users (which Alexa counts). And the number of these visitors will vary greatly from country to country: i.e. for example, half of users are counted from the States, and only 20% from Russia. What to do?

Here we remember the proportions and take the Liveinternet data as a basis. Having built two models (for the largest and medium-large sites), we can, knowing the proportion of Alexa traffic, calculate the approximate number of users of a web resource for which the Liveinternet counter is not installed. Accuracy is approximately 20%, which is quite enough for an assessment and rating.

The most popular web resources of the Runet were selected on the basis of both Liveinternet statistics itself and Alexa rating, and TNS rating. Combined and rechecked data provided a fairly accurate picture of popular sites.

For all sites in the ranking, availability ( in terms of “nines” ), actual downtime (in hours and minutes) and the number of visitors who could not get to the site are calculated. Knowing the profitability of the site (for example, 25 or 50 kopecks per visitor), you can easily calculate losses from its downtime. Unfortunately, the financial information on the site’s revenues is closed, otherwise we would automatically calculate the losses. :)

PS Now the rating is recalculated once a day, about 150 top sites are covered. We plan to bring this number to 250-300. For questions about the inclusion (or availability) of the site in the ranking, you can write to us by mail or in the comments to the article. Now we have counted 1.3 million lost visitors in a week. Given the coverage of only 10-20%, it turns out the figure voiced above - a million lost visitors per day .