Summary of registered second-level domains in the .RU zone

The purpose of the study was to obtain an up-to-date list of all active domains in the .RU zone, by 01.01.2016 it was registered with 5040277 names. They made a decision to go through each name with a crawler and analyze the result.

Server responses were as follows:


')


IPFAIL - the domain could not be resolved (not delegated, name servers are not specified, etc.).
TIMEOUT - IP was received, but did not give anything and fell off on timeout.
CHARSETFAIL - content coding could not be recognized.
NONHTML - sites whose web servers did not interpret the scripts, but gave them to the text along with the connection details to the databases and other delights.
CLEX - crawler exceptions by response size> 10mb.

301 redirect (permanent):


bulk - a zoo from satellite nets, alternative website addresses, and so on.

	cnt	%
http: // www.domain	215289	48.4
bulk	144275	32.4
http: // domain / page	76417	17.2
https: // domain	7617	1.7
https: // www.domain	1121	0.3
Total :	444719	100.0

302 redirect (temporary):



bulk - all the same grids, errors, installers of various cms, etc.

	cnt	%
bulk	135464	76.9
http: // domain / page	22658	12.9
http: // www.domain	10660	6.1
https: // domain	7168	4.1
https: // www.domain	183	0.1
Total :	176133	100.0

In the redirect through meta refresh, we also look, but this time there is nothing interesting there. The most popular way to send a user to a bunch of exploits.

All 2670175 domains that donated 200 OK are running on 192213 IP addresses, top 10:

ip	cnt	example_url
109.206.190.54	180983	http://ksimelin.ru
188.166.49.110	40199	http://samoobsluga.ru
81.177.141.15	15040	http://plusheviy-buket.ru
89.111.167.3	11887	http://ipoezd.ru
94.76.205.132	11774	http://mediatr.ru
78.108.80.238	10937	http://bikesurfing.ru
78.108.86.63	10883	http://asp-v.ru
185.32.57.224	10157	http://salgroup.ru
188.120.232.174	9431	http://moscowspa.ru
199.59.243.120	9057	http://asolutbank.ru

Here we meet really interesting guys: 180983 domains on ip 109.206.190.54 ( 6.77% of all active ) are mirrors of www.homes.ru (compared not only by ip, of course). With a huge margin go even from parking. Work in a big way.

Few average values ​​of the content component of the main pages of the RuNet:

Average title length	47
Average Keyword Length	220
Average words per page	515
Average page weight (in octets)	42320

On 262 domains in the text occurs the word ' habrahabr '.


List of domains that donated 200 OK dataoperator.ru/ru_domains_200_ok.zip