The most common structure of sentences in Russian according to the version of the library Flibusta

I am a php programmer, but I wanted to expand my horizons, to learn something new. Therefore, I decided to teach other languages ​​and technologies. The choice fell on perl, python and mysql.

A wonderful pymorphy package was taken, Flibusta library (.fb2 only), sedna for storing fb2, mysql percona 5.1 for storing statistics and a small file. A primitive myisam plaque was created where it was recorded how many sentences occurred, and a description of the parts of the speech of this sentence.

According to the description, I made a unique text index, but I forgot to make an index by a numerical field (I thought it would not be useful).

Fb2 with filibustra placed in sedna base, it turned out the base is somewhere in 90 GB.
')
The first step I collected all the unique words found in the library. It turned out about 14 million words. These were both unique words and their forms, and other garbage.

It was experimentally revealed that in the pickle mode, pymorphy spends a lot of time on initialization, and it will take a very long time to pull it out of php through the shell. Since I just started learning python, and to get a part of speech, I had to use pymorphy, I decided to run the latter as a server. This eliminated the problem of long initialization, and this is what happened:

#!/usr/bin/env python <br/>
# -*- coding: utf-8 -*- <br/>
from pymorphy import get_morph<br/>
import re , sys , socket , pprint ,json,chardet, ConfigParser <br/>
<br/>
def do_word ( data ) :<br/>
if ( data == False or data == '' ) : print "error incoming data !!!" <br/>
tmpWord = re . sub ( r "rn" , '' ,data ) <br/>
word = tmpWord. decode ( 'utf-8' ) . upper ( ) <br/>
if ( DEBUG == 1 ) : print word<br/>
info = morph._decline ( word ) <br/>
if ( DEBUG == 1 ) : pprint . pprint ( info ) <br/>
sjSon = json. dumps ( info ) <br/>
return sjSon<br/>
<br/>
config = ConfigParser . ConfigParser ( ) <br/>
config. read ( 'pymorphy_conf.ini' ) <br/>
DEBUG = config. getint ( 'decline' , 'DEBUG' ) <br/>
HOST = config. get ( 'decline' , 'HOST' ) <br/>
PORT = config. getint ( 'decline' , 'PORT' ) <br/>
<br/>
morph = get_morph ( "/pymorphy/dicts/converted/ru/morphs.pickle" , 'pickle' ) <br/>
<br/>
srv = socket . socket ( socket . AF_INET , socket . SOCK_STREAM ) <br/>
srv. bind ( ( HOST,PORT ) ) <br/>
while 1 :<br/>
if ( DEBUG == 1 ) : print " " ,PORT<br/>
srv. listen ( 1 ) <br/>
sock,addr = srv. accept ( ) <br/>
while 1 :<br/>
pal = sock. recv ( 1024 ) <br/>
if not pal:<br/>
break <br/>
lap = do_word ( pal ) <br/>
sock. send ( lap ) <br/>
sock. close ( )


Then, turning to this improvised server, put down parts of speech for previously found words. All this was done in the evening, and did not cause much difficulty.

After that, the simplest thing is to collect statistics. I collected it for 2 months, after which I got tired. What became a bottleneck is difficult to say. The search for a part of speech is the words ~ 0.0046 sec, the rest of the operations are also sane, since simply the words were quickly assembled. Sedna was also tested at the previous stages, and of course it didn’t fly, but the words could be collected from it in a couple of hours, hence its performance would be enough to collect statistics on the proposals.
The result was the following data:



Etc.
supplemented
According to (this data is not only Flibusta, there is also a bit of other texts mixed up)

TOP		1500 and less		150 and less		15 and less
not	60300	window sill	1500	nazis	150	Eich	15
I	53847	wooden	1500	crooked	150	Lanter	15
Yes	50813	threw off	1499	Jure	150	Butovsan	15
what	48819	by names	1499	the afterlife	150	iksaytov	15
So	45926	grew up	1499	stalls	150	tragedy	15
everything	44672	high	1499	Alla	150	Atcal	15
this	42825	DEEP	1499	sheep	150	Yummy	15
he	42154	actively	1499	shifted	150	molluscore	15
you	38073	miracles	1498	lasers	150	medvyany	15
me	35976	Explain	1498	transporter	150	chronosurgeons	15
but	35900	sounded	1498	brunettes	150	Iwashura	15
him	35394	broke into	1498	schoolgirls	150	chronus accelerator	15
as	34347	a figure	1498	thieves	150	oscillations	15
to me	34078	nights	1498	scouts	150	pentarch	15
and	33836	WORKS	1498	physiological	150	Gorshin	15
she is	31373	ok	1498	poetic	150	roida	15
OK	30932	discontent	1498	kicked up	150	marshallas	15
you	30738	bucks	1497	cylinders	150	evmenarch	15
here	29969	called	1497	triple	150	Eumenarchus	15
there is	29579	turned	1497	debates	150	chronogen	15
It was	29297	rushed	1497	a minimum	150	Kovalyakh	15
but	29009	champagne	1497	characteristic	150	Yagya	15
there	28955	bomb	1497	raid	150	energy information	15
Well	28612	demanded	1496	counters	150	consortline	15
nothing	28605	thinking	1496	exiled	150	ephanalysis	15
a business	27653	delighted	1496	genetic	150	pisspheres	15
same	26973	inspect	1496	Anniversary	150	foreing	15
here	26898	reached	1495	old	150	Reflesians	15
still	26640	of sources	1495	wasteland	150	Reflesians	15
her	26556	turned	1495	onion	150	Slayer	15
person	26415	got up	1495	by generator	150	Levikov	15
now	26408	pulled out	1495	unpretentious	150	Glenke	15
need to	25657	gray	1495	wrappers	150	Irrashi	15
of course	25568	by misfortune	1494	mouthwatering	150	camra	15
why	25216	looked at	1494	turned	150	Arvaroche	15
can	25012	amazingly	1494	tablets	150	Uanduk	15
we	24847	uncles	1493	interception	150	chefattashe	15
I KNOW	24843	checks	1493	cockroaches	150	Tarkhanov	15
they	24070	invitations	1493	scanty	150	blackhead	15
you	23922	heavy	1493	glade	150	Ornumhoniorov	15
at	23600	intervene	1493	unprecedented	150	Ganfal	15
time	23598	sure	1493	intended	150	Hamelinam	15
their	22956	have seen	1493	profits	150	Uraine	15
you	22780	continue	1492	feeding	150	Ellata	15
that	22730	objected	1492	commenting	150	Uraina	15
time	22508	whole	1492	chic	150	gervrites	15
Now	22488	Eibogu	1492	enterprises	150	Harrens	15
not	22450	I'll start	1492	languishing	150	giazir	15
you	22107	snapshot	1492	Drozdov	150	Cutletkin	15
his	22014	Understand	1491	valid	150	black magic	15
said	21970	meeting	1491	will obey	150	Durnevu	15
us	21950	in heat	1491	foggy	150	Cutletkin	15
eyes	21763	jump	1491	departure	150	Spurius	15
Who	21740	imagine	1491	Indifferent	150	ROBUSGROBUS	15
myself	21656	riddle	1491	habitual	150	Yoha	15
will be	21642	Vasiliy	1491	referred to as	150	Tetlucoacle	15
one	21576	sounds	1490	valerian women	150	Yaraat	15
then	21438	connected	1490	looked like	150	maglody	15
be	21415	quietly	1490	cores	150	Ligul	15
people	21368	slower	1490	responding	150	Lamas	15
a life	21322	of yesterday	1489	painted	150	Varnana	15
true	20972	thin	1489	audible	150	Beliria	15
with	20645	told	1489	ladders	150	Nimrobec	15
only	20603	dived	1489	waterways	150	Dunk	15
of life	20517	return	1489	rampant	150	Legiara	15
about	20516	flag	1489	civilizations	150	Charlock	15
whether	20450	up	1489	scandalous	150	Midwinter	15
more	20328	call	1489	Englishman	150	Bow	15
to tell	20318	fidelity	1489	protested	150	Zorik	15
to myself	20299	replaced	1488	strangle	150	outland	15
than	20291	serious	1488	round	150	Bjart	15
then	20244	change	1488	saucepan	150	Renaldo	15
it is better	20047	Vladimir	1488	then	150	Samonenko	15
you	20024	the atmosphere	1487	mocking	150	Dyushka	15
farther	19819	ran away	1487	extreme	150	Morphichev	15
Where	19600	opposite	1487	roar	150	Varykin	15
arms	19545	a raincoat	1487	devilish	150	Gryzach	15
do	19525	snorted	1487	meat	150	Conch	15
if a	19522	the eldest	1487	slippers	150	Growlis	15
on	19444	was starting	1487	have called	150	Lebedinskaya	15
years old	19444	to drag	1487	mineral	150	Valyushok	15
houses	19361	a pair	1487	legends	150	I wanted	15
him	19096	to retreat	1487	slammed	150	Schaaaas	15
the words	19026	incomprehensible	1487	so	150	Gugim	15
can	18966	affably	1487	adventurers	150	glongov	15
of this	18860	widow	1487	romantic	150	Oberporuchik	15
day	18772	made a mistake	1486	sheepskin coat	150	Beckham	15
what for	18730	have given away	1486	touching	150	In which	15
people	18680	gave it away	1486	cured	150	Kutevanova	15
also	18583	we will wait	1486	painted	150	kopach	15
was	18450	fresh	1486	responsible	150	Strobach	15
would	18438	fresh	1486	cultural	150	Zonova	15
means	18361	a photo	1486	mixing up	150	Portfolio	15
human	18312	slave	1486	will read	150	Lael	15
by him	18293	lanterns	1485	intimate	150	Bepe	15
question	18276	furnishings	1485	food	150	clastapug	15
backwards	18186	having time	1485	unworthy	150	Kupling	15
us	18167	behavior	1485	Shudder	150	Kuplinga	15
or	18155	the subject	1485	new arrivals	150	Tables	15
such a	18116	result	1485	collar	150	Power	15

A total of 90 611 059 sentences were processed.

Forgetting to immediately make an index on a numeric field, I ran into a serious problem. After getting a table of 58 million records and a size of 12 GB, the index was built on it for more than a day and was never built. Bailed out
myisam_sort_buffer_size=1024MB
installed in 1 GB, the index was built in hours.

Ps. Server configuration: AMD Athlon (tm) II X4 635 Processor, 16 gb DDR3, WDC WD7500AACS-00D6B1