Patties in distribution semantics

For several months I am looking curiously towards distributive semantics - I got acquainted with the theory, learned about word2vec , found the appropriate library for Python (gensim), and even got a model of lexical vectors formed from the national corpus of the Russian language. However, for creative immersion in the material there was not enough soul-collecting data that would be interesting to twist through distributional semantics. At the same time, he enthusiastically read poems-pies (a kind of synthesis of snooty chastooshkas and profound hockey) - some even memorized and recalled their acquaintances on the occasion. And finally, enthusiasm and curiosity found each other, giving rise to an inspiring idea in the associative depths of consciousness - why not combine the pleasant with the useful and not collect some “poetic” search engine based on the improvised means.

from false conclusions
we can lay the truth
about how to multiply
two negative numbers

The “poetry” of the search was supposed to be realized due to the innate ability of distribution vectors to show the degree of semantic similarity of lexemes with a real number (the smaller the angle between word vectors, the more likely these words are close in meaning - cosine measure , classic of the genre, in general ). For example, the "princess" and "shepherd" are much less close than the "shepherd" and "sheep": 0.139 against 0.603, which is probably logical - the vectors of the national corps should reflect the harsh reality, and not the fairy-tale world of G.Kh. Andersen. The method of calculating the depth of correlation (diffusion) of the query and pie appeared almost by itself (cheap and angry) as a normalized sum of similarities of each word from list X with each word of list Y (stop words were thrown out, all others were reduced to normal form, but about this later).


The results of the poetic search and pleased, and amused. For example, the following poem-list was issued for the request “music”:

 [('  ' '   ' '    ' '   ', 0.25434666007036322), ('   ' '    ' '    ' '   ', 0.19876923472322899), ('    ' '    ' '    ' '   ', 0.19102709737990775), ('   ' '  ' '     ' '    ', 0.15292901301609391), ('   ' '   ' '   ' '', 0.14688091047781876)] 

It is noteworthy that the word "music" is not in one pie, from those entered into the base. However, all pie associations are very musical and the degree of their semantic diffusion with the query is rather high.
')
Now, in order of the work done ( source code on GitHub).

Libraries and Resources

• Module pymorphy2 - to bring words to normal grammatical form
• Module gensim - connect word2vec models for semantic processing
• A distributive model of ruscorpora lexical vectors is also required for work (320 MB)

Data model formation

Oleg presented in the form of text
everything that oksana says
broke into chapters and paragraphs
and on separate words

First of all, from a text file ( poems.txt ), which contains rhymes-pies (about eight hundred), the list is actually cut, containing, in fact, these cakes in the form of lines. Further, a bag of words ( bag of words ) is squeezed out of each string-pie, in which each word is reduced to normal grammatical form and from which noise words are kicked out. After that, for each bag, the semantic “density” (intro-diffusion) of the pie is calculated and a specific associative list is formed (it helps to understand which semantic layer prevails from the point of view of the distribution vectors model). All this pleasure fits into the dictionary under the corresponding keys and is written to a file in json format.

General analysis of the model

The most "hard" pie:

kill offense anger and lust
pride envy and longing
and what's left of you
out of mercy finish

As expected, the top of the “hard” ones included pies, in which there are many words that are close in meaning - both synonymous and so antonymic (and not so clearly) and easily generalized into some category (in this case, emotions).

The most "soft" pie:

on this ship we will be saved
don't be my ancestor
put the ship do not be fooled
sick

Here, the negative density, as it seems to me, is largely due to the fact that the hospital meaning of the word “vessel” was not included in the associative list. This is generally one of the weak points of distributional-semantic models - in them, as a rule, one value of a lexeme suppresses all the others with popularity.

Model Searches

About how the search works, in essence, it was written above - the pies are unceasingly sorted by the level of their diffusion (generalized semantic similarity) with the words of the query.


A few examples:




Obviously, a search through the prism of other distributive-semantic models (different shells, learning algorithms, vector dimensions) will give different results. In general, the technology works, and works very satisfactorily. Fuzzy semantic search is implemented easily and carelessly (at least, on a relatively small amount of data). In the future, if the hands reach and, most importantly, the head will catch up, I plan to implement an assessment of the rating of pies based on the training sample. To begin with, a simple weighted sum (semantic diffusion will act as a weighting factor). Then, perhaps, something useful from machine learning.