Heuristic network - analogue of the recurrent neural network for chat bot program

The article presents the heuristic network algorithm for some properties similar to the recurrent neural network for the program of the virtual interlocutor. The algorithm has been improved using the explanatory dictionary of the Russian language. A generator of new answers based on statistical information of the knowledge base has been introduced into the heuristic network.

Introduction

When training a recurrent neural network on large amounts of data, there is a problem of rewriting and the lack of quality network training, as well as long training and the complexity of changing the network to introduce new data [1]. Therefore, for the chat program, it is important to create a heuristic network for the properties of a similar recurrent network, in particular, the properties of word interchangeability and the statistical connection of questions and answers from the knowledge base. The property of the recurrent neural network for generating new texts is implemented as a separate procedure of the heuristic network.

Often, in the understanding of incoming user questions, the network is limited in a small knowledge base, therefore, the introduction of an explanatory dictionary of the Russian language to display the original words into words is probably relevant to the values ​​possibly contained in the knowledge base questions.

The heuristic network algorithm is implemented in the new modes of the ChatBot program, created earlier using the AIML knowledge base markup [2].
')

Recurrent neural network

A recurrent neural network is a type of multilayer perceptron in which signals from neurons of the output layer arrive at additional neurons of the input layer, the so-called. context neurons [1, 2]. A recurrent network to simulate speech communication has a number of properties:

1. statistical relationship of questions and answers in the knowledge base,
2. interchangeability of words
3. generation of new texts.

Since during the training of a recurrent neural network on large amounts of data, there is often a problem of rewriting and poor-quality training it is possible to implement the basic properties of a recurrent neural network in the form of heuristics.

Heuristic network

The property of the statistical relationship of questions and answers in the knowledge base and the selection of the appropriate answer from the knowledge base to a user question can be implemented based on the following algorithm:

1. All questions in the knowledge base denote the word patterns .
2. The corresponding answers in the knowledge base with the word templates .
3. Each question and the corresponding answer is broken down into an array of words: pwords and twords .
4. Calculate correlations between unique words:
   If the word tword is found in the array of words of the question twords along with the word pword in the array of words of the answer pwords , then the correlation functions between the words tword and pword are added 1:
   $$$R \ left (tword, pword \ right)$+ = 1.
5. A new user question is denoted by the word question .
6. User question is divided into words qwords .
7. Templates from the knowledge base are searched for in order to find the maximum amount of correlations between words from the twords answer and the words of the user question qwords :
   $$$\ mathop {\ max} \ limits_ {k} \ left [\ sum _ {i} \ sum _ {j} R \ left (twords_ {k} [i], qwords [j] \ right) \ right]$where the index k runs through all the templates in the knowledge base, i and j are word indices in word arrays.
8. To reduce the influence of common words, the search for maxim can be limited from the amounts not exceeding a certain threshold porog .

Let us give an example of the property of interchangeability of words when searching for an answer to a user question. For example, the following AIML structure in the knowledge base:

<category> <pattern>  ?</pattern> <template> <random> <li> </li> </random> </template> </category> <category> <pattern>  </pattern> <template> <random> <li>   </li> </random> </template> </category> 

It should lead to the answer “Yes, I love” to the user question “Do you love women ?”. Those. the words "peace" and "women" are becoming interchangeable.

The corresponding algorithm for accounting for the interchangeability of words from the knowledge base can be implemented in the following form:

1. Introduces an internal correlation function between words:
   For this word from the array of words of the question pwords and answer twords of the knowledge base is combined into an array of ptwords .
   Introduces the function
   $$$r_ {k} \ left (ptwords [i], ptwords [j] \ right) = \ frac {\ beta} {j-i}$where j> i,
   $$$\ beta = \ left \ {\ begin {array} {c} {1, \ begin {array} {ccc} {} & {if} & {i, j \ le g \ begin {array} {ccc} { } & {and} & {i, j \ ge g} \ end {array}} \ end {array}} \\ {0.25, \ begin {array} {ccc} {} & {if} & {\ begin { array} {ccc} {} & {i \ le g} & {\ begin {array} {ccc} {} & {and} & {j \ ge g} \ end {array}} \ end {array}} \ end {array}} \ end {array} \ right.$ ,
   g ̶ length of the pwords array.
   Then the function of internal correlation between words $$$R_ {in} \ left (tword, pword \ right)$defined either as a sum of functions $$$r_ {k}$, or as their maximum , or as their average for all categories from the knowledge base.
2. For the word qword , all ptword words are selected from the user question, correlating with it by the function $$$R_ {in} \ left (qword, ptword \ right)$. The choice of the answer from the knowledge base affects the user question with the coefficient α the maximum of the sum of the functions of the internal correlation between words $$$R_ {in} \ left (qword, ptword_ {k} \ right)$where the index k runs over all pairs a question plus the corresponding answer from the knowledge base.

The generation of new text can be implemented using the following algorithm:

1. Sets the number of influence words k .
2. Served as input k words $$$\ left \ {i_ {1}, i_ {2}, \ ldots, i_ {k} \ right \}$ and a new word s is searched for with the highest amount $$$\ sum _ {j = 1} ^ {k} \ frac {1} {k-j + 1} R_ {in} \ left (s, i_ {j} \ right)$. Function $$$R_ {in}$must take into account that the word s stood in the sentence before the word $$$i_ {j}$. Function $$$R_ {in}$calculated on type sentences $$$pattern_ {i}$+ "" + $$$template_ {j}$+ "__End__", where $$$pattern_ {i}$and $$$template_ {j}$taken from the same category, and the word “__end__” symbolizes the end of the training sentence.
3. After a new word s has been found, k incoming words are replaced with words $$$\ left \ {i_ {2}, \ ldots, i_ {k}, s \ right \}$ and looking for a new word in step 2.
4. The process of searching for new words is completed when the word “__end__” is found, or the repetition of a sequence of words begins (sequences of types 1, 2, 3, 4, 5, 4, 5 ), or the required number of words d is exceeded.

The algorithm for generating a new text resembles an algorithm on a Markov chain. The difference here is in using the function of correlations between words, instead of transition probabilities, as well as sum-correlation functions instead of conditional probabilities for chains of kth order.

The introduction of explanatory dictionary

When in a user question there are words that were not in the knowledge base, it is possible to use the explanatory dictionary of the Russian language according to the following algorithm:

1. To make a base of definitions of words from the explanatory dictionary.
2. Expand all words from the user question by definition. Those. instead of each word from the user question, take the word sets from the definitions.
3. All words found in the question pattern from the knowledge base and corresponding to the words of the user question should be evaluated by a coefficient, for example, 100. All words of a user question corresponding to the 1st level of definitions should be estimated by a factor, for example, 50.
4. Find the sum of such estimates and choose such a pattern from the knowledge base, which scored the maximum.
5. Answer template from the knowledge base corresponding to the pattern found question, to take as an answer to a user question.

Morphological analyzer

It is possible to use a morphological analyzer to bring words into the basic form and improve the quality of all algorithms for simulating speech communication.

Software implementation

The chat bot program was implemented as an Android application with an age limit of 18+. In the application there are several new chat bot modes based on the heuristic network.
As a morphological analyzer , a free JavaScript-library for processing texts in Russian Az.js was used .

The application is available at https://play.google.com/store/apps/details?id=svlab.chatbot2 .

Conclusion

The chat bot modes are programmatically implemented based on a heuristic network simulating the properties of a recurrent neural network. The advantage of the heuristic network is fast learning based on indexing and the ability to simply add new knowledge to the network structure. The algorithm has been improved with the help of the explanatory dictionary of the Russian language for words not represented in the knowledge base. The ChatBot app is available for the Android platform in the Play Store.

Literature

1. Oriol Vinyals, Quoc Le A Neural Conversational Model // arXiv preprint arXiv: 1506.05869, 22 Jul 2015.
2. Chovin V.A. ChatBot program - chat bot or virtual interlocutor. // Mathematical structures and modeling. 2016. № 4 (40). Pp. 96-101.