Chat bot algorithms based on recurrent neural network and AIML language extensions
Today, the creation of programs simulating human communication remains relevant. The simplest model of communication is the database of questions and answers to them [1]. In this case, there is the problem of describing the knowledge base and the implementation of the interpreter program. The markup language of the knowledge base can include question patterns and corresponding response patterns, as well as the background history of the dialogues to them and the name of the corresponding communication topic.
Chat bot can perform additional functions, such as music search, pictures, facts, calculator, weather forecast, display of exchange rates. Most of these functions have an implementation on the Internet and are available as an external API.
An alternative option for creating a virtual interlocutor program is to use machine learning algorithms based on communication dialogues, namely, artificial neural networks. A suitable ANN model is a recurrent neural network capable of storing, synthesizing, and predicting various sequences. In this paper, we propose using the corresponding words in the knowledge and question knowledge base as elements of the sequence.
One of the knowledge markup formats is the AIML (Artificial Intelligence Markup Language) markup language standard. The key words in the language are category, pattern and template:
')
The category tag is the parent of the pattern and template tags that hold the question and answer templates. The random tag allows you to specify multiple answers to the question, selected by the interpreter randomly. The paper proposes to introduce additional tags relevant stories and topics of conversation.
Several pattern tags allow you to describe different versions of questions that correspond to a given category, to which the same answers should follow. The history tag stores the history of the dialog that precedes this question. The theme tag stores the name of the conversation topic. These tags allow the interpreter to select the question pattern corresponding to the pre-history of the dialogue and the topic of communication, which should have an impact on improving the quality of communication imitation by the chat bot.
The markup interpreter should allow finding the most relevant questions in the following complementary ways:
The corresponding algorithm for selecting the best match is formed on the basis of the sorting algorithm:
As a result, the first element of the sorted array of matches is selected.
To improve the quality of the search for relevant answers, a morphological analyzer module was introduced in the word comparison procedure, which allows finding the basic forms of words. Thus, the comparison is based on the basic forms of words, which eliminates the inconsistencies of words associated with their inclinations.
It is proposed to use an algorithm for determining the names of categories when only a part of them is defined.
A recurrent neural network is a type of multilayer perceptron in which signals from neurons of the output layer arrive at additional neurons of the input layer, the so-called. context neurons.
The input signal vector is fed to the INPUT group of neurons, the zero signal on the CONTEXT neuron group. Then the signal propagates into the group of neurons of the hidden layer HIDDEN, and then is transformed by them and falls on the neurons of the output layer OUTPUT. At the next iteration, along with the INPUT signal vector, copies of the signals from the output OUTPUT layer of the previous iteration arrive at the context group of neurons (Fig. 1).
Fig. 1. General view of the structure of the recurrent neural network.
The structure of the recurrent neural network for memorizing sentences is as follows:
The CONTEXT, INPUT and OUTPUT layers each have one neuron, the signal values at the output of which correspond to the index of a word in a set of words. Additionally, the word __end__ is entered corresponding to the end of the sentence [2]. The network is consistently trained in sentences of the form:
"Hello. How are you? __end__ Hi. Fine. __end__ "
Receiving answers to questions by a recurrent neural network is as follows (Fig. 2).
Fig. 2. Getting an answer to a question using a recurrent neural network.
The volume of HIDDEN layers should allow you to remember the entire set of sentences The network is trained in the back propagation method.
The chat bot program was implemented as an Android application. The application has several chat bot modes:
As a morphological analyzer, a free JavaScript-library for processing texts in Russian Az.js was used.
To create and train a recurrent neural network, the free JavaScript library RecurrentJS was used.
In all modes, automatic translation of answers into the user's question language is available based on the Yandex Translate API and Bing Translate API services. Also, as the external services, the image search service Custom Search API and Bing Image Search API are used. Knowledge Search is based on the Google Knowledge Graph Search API. Music search is based on the SoundCloud API. Calculator, weather forecast, exchange rate, time implemented on the basis of Wolfram | Alpha API.
ChatBot is available on Google Play at https://play.google.com/store/apps/details?id=svlab.chatbot .
Algorithms for building a chat bot based on AIML advanced markup and a recurrent neural network are considered. The AIML markup extension includes new theme and history tags to more effectively search for relevant questions and answers in the context of the dialogue. To determine the names of categories, when only their part is defined, it is proposed to use a heuristic algorithm for classifying topics. Using the module of the morphological analyzer of individual words and bringing them into the basic form allows you to improve the quality of the search for relevant answers. A recurrent neural network allows you to get answers to questions that were not in the knowledge base, using the network's ability to generalize. The ChatBot app is available for the Android platform in the Play Store.
Chat bot can perform additional functions, such as music search, pictures, facts, calculator, weather forecast, display of exchange rates. Most of these functions have an implementation on the Internet and are available as an external API.
An alternative option for creating a virtual interlocutor program is to use machine learning algorithms based on communication dialogues, namely, artificial neural networks. A suitable ANN model is a recurrent neural network capable of storing, synthesizing, and predicting various sequences. In this paper, we propose using the corresponding words in the knowledge and question knowledge base as elements of the sequence.
AIML
One of the knowledge markup formats is the AIML (Artificial Intelligence Markup Language) markup language standard. The key words in the language are category, pattern and template:
')
<aiml> <category> <pattern>! ?</pattern> <template> <random> <li>. .</li> <li>. . ?</li> </random> </template> </category> </aiml> The category tag is the parent of the pattern and template tags that hold the question and answer templates. The random tag allows you to specify multiple answers to the question, selected by the interpreter randomly. The paper proposes to introduce additional tags relevant stories and topics of conversation.
<aiml> <category> <pattern> ?</pattern> <pattern> ?</pattern> <template> <random> <li> .</li> <li> .</li> </random> </template> <history>! ? . .</history> <theme></theme> </category> </aiml> Several pattern tags allow you to describe different versions of questions that correspond to a given category, to which the same answers should follow. The history tag stores the history of the dialog that precedes this question. The theme tag stores the name of the conversation topic. These tags allow the interpreter to select the question pattern corresponding to the pre-history of the dialogue and the topic of communication, which should have an impact on improving the quality of communication imitation by the chat bot.
The markup interpreter should allow finding the most relevant questions in the following complementary ways:
- 1. Search the entire phrase of a question based on a regular expression.
- 2. Search by the number of matching words in the question and patterns.
- 3. Search by current topic matching and category topics.
- 4. Search by the number of matching words in the current conversation history and category history.
The corresponding algorithm for selecting the best match is formed on the basis of the sorting algorithm:
sortMatches = allMatches.sort(function(a, b) { if(a.pattern == inputText && b.pattern != inputText) return -1; if(b.pattern == inputText && a.pattern != inputText) return 1; if(a.matches < b.matches) return 1; if(a.matches > b.matches) return -1; if(a.theme == bot.theme && b.theme != bot.theme) return -1; if(b.theme == bot.theme && a.theme != bot.theme) return 1; if(a.historyMatches < b.historyMatches) return 1; if(a.historyMatches > b.historyMatches) return -1; return 0; }) As a result, the first element of the sorted array of matches is selected.
To improve the quality of the search for relevant answers, a morphological analyzer module was introduced in the word comparison procedure, which allows finding the basic forms of words. Thus, the comparison is based on the basic forms of words, which eliminates the inconsistencies of words associated with their inclinations.
Theme classification algorithm
It is proposed to use an algorithm for determining the names of categories when only a part of them is defined.
- 1. Let the set of categories in which the themes are defined are T. The set of categories in which the themes are not defined are D. The elements of these sets are associated with the union of the strings of all the values of patterns and category templates.
- 2. A threshold of determination is introduced, for example, p = 70%.
- 3. The elements of the set D. are sequentially sorted out. A subset V of the elements of the set T is selected, in which p is the percentage of unique words defined.
- 4. If V is empty, then the topic is determined by the string of the element of the set D.
- 5. If V is not empty, then an element is selected from V for which the ratio of the number of matching unique words to the total number of unique words in the string of this element is maximum. The corresponding topic of this element V is defined as the topic of element D.
Recurrent neural network
A recurrent neural network is a type of multilayer perceptron in which signals from neurons of the output layer arrive at additional neurons of the input layer, the so-called. context neurons.
The input signal vector is fed to the INPUT group of neurons, the zero signal on the CONTEXT neuron group. Then the signal propagates into the group of neurons of the hidden layer HIDDEN, and then is transformed by them and falls on the neurons of the output layer OUTPUT. At the next iteration, along with the INPUT signal vector, copies of the signals from the output OUTPUT layer of the previous iteration arrive at the context group of neurons (Fig. 1).
Fig. 1. General view of the structure of the recurrent neural network.
The structure of the recurrent neural network for memorizing sentences is as follows:
The CONTEXT, INPUT and OUTPUT layers each have one neuron, the signal values at the output of which correspond to the index of a word in a set of words. Additionally, the word __end__ is entered corresponding to the end of the sentence [2]. The network is consistently trained in sentences of the form:
"Hello. How are you? __end__ Hi. Fine. __end__ "
Receiving answers to questions by a recurrent neural network is as follows (Fig. 2).
Fig. 2. Getting an answer to a question using a recurrent neural network.
The volume of HIDDEN layers should allow you to remember the entire set of sentences The network is trained in the back propagation method.
Software implementation
The chat bot program was implemented as an Android application. The application has several chat bot modes:
- 1) based on external API www.pandorabots.com ;
- 2) on the basis of the original generalization of the AIML language;
- 3) based on a recurrent neural network.
As a morphological analyzer, a free JavaScript-library for processing texts in Russian Az.js was used.
To create and train a recurrent neural network, the free JavaScript library RecurrentJS was used.
In all modes, automatic translation of answers into the user's question language is available based on the Yandex Translate API and Bing Translate API services. Also, as the external services, the image search service Custom Search API and Bing Image Search API are used. Knowledge Search is based on the Google Knowledge Graph Search API. Music search is based on the SoundCloud API. Calculator, weather forecast, exchange rate, time implemented on the basis of Wolfram | Alpha API.
ChatBot is available on Google Play at https://play.google.com/store/apps/details?id=svlab.chatbot .
Conclusion
Algorithms for building a chat bot based on AIML advanced markup and a recurrent neural network are considered. The AIML markup extension includes new theme and history tags to more effectively search for relevant questions and answers in the context of the dialogue. To determine the names of categories, when only their part is defined, it is proposed to use a heuristic algorithm for classifying topics. Using the module of the morphological analyzer of individual words and bringing them into the basic form allows you to improve the quality of the search for relevant answers. A recurrent neural network allows you to get answers to questions that were not in the knowledge base, using the network's ability to generalize. The ChatBot app is available for the Android platform in the Play Store.
Literature
- 1. Provar A.I., Klochko K.A. Features and problems of virtual communication using chat bots // Scientific works of Vinnitsa National Technical University. 2013. № 3. S. 2.
- 2. Oriol Vinyals, Quoc Le A Neural Conversational Model // arXiv preprint arXiv: 1506.05869, 22 Jul 2015.
Source: https://habr.com/ru/post/304284/
All Articles