Bot for telegram with status in DBMS and text classification

Since My classifier from the past post still works (however, the parameters out of the box are not always successful, so I rendered the opportunity to slightly adjust the Conv1d layers and the hidden layer) - I decided to fasten it to the bot. Yes, I was late for this HYIP) By the way, I’ll clarify in advance that I haven’t tried to fasten Russian yet, although this shouldn’t be a problem - the necessary features are supported in nltk, word2vec training is not conceptually different from English, and pre-trained models seem to be available .

Well, immediately there are questions:

under what platforms to cut it - while decided to stop on the telegram. In theory, the design makes it easy to add wrappers for other platforms (as if someone would need it)
how to describe the "script". Bike your structure with classes and entities on top of YAML
well, it would be nice to store bots / state in some database

Requirements

You will need:
')

pynlc . Instructions for installing aids (keras, nlkt, gensim and their dependencies) - in my last post
robot
robotframework . You need to put sqlalchemy
robotframework_telegram . Python-telegram-bot is required

To hell with details, give an example

Python code

import logging from robotframework_telegram import RobotFrameworkTelegram from config import config if __name__ == '__main__': bots = RobotFrameworkTelegram({ 'DATABASE_URI': 'URI', 'KEEP_WORD_VECTORS': 3, 'ROBOT_CONFIGURATIONS': 'robot_configurations', 'KEEP_ROBOTS': 5, 'OUTPUT_HANDLERS': {}, 'TOKENS': [ # Your tokens here ], 'NO_ANSWER_MESSAGE': 'Sorry, I can\'t answer now.', 'NO_INSTANCE_MESSAGE': 'Sorry, robot not instantiated yet. ' + \ 'You\'ll get answer after instantiation - it can take few minutes', 'TELEGRAM_WORKERS_PER_BOT': 1, 'WORKER_COUNT': 2, 'WORKER_SLEEP_TIME': 0.2, 'LOGGER': logging, }) bots.start_polling()

For the data (structure and dump of the postgres database, word2vec, the configuration of the trained bot) - here .

Script Description

In my case, the bot script can be described as a tree (well, in fact - taking into account "goto" - rather a graph), whose node can contain conditions:

about the custom text class
about the types of entities used

In addition to the tree in the script are:

data for classifier training
entity types (conditional {"city": ["Moscow", "Tokeyo", "Ottava"]})
Their synonyms (no less conditional {"Moscow": ["Default city"]})
(optional) additional parameters of the classifier. For example, in the case of my “weather” example, the network configuration, which will be built according to standard parameters - the worst overkill, because I reduced the number of sizes of filters and neurons of the output layer.

for example

a bunch of yaml

 script: text: Hello. I'm weather robot. How can I help you? name: main children: - name: city_conditions text: "Conditions in {%print $city%} is next." conditions: class: conditions entity_variables: city: $city - name: city_temperature text: "Temperature in {%print $city%} is next." conditions: class: temperature entity_variables: city: $city - name: conditions text: Okay, type city name. conditions: class: conditions children: - name: conditions_city_selected text: "{%goto city_conditions%}" conditions: entity_variables: city: $city - name: temperature text: Okay, type city name. conditions: class: temperature children: - name: temperature_city_selected text: "{%goto city_temperature%}" conditions: entity_variables: city: $city entities: city: - Moscow - Tokyo - Ottava synonyms: Moscow: - default city - DC classes: temperature: - How hot is it today? - Is it hot outside? - Will it be uncomfortably hot? - Will it be sweltering? - How cold is it today? - Is it cold outside? - Will it be uncomfortably cold? - Will it be frigid? - What is the expected high for today? - What is the expected temperature? - Will high temperatures be dangerous? - Is it dangerously cold? - When will the heat subside? - Is it hot? - Is it cold? - How cold is it now? - Will we have a cold day today? - When will the cold subside? - What highs are we expecting? - What lows are we expecting? - Is it warm? - Is it chilly? - What's the current temp in Celsius? - What is the temperature in Fahrenheit? conditions: - Is it windy? - Will it rain today? - What are the chances for rain? - Will we get snow? - Are we expecting sunny conditions? - Is it overcast? - Will it be cloudy? - How much rain will fall today? - How much snow are we expecting? - Is it windy outside? - How much snow do we expect? - Is the forecast calling for snow today? - Will we see some sun? - When will the rain subside? - Is it cloudy? - Is it sunny now? - Will it rain? - Will we have much snow? - Are the winds dangerous? - What is the expected snowfall today? - Will it be dry? - Will it be breezy? - Will it be humid? - What is today's expected humidity? - Will the blizzard hit us? - Is it drizzling? classifier_params: filter_sizes: [1] hidden_size: 100 nb_filter: 150 min_class_confidence: 0.8

Switching from one state to another (and at the same time - extracting entities) to another happens like this:

classify input. We retrieve the class with the greatest "similarity". We add its name before the user text. For example - “How cold in Moscow now?” Is converted to “temperature How cold in Moscow now?”
we get the child nodes of the current node (by default - main)
for each descendant (in order):
- if it imposes restrictions on the class - we check the entry into the class, if it does not enter - go to the next descendant
- if there are types of entities whose references need to be saved - we check the occurrences of all possible entities (and their synonyms) of this type and save them in the dictionary under the corresponding name. If not all types are found, go to the next child. In the process of a “pattern” {“city”: "$ city"} from "what temperature expected in Moscow and Ottava" {"$ city" will be extracted: [“Moscow”, “Ottava”]}

Under the hood, part 1

Awful bitbucket code . For starters - singled out such entities:

Classifier With him, probably, everything seems to be clear
Scenario. It contains complete information about scripts - a tree, a list of classes, a list of entity types, a list of their equivalent names, etc. robot / script.py
Script tree node. Contains data about 1 node - text template, conditions (including class and entities), child elements. robot / script.py
The state of the robot. Includes the name of the current node and the variables selected from user input. robot / robot_state.py
Output handler It can be used, for example, to display the value of a variable (print), go to another node (goto). robot / output_processing.py
Robot. Actually, it connects the classifier, the script and the output handler. The input has its own state and text, and the output has a new state. robot / robot.py

Now this is possible:

 script = Script.load(os.path.join(os.path.dirname(__file__), "script.yaml")) text_processor = TextProcessor("english", [["turn", "on"], ["turn", "off"]], Word2Vec.load_word2vec_format( os.path.join(os.path.dirname(__file__), "100d.txt") )) robot = Robot(script, text_processor) state, output = robot.output() self.assertEqual(output, "Hello. I'm weather robot. How can I help you?") self.assertIsNotNone(state) self.assertEqual(state.stage, "main") self.assertEqual(state.variables, {}) state, output = robot.answer(state, "How cold it'll be in Moscow today?") self.assertIsNotNone(state) self.assertEqual(state.stage, "city_temperature") self.assertEqual(state.variables, {"$city": ["Moscow"]}) self.assertEqual(output, "Temperature in Moscow is next.")

You can already try to run something, but I went further.

Wrapper around robot and sqlalchemy. Yet without telegram

Even more awful bitbucket code . Again we divide entities:

Tongue. Contains a name for nltk and a set of exceptions for cutting stop words. robotframework / models / language.py
Subject area. Contains a link to the language and word2vec, which is trained on the texts of the desired subject area in this language (will be loaded into memory at the first call and unloaded if more than the specified number of word2vec-in is already loaded). robotframework / models / domain.py
Robot. Contains a reference to the subject area, a script and a reference to the classifier's config (again, the classifier will be created upon the first call. If the classifier is not trained, it will be pre-trained and saved. Unloaded if the corresponding parameter is exceeded). robotframework / models / robot.py
User. All that is needed from him here is to be uniquely identifiable. robotframework / models / user.py
Dialogue. Just refers to the user and the robot. robotframework / models / conversation.py
Record dialogue. It contains its source (robot / user), text and status of the robot. robotframework / models / conversation_item.py
The app is a singleton with all sorts of technical functionality. Now - initializes sqlalchemy

Something like this:

Code example:

 from robotframework import * from sqlalchemy import select app = Application({ 'DATABASE_URI': '', 'KEEP_WORD_VECTORS': 4 }) robot = Robot.filter(lambda query: query.where(Robot.id == 1))[0] user = User.filter(lambda query: query.where(User.id == 1))[0] conversation = robot.converse(user) #print(robot.instance) items = lambda: print([str(item) for item in conversation.items]) items() conversation.output() conversation.answer('How hot it\'ll be in Moscow today?') conversation.answer('Okay, turn off') items()

Well, as a result:

 [] ["Hello. I'm weather robot. How can I help you?", "How hot it'll be in Moscow today", "Temperature in Moscow is next", ""]

The last entry is empty, because in the script there is no node on which to go in this case.

In the configuration - the following options:

 { 'DATABASE_URI' : '', # URI   sqlalchemy   'KEEP_WORD_VECTORS': 3, #       "" word2vec,    ,   ()     'KEEP_ROBOTS': 3, #     'ROBOT_CONFIGURATIONS': 'robot_configurations', # ,     , 'OUTPUT_HANDLERS': {} }

Now we seem to have all the means for more or less adequate storage of robots, their states and auxiliary data. And - the ability to fasten them to different messengers (and not only), screwing the appropriate wrapper.

Before viewing the code is recommended stock up on sedation . The wrapper adds a couple more fields to the models (telegram_chat_id in Conversation, telegram_bot_id in Robot). Well, multithreading in all fields.

Here is an example:

 import logging from robotframework_telegram import RobotFrameworkTelegram from config import config if __name__ == '__main__': bots = RobotFrameworkTelegram({ 'DATABASE_URI': 'URI', 'KEEP_WORD_VECTORS': 3, 'ROBOT_CONFIGURATIONS': 'robot_configurations', 'KEEP_ROBOTS': 5, 'OUTPUT_HANDLERS': {}, 'TOKENS': [ # Your tokens here ], 'NO_ANSWER_MESSAGE': 'Sorry, I can\'t answer now.', 'NO_INSTANCE_MESSAGE': 'Sorry, robot not instantiated yet. ' + \ 'You\'ll get answer after instantiation - it can take few minutes', 'TELEGRAM_WORKERS_PER_BOT': 1, 'WORKER_COUNT': 2, 'WORKER_SLEEP_TIME': 0.2, 'LOGGER': logging, }) bots.start_polling()

And the dialogue obtained during the test

[7:14:55 AM]
/ start
[7:14:56 AM] robotframework_demo_weather:
Sorry, robot not instantiated yet. You'll get answer after instantiation - it can take a few minuutes
Hello. I'm weather robot. How can I help you?
[7:16:07 AM] #:
It's cold?
[7:16:08 AM] robotframework_demo_weather:
Okay, type city name.
[7:16:11 AM] #:
Moscow
[7:16:11 AM] robotframework_demo_weather:
Temperature in Moscow is next.

[7:40:25 AM] #:
/ start
[7:40:26 AM] robotframework_demo_weather:
Hello. I'm weather robot. How can I help you?
[7:40:48 AM] #:
What is your destination, robot? Seems like you must not answer
[7:40:48 AM] robotframework_demo_weather:
Sorry, I can't answer now.

Source: https://habr.com/ru/post/316838/

All Articles