Some modern approaches to natural language processing
The research results obtained in recent years in the problems of speech recognition [1], machine translation [2], determination of the tone of the sentence [3] and parts of speech [4] showed the promise of deep learning neural network algorithms in comparison with the classical methods of natural language processing (natural language processing). However, in the field of question-answer and dialogue systems there are still many unsolved problems [5, 6]. This article provides an overview of the results of applying modern algorithms for processing and understanding natural language. The review contains a description of several different approaches and does not purport to be comprehensive.
Human : how many legs does a cat have?
Machine : four, i think.
Human : what do you think about messi?
Machine : he 'sa great player.
Human : where are you now?
Machine : i'm in the middle of nowhere.
(from the article A Neural Conversational Model . KDPV from the movie Ex Machina)
The solution of the problem of self-learning question-answer systems of IT companies and research organizations approach with a different vision of the initial point of reference.
')
Facebook has gone the way of defining a list of 20 specific logical operations and generating an artificial set of tasks for them (the so-called bAbi task , detailed description ). From their point of view, these operations are necessary, but not sufficient to create artificial intelligence [ 7 ]. For example, the system should be able to: respond positively or negatively to questions, answer questions based on one or several known facts, count, work with uncertainty, etc.
To solve the proposed set of tasks, an original neural network self-learning architecture with memory — Memory Networks [ 8 ] and its End-to-End implementation [ 9 ] ( code from the authors, implementation on tensor flow) were developed.
Google , developing the architecture of Neural Turing Machine [ 10 ], uses a more fundamental approach - a system that learns itself, what information and when it is necessary to write and read from memory to solve the problem.
However, the results of this approach are still less competitive in solving real problems. Neural Turing Machine solves the problem of sorting and obtaining information from memory, while operating with a small memory size of 128 cells. A little more functionality is demonstrated by the Neural Programmer [ 11 ].
The system is able to learn how to perform basic logical and arithmetic operations on a data table. The task is set as follows: there is a set of columns with data, there is a set of basic operations, and the system independently learns the necessary sequence of actions - selecting data and applying an operation to them to obtain the required solution.
The Allen Institute for Artificial Intelligence for the project to create a question-answer system about the surrounding world (ARISTO system) uses an ontological approach [ 12 ], including with the ability to train the system through user interaction [ 13 ]. The project is divided into 3 stages - the solution of tests for 4, 8 and 12 classes of the American school. If with the 4th class, more or less, we managed to cope, then for the 8th the task was not easy, and the institute decided to involve the world community of data scientists at Kaggle to solve it - The Allen AI Science Challenge .
Participants were given a training (2 500 questions) and test (8 132 questions) sample of questions in text form with 4 answers. The correct answers are known for the training set, but not for the test sample. Due to the small size, the training sample is more likely not intended to teach the system, but to be used when working on a solution to evaluate its quality in general and the degree of “covering” them with the main topics of physics, biology, geography and other subjects for the 8th grade.
The competition has a number of features ( here you can find a review-squeeze from the competition forum) - for example, the final solution should work without access to the Internet, so applying the long-awaited Google Knowledge Graph API will not work.
The table below shows a comparative review of modern approaches to creating question-answer systems prepared during the Memory and Q & A systems seminar at Deep Learning Moscow (the group has a full version of the presentation with links to sources).
* IR - information retrieval; KB - knowledge base; IE - information extraction; BiLSTM - bidirectional long-short term memory; NN - neural net; NTM - Neural Turing Machine; IGOR - Memory Networks architecture - Input feature map, Generalization, Output feature map, Response.
Earlier , a new hackathon, combined with a scientific school, DeepHack.Q & A, was already mentioned on Habré, where you can try out all the above-mentioned classical and neural network question-answer methods, as well as directly ask questions to their authors.
[1] Andrew Y. Ng et al. (2014), Deep Speech: Scaling up end-to-end speech recognition
[2] Bengio Y., Cho K., Bahdanau D. (2015), International Conference on Learning Representations 2015
[3] Blunsom, P., Grefenstette, E., Kalchbrenner, N. (2014), A Convolutional Neural Network for Modeling Sentences, The 52nd Annual Meeting of the Association for Computational Linguistics
[4] Kumar A. et al. (2015), Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
[5] T. Mikolov et al. (2015), A Roadmap towards Machine Intelligence
[6] Serban JV et al. (2015), A Survey of Available Corpora Data Building-Driven Dialogue System
[7] Jason Weston et al. (2015), Towards AI Tasks
[8] Jason Weston et al. (2015), Memory Networks
[9] Sainbayar Sukhbaatar et al. (2015) End-To-End Memory Networks
[10] Alex Graves et al. (2015) Neural Turing Machines
[11] Arvind Neelakantan et al. (2015), Neural Programmer: Inducing Latent Programs with Gradient Descent
[12] Clark P., et. al (2015), Automatic Construction of Inference-Supporting Knowledge Bases
[13] Hixon B., et. Al (2015), Learning Knowledge Graphs for Question Answering through Conversational Dialog
Human : how many legs does a cat have?
Machine : four, i think.
Human : what do you think about messi?
Machine : he 'sa great player.
Human : where are you now?
Machine : i'm in the middle of nowhere.
(from the article A Neural Conversational Model . KDPV from the movie Ex Machina)
The solution of the problem of self-learning question-answer systems of IT companies and research organizations approach with a different vision of the initial point of reference.
')
Facebook has gone the way of defining a list of 20 specific logical operations and generating an artificial set of tasks for them (the so-called bAbi task , detailed description ). From their point of view, these operations are necessary, but not sufficient to create artificial intelligence [ 7 ]. For example, the system should be able to: respond positively or negatively to questions, answer questions based on one or several known facts, count, work with uncertainty, etc.
To solve the proposed set of tasks, an original neural network self-learning architecture with memory — Memory Networks [ 8 ] and its End-to-End implementation [ 9 ] ( code from the authors, implementation on tensor flow) were developed.
Google , developing the architecture of Neural Turing Machine [ 10 ], uses a more fundamental approach - a system that learns itself, what information and when it is necessary to write and read from memory to solve the problem.
However, the results of this approach are still less competitive in solving real problems. Neural Turing Machine solves the problem of sorting and obtaining information from memory, while operating with a small memory size of 128 cells. A little more functionality is demonstrated by the Neural Programmer [ 11 ].
The system is able to learn how to perform basic logical and arithmetic operations on a data table. The task is set as follows: there is a set of columns with data, there is a set of basic operations, and the system independently learns the necessary sequence of actions - selecting data and applying an operation to them to obtain the required solution.
The Allen Institute for Artificial Intelligence for the project to create a question-answer system about the surrounding world (ARISTO system) uses an ontological approach [ 12 ], including with the ability to train the system through user interaction [ 13 ]. The project is divided into 3 stages - the solution of tests for 4, 8 and 12 classes of the American school. If with the 4th class, more or less, we managed to cope, then for the 8th the task was not easy, and the institute decided to involve the world community of data scientists at Kaggle to solve it - The Allen AI Science Challenge .
Participants were given a training (2 500 questions) and test (8 132 questions) sample of questions in text form with 4 answers. The correct answers are known for the training set, but not for the test sample. Due to the small size, the training sample is more likely not intended to teach the system, but to be used when working on a solution to evaluate its quality in general and the degree of “covering” them with the main topics of physics, biology, geography and other subjects for the 8th grade.
The competition has a number of features ( here you can find a review-squeeze from the competition forum) - for example, the final solution should work without access to the Internet, so applying the long-awaited Google Knowledge Graph API will not work.
The table below shows a comparative review of modern approaches to creating question-answer systems prepared during the Memory and Q & A systems seminar at Deep Learning Moscow (the group has a full version of the presentation with links to sources).
* IR - information retrieval; KB - knowledge base; IE - information extraction; BiLSTM - bidirectional long-short term memory; NN - neural net; NTM - Neural Turing Machine; IGOR - Memory Networks architecture - Input feature map, Generalization, Output feature map, Response.
Earlier , a new hackathon, combined with a scientific school, DeepHack.Q & A, was already mentioned on Habré, where you can try out all the above-mentioned classical and neural network question-answer methods, as well as directly ask questions to their authors.
[1] Andrew Y. Ng et al. (2014), Deep Speech: Scaling up end-to-end speech recognition
[2] Bengio Y., Cho K., Bahdanau D. (2015), International Conference on Learning Representations 2015
[3] Blunsom, P., Grefenstette, E., Kalchbrenner, N. (2014), A Convolutional Neural Network for Modeling Sentences, The 52nd Annual Meeting of the Association for Computational Linguistics
[4] Kumar A. et al. (2015), Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
[5] T. Mikolov et al. (2015), A Roadmap towards Machine Intelligence
[6] Serban JV et al. (2015), A Survey of Available Corpora Data Building-Driven Dialogue System
[7] Jason Weston et al. (2015), Towards AI Tasks
[8] Jason Weston et al. (2015), Memory Networks
[9] Sainbayar Sukhbaatar et al. (2015) End-To-End Memory Networks
[10] Alex Graves et al. (2015) Neural Turing Machines
[11] Arvind Neelakantan et al. (2015), Neural Programmer: Inducing Latent Programs with Gradient Descent
[12] Clark P., et. al (2015), Automatic Construction of Inference-Supporting Knowledge Bases
[13] Hixon B., et. Al (2015), Learning Knowledge Graphs for Question Answering through Conversational Dialog
Source: https://habr.com/ru/post/274577/
All Articles