IBM Watson cognitive system: principles of working with natural language

IBM Watson is one of the first cognitive systems in the world. This system can do a lot, thanks to which the capabilities of Watson are used in many areas - from cooking to predicting accidents in human settlements. In general, most of the capabilities of Watson are not something unique, but together they all represent a very powerful tool for solving a variety of issues.

For example, natural language recognition, dynamic learning of the system, construction and evaluation of hypotheses. All this allowed IBM Watson to learn how to give direct correct answers (with a high degree of confidence) to the questions of the operator. In this case, the cognitive system can use to work large arrays of global unstructured data, Big Data. What are the basic principles of the IBM Watson language? About this - in the sequel.

The main difficulties of natural language recognition

For a person, language is a means of expressing thought. We use language to convey our opinions, any data and information. We can make predictions and formulate theories. It is language that is the cornerstone of our consciousness. At the same time, here is a paradox, the language of man is very inaccurate.
')
Many terms are illogical, and computer systems can be very difficult to understand. For example, how can a thin voice be? How can you burn with shame? For a car, this is a problem, for a person it is quite a common thing. The fact is that for the correct answer to a question in many cases it is necessary to take into account the existing context. In the absence of sufficient factual information, it is difficult to correctly answer a question, even if you can find the exact answer to the elements of the question in a literal sense.

Natural language processing - the beginning

Many computer systems are capable of analyzing language, but superficial analysis is also performed. This may make sense, for example, in order to put a statistically valid estimate of trends in emotions over large amounts of information. Here, the accuracy of information transfer is not too important, because if even if we assume that the number of erroneously positive results is approximately equal to the number of erroneously negative results, then they compensate each other.

But if all cases are important, then systems that work with the superficial analysis of a language can no longer do their job normally. A clear example of this can be a task for the voice assistant of any mobile device. If you say “find me a pizza”, the assistant will display a list of pizzerias. If you say "do not look for me a pizza in Madrid," for example, the system will still search. Such systems work by identifying certain keywords and using a specific set of rules. The result may be exact in a given system of rules, but incorrect.

Deep processing of natural language

In order to teach the system to analyze complex semantic constructions, taking into account emotions and other factors, specialists used deep processing of natural language. Namely, the question-answer system of content analytics (Deep Question * Answering, DeepQA). If greater precision is required, then additional methods of processing natural language have to be used.
IBM Watson is a natural language deep processing system. When analyzing a specific question, in order to give the correct answer, the system tries to evaluate the broadest possible context. It uses not only the information of the question, but also the data of the knowledge base.
Creating a system capable of deep processing of natural language, allowed to solve another problem - the analysis of a huge amount of information that is generated daily. This is unstructured information, such as tweets, social networking posts, reports, articles and more. IBM Watson learned to use all this for solving problems posed by man.

IBM Watson cognitive system

Watson is a different level of computing power. The system is able to separate certain statements in natural language and find links between these statements. In this case, Watson copes with the task, in many cases, even better than a person, while data processing goes much faster, work is carried out with much larger volumes - a person is simply incapable of this.

The main characteristics of the cognitive system

The system works in this order:

1. After receiving the question, Watson parses it to highlight the main features of the question.

2. The system generates a series of hypotheses by looking at the corpus in search of phrases, which with a certain degree of probability may contain the necessary answer. In order to conduct an effective search in streams of unstructured information, we need completely different computational capabilities * they are called cognitive systems. (I do not really understand the last sentence and the role of the asterisk)

3. The system performs in-depth comparisons of the language of the question and the language of each of the possible answers, using different inference algorithms.

This is a difficult stage. There are hundreds of logical inference algorithms, and they all perform different comparisons. For example, some search for matching terms and synonyms, while others look at temporal and spatial features, while others analyze suitable sources of contextual information.

4. Each inference algorithm gives one or several estimates, which show the extent to which a possible answer follows from a question, in the area considered by this algorithm.

5. Each obtained estimate is then assigned a weighting factor according to the statistical model, which records how well the algorithm coped with identifying logical connections between two similar phrases from this area in the “learning period” of Watson. This statistical model can be used subsequently to determine the overall level of confidence in the Watson system that a possible answer follows from a question.

6. Watson repeats the process for each possible answer until he finds answers that are more likely to be correct than others.

As mentioned above, for the correct answer to a question, the system needs to refer to additional data sources. It can be textbooks, manuals, FAQ, news and everything else. Watson processes huge amounts of information in seconds to get the right answer. In this case, the found content is also checked, outdated and useless data are eliminated.

Elements of the cognitive system

The general meaning of the text Watson derives from the information received, from the additional database. This uses the title of the document, part of the text of the document or the entire text.

Cognitive systems, their methods of collecting, storing and retrieving information are similar to how a person analyzes information. In this case, cognitive systems can transmit information and act. Here are examples of behavioral constructs that are used in this case:

- the ability to create and test hypotheses;
- the ability to break down into components and build logical conclusions about the language;
- the ability to extract and evaluate useful information (such as dates, locations and characteristics).

Without these abilities, neither the computer nor the person will be able to determine the correct relationship between questions and answers.
Cognitive processes of a higher order can achieve a high level of understanding, focusing on basic behaviors. In order to understand something, we must be able to divide information into smaller elements that are fairly well ordered at the level under consideration. Physical processes in humans do not proceed in the same way as processes on a cosmic scale or at the level of elementary particles. Similarly, cognitive systems are designed to work at the human level, although they represent a great many people.

In this regard, the understanding of a language begins with an understanding of the simpler rules of the language - not only formal grammar, but also informal agreements that are observed in everyday use.

Why all this?

Now the IBM Watson cognitive system, thanks to years of learning and improvement, can perform work in a variety of areas. Here and medicine, and cooking, and linguistics, and the solution of business problems with scientific tasks.

Initially, the specialists had a choice - to make the system universal or specialized. Each of the options has its advantages and disadvantages, but the choice was made in the direction of universality.

The company has already been convinced many times about the correctness of the perfect choice - a huge number of possibilities have opened up before IBM Watson . For example, the cognitive system helps to find an individual method of treating cancer, or to create an original recipe, or to establish a business process in a company. Many problems have been solved, but more is yet to be solved.