"Dialogue 2012": competition in analyzing the tonality of texts and the competition of syntactic analyzers

Last year we wrote in some detail about the international conference on computational linguistics “Dialogue” , one of the main organizers of which is our company. This year's conference was remarkable in that the results of two competitions between automatic text analysis systems were summed up at once. For details, welcome under cat.



In the world of computational linguistics and, in particular, in European, for a long time there has been a practice of comparing and evaluating automatic text analysis systems, while Russian science is still taking only the first steps in this direction. Thus, at Dialogue-2010, a competition was held for systems of automatic morphological analysis of the Russian language (systems that can do grammatical analysis of words). Last year, it was decided to hold a contest of parsers (parsers), and at the 2011 Dialogue the conditions for its holding were discussed. The competitions themselves were held in the fall of 2011, and their results were summed up at the Dialogue this year. In addition to the competition of parsers, the conference summed up the competition for the analysis of the tonality of texts (sentiment analysis) - during it were compared systems that allow the author to determine the attitude of the text to the object of description. The value of holding such competitions is to develop uniform standards: from now on, any researcher will be able to focus on certain standards and evaluate their own developments in accordance with them.
')
So, let's see who competed with whom and why.

Sentiment analysis is one of the particular applied problems of computational linguistics, which is highly demanded today. What does it mean? For example, we have a selection of reviews of a feature film, and the task is to determine whether those reviews are positive or negative. This problem can be solved with the help of an automatic system for assessing the tonality of the text: the system determines the nature of the review by analyzing language means. It is clear that the list of tasks for analyzing tonality is huge: for example, companies can quickly find out how buyers evaluate their new product, and, say, the government will be able to track citizens' reaction to the new law, analyzing blogs, etc.

The competition for the analysis of tonality was held in conjunction with the Russian seminar on the evaluation of information retrieval methods ( ROMIP ). ROMIP is a non-profit project that helps to conduct an independent assessment of information retrieval methods according to European standards, but at the same time focuses on working with Russian-language information. The comparison of tonality assessment systems on the material of the Russian language was conducted for the first time.

Initially, all participants in the competition were given test collections: reviews of books and films from the portal imhonet.ru, as well as customer reviews of digital cameras from Yandex.Market. At these collections, participants trained their systems for automatic assessment of tonality, i.e. tried to determine exactly which language tools affect the nature of the review.

For the competition itself, a collection of reviews and reviews from blogs was manually prepared. The participants' systems analyzed these texts on three scales (two, three and five points), i.e. in the first case, the system had to determine whether the feedback is positive or negative, in the second - positive, negative or neutral, and in the third - whether it is purely positive; positive, but with elements of negative evaluation; neutral; negative with elements of a positive rating; entirely negative. Then the experts analyzed the accuracy of the analysis manually.

The competition was attended by 12 systems. The best results were shown by the group system under the direction of Nicolas Chetverkin (MSU Computing Center). In addition, as a result of the competition, a verified and marked collection of texts (available on the ROMIP website) was created, which other developers can use. Publications of participants posted on the website "Dialogue" .

Competition parsers wore, rather, scientific in nature, since automatic parsing is usually one of the pieces of a more general system of automatic text processing (for example, machine translation systems or monitoring systems for news, blogs, etc.). Recall that in the course of the syntactic analysis of the text, the interrelationship of the words within the sentence, i.e. What word depends on what and how. (Remember, at school, arrows were drawn from the subject - to definition, from the predicate - to circumstance, etc.)

The goal of this competition was not only to compare the quality of the systems, but also to develop a unified standard for parsing the text. The fact is that different systems often use different principles of representing the syntactic structure. In the described competition, we agreed to present the results of the analysis as a dependency tree, but in this case they were very different from each other, because different systems interpret the connections between the word forms differently (for example, among some syntaxists the preposition controls the noun, others vice versa).

That is why such a competition was very difficult to organize, it was difficult to bring the different theories of all participants to a common denominator. As a result, for quite a long time, it took the preparation and development of a standard with respect to which the evaluation of the participating systems was carried out.

Applications from 11 development teams from Moscow, St. Petersburg, Nizhny Novgorod, Donetsk (Ukraine) were submitted to the competition. The final results were obtained from 8 of 10 forum participants: SynAutom, DictaScope Syntax, SemSin, STAGE – 3, the syntactic – semantic analyzer of the Russian language of the SemanticAnalyzer Group, the project AotSoft, ABBYY Compreno and the linkage grammar parser.

How was the competition itself? At the beginning, potential participants tested their systems on a test collection of 100 sentences, as a result of which it turned out that the systems do syntactic analysis in different ways. Based on this, unified parsing rules were developed, on which all systems should have been oriented. The so-called “Gold Standard” was also prepared - a reference collection of texts, which, according to these rules, were manually marked by experts. Further participants were offered competitive texts for markup. The responses received by the systems were compared with the “Gold Standard”. Non-compliance with the standard was evaluated by experts: if the participant could prove that in the logic of his system this corresponds to the same type of connection as in the Gold Standard, then such an error was not counted.

The best results were shown by our ABBYY Compreno and the ETAP-3 system (for many years it has been developed at the AA Kharkevich Institute for Information Transmission Problems). For ABBYY, this result is certainly important, but it is also important that the first places were taken by systems based on fundamental linguistics. This means that in the field of syntactic analysis it is impossible to manage only with superficial, statistical methods of estimation, and that the way that ABBYY chose many years ago is promising.

The students of the Department of Theoretical and Applied Linguistics of the Faculty of Philology of the Moscow State University and the Institute of Linguistics of the RSUH assisted in the preparation and conduct of the competition: for them it was an excellent practice of working in a live project on computational linguistics, they were able to get acquainted with different syntactic analyzers, see their strengths and weaknesses, etc d.

As a result of the parsers competition, a corpus of manually marked and verified texts was obtained, which can be used for research purposes (it is freely available at testsynt.soiza.com ). You can get acquainted with the works of participants of the competition here .

Tatyana Panfyorova
with the support of the organizing committee "Dialogue"