Automatic summarization system for three languages

I want to tell you about the news text reporting service developed by me in English, Russian and German.

Automatic summarization (summary) systems (CAP) are a rather specific topic and will be of interest mainly to those involved in automatic language processing. Although a perfectly executed Sammaraiser could be a useful assistant in areas where it is necessary to overcome the information overload and quickly decide what information is worth further consideration.

What is the situation?

On the one hand, in the process of searching for analogs, I noticed an interesting thing - most of the articles I found, services, repositories, etc., date back to 2012 at the latest. On Habré there is an article on the topic of automatic summarization, published in 2011. In the same year, the news summarization was last included in the track list of the TAC conference.

On the other hand, mobile applications are gaining popularity, which process news feeds and present the user with short essays on selected topics. A striking example of such demand is the relatively recent (2013) purchase of Google and Yahoo by Wavii and Summly startmars-startups, respectively, as well as the presence of various browser-plug-ins that review the web pages ( Chrome , Mozilla ).
')
A quick test of free online summarization services shows that most of them work the same way, producing equally average (bad?) Results, among which, perhaps, the Autosummarizer stands out for the better .

Why another SAR?

The initial goal of the project is to serve as a platform for learning programming in general and programming in python in particular. And since the topic of computational linguistics is close to me by type of activity, I chose reviewing as the object of development, besides, there were already some ideas and materials on it.

If you walk through the services from the above list, you can see that they mostly work with English texts (if you can get them to function at all). You can choose another language in MEAD , OTS , Essential-mining , Aylien, and Swesum . At the same time, the first one doesn’t have a web interface, the third after 10 test texts requires registration, and the fourth, giving the opportunity to set the settings in the demo, for some reason does not want to refer to anything.

As I got something worthwhile with the processing of English texts, I wanted to make a service that would work with Russian and German news articles and would work as well as the ones listed, and also provide an opportunity to compare the developed algorithm with the TextRank methods that are popular today. , LexRank, LSA and others . Besides, this is a good opportunity to practice with html, css and wsgi.

Where to look?

Project website: t-CONSPECTUS

How does it work?

t-CONSPECTUS is an extract type self-extractor, i.e. he forms a summary of the sentences of the original article, which in the process of analysis received the greatest weight, and, therefore, best convey the meaning of the content.

The whole process of summarization is carried out in four stages: preprocessing of the text, weighting of terms, weighing sentences, extracting meaningful sentences.

During preprocessing, the text is divided into paragraphs and sentences, a heading is found (needed for correcting the weights of terms), tokenization and stemming is carried out. For Russian and German languages ​​lemmatization is carried out. Pymorhpy2 lemmatizes Russian texts, to process Germans, I had to write my lemmatization function based on the CDG parser lexicon , because neither NLTK, nor Pattern, nor TextBlob German, nor FreeLing provided the necessary functionality, and the selected hosting does not support Java, which excluded the use of Stanford NLP .

At the stage of weighting the terms, keywords are determined using the TF-IDF. The term gets an additional factor if:

1. met in title
2. met in the first and last sentences of paragraphs
3. met in exclamation or interrogative sentences,
4. is a proper name.

Weighing proposals is carried out according to the method of symmetric summarization.

A detailed description is given in the article “Yatsko V.A. Symmetric summarization: theoretical foundations and methods // Scientific and technical information. Ser.2. - 2002. - № 5. "

With this approach, the weight of a sentence is defined as the number of links between a given sentence and the sentences to the left and right of it. Links are keywords common to the offer and its neighbors. The sum of left and right relations is the weight of the proposal. There is a limitation - the text should consist of at least three sentences.

In addition, when calculating the final weight of a sentence, its position in the text (the first sentence is the most informative in the news texts), the presence of proper and numeric sequence names and the length of the sentence are taken into account. Additionally, a penalty factor is applied, which reduces the weight of long sentences.

The specified number of significant sentences is selected from the list sorted in descending weights, and the extracted sentences are placed in the order in which they appeared in the original in order to at least somehow comply with the coherence of the text. The size of the sammari is by default equal to 20% of the original volume.

What is the quality of sammari?

The traditional approach to assessing the quality of sammari is a comparison with a human abstract. The ROUGE package is by far the most popular tool for such an assessment.

Unfortunately, it is not so easy to get the standards, although, for example, the DUC conference provides the results of past competitions of summarizers, including human abstracts, if you go through a number of bureaucratic procedures.

I have chosen two fully automatic metrics, justified and described in paragraph 3 here (pdf), which compare summaries with the original article. These are cosine similarity and distance of Jenson-Shannon ( Jensen – Shannon divergence ).

The Jenson-Shannon distance shows how much information will be lost if you replace the original with an abstract. Accordingly, the closer the indicator is to zero, the better the quality.

The cosine coefficient, classical for IR, shows how close the vector of documents is to each other. I used tf-idf words to build vectors. Accordingly, the closer the indicator is to 1, the more the abstract on the density of keywords corresponds to the original.

The following systems were chosen for comparison:

• Open Text Summarizer , which works with the three languages ​​I have chosen, and according to the developers themselves “Several academic publications have benchmarked it and praised it” ;
• TextRank, a popular algorithm today in the implementation of Sumy ;
• Random - compare the algorithm with randomly selected article sentences.

For each language, uninterested users selected 5 texts from the areas of “popular science article” (popsci), “environment” (environ), “politics” (politics), “social sphere” (social), “information technologies” (IT ). Evaluated 20% of essays.

Table 1. English:

	Cosine Similarity (-> 1)				Jensen – Shannon divergence (-> 0)
	t-CONSP	Ots	Textrank	Random	t-CONSP	Ots	Textrank	Random
popsci	0.7981	0.7727	0.8227	0.5147	0.5253	0.4254	0.3607	0.4983
environ	0.9342	0.9331	0.9402	0.7683	0.3742	0.3741	0.294	0.4767
politics	0.9574	0.9274	0.9394	0.5805	0.4325	0.4171	0.4125	0.5329
social	0.7346	0.6381	0.5575	0.1962	0.3754	0.4286	0.5516	0.8643
IT	0.8772	0.8761	0.9218	0.6957	0.3539	0.3425	0.3383	0.5285

Table 2. German:

	Cosine Similarity (-> 1)				Jensen – Shannon divergence (-> 0)
	t-CONSP	Ots	Textrank	Random	t-CONSP	Ots	Textrank	Random
popsci	0.6707	0.6581	0.6699	0.4949	0.5009	0.461	0.4535	0.5061
envir	0.7148	0.6749	0.7512	0.2258	0.4218	0.4817	0.4028	0.6401
politics	0.7392	0.6279	0.6915	0.4971	0.4435	0.4602	0.4103	0.499
social	0.638	0.5015	0.5696	0.6046	0.4687	0.4881	0.456	0.444
IT	0.4858	0.5265	0.6631	0.4391	0.5146	0.537	0.4269	0.485

Table 3. Russian language:

	Cosine Similarity (-> 1)				Jensen – Shannon divergence (-> 0)
	t-CONSP	Ots	Textrank	Random	t-CONSP	Ots	Textrank	Random
popsci	0.6005	0.5223	0.5487	0.4789	0.4681	0.513	0.5144	0.5967
environ	0.8745	0.8100	0.8175	0.7911	0.382	0.4301	0.4015	0.459
politics	0.5917	0.5056	0.5428	0.4964	0.4164	0.4563	0.4661	0.477
social	0.6729	0.6239	0.5337	0.6025	0.3946	0.4555	0.4821	0.4765
IT	0.84	0.7982	0.8038	0.7185	0.5087	0.4461	0.4136	0.4926

The texts of the original articles and the summaries received can be found here .

Here and here you can download third-party packages for automatic assessment of the quality of sammari.

What's next?

Further, it is planned to gradually improve the algorithm, for example, to take into account synonyms when searching for keywords of an article, or to use for these purposes something like Dirichlet latent placement ( Latent Dirichlet Allocation ); decide which parts of the text need special weighting (for example, numbered lists); try adding more languages ​​etc. etc.

On the site itself, add quality indicators to the statistics, add a visual comparison of the results of the "native" algorithm and third-party, etc.

Thanks for attention!