Hackathon and the winter scientific school on deep learning and question-answer systems

Today, machines can easily “bind two words” ( 1 , 2 ), but so far they are not able to conduct a dialogue on common topics with guarantee. However, tomorrow you will ask them to make a resume correctly and choose the best chess section for your children not far from home. Want to understand more about how scientists from Facebook, Google, etc. work in this direction? Come listen to them.


From January 31 to February 5, the second international hackathon for deep learning and machine intelligence DeepHack.Q & A will take place at the Moscow Institute of Physics and Technology with the scientific support of the Laboratory of Neural Systems and Deep Learning. As part of the scientific school hackathon, the world's leading experts from Google Brain, Facebook AI Research, OpenAI, Skoltech, University of Oxford, Courant Institute of Mathematical Sciences at NYU, will give a series of lectures on deep learning and its application to natural language processing tasks. Most of them are remote, but the personal presence of Rob Fergus and Phil Blunsom is expected.

About the topic. Hackathon will be devoted to the task of The Allen AI Science Challenge . Participants are required to develop a program that can independently learn how to answer questions of the grade 8 level of the American school. For this, participants were given a training (2,500 questions) and a validation (8,132 questions) sets of questions in a csv-file format with 4 possible answers. For the test suite, the correct answers are known. Validation set is necessary in order to determine the level of accuracy of the answers of your system and, accordingly, to rank the stated decisions on this criterion. The sets contain questions on the main subjects of the school curriculum: physics, biology, geography, etc.

The formulation of the problem looks so general that it seems that it is impossible to approach its solution without having deep expertise in the field of natural language processing. However, this is not quite true. It is possible to apply already developed neural network methods in a few lines of code and get the result in 32% accuracy.
')
This is what the guys from the 5vision team did (by the way, the winners of the summer hackathon) and published their decision on the Kaggle and Github forum . Installation instructions can be found here . If suddenly there is an overwhelming desire to use Linux, but you don’t have it at hand, you can register for free at koding.com (or where else) and run everything there. Now I would like to elaborate on what this decision is doing.

It is based on one of the implementations of the representation of words in a multidimensional vector space - GloVe (Global Vector for Word Representation), where similar words in the Euclidean metric of a vector are associated with words of a similar meaning. The more well-known implementation of word2vec has already been covered in Habré ( 1 , 2 , 3 , etc.) (and one of the authors of this implementation, Tomas Mikolov, will lecture on the hackathon).

The application of GloVe ( glove_predict.py ) to the question is as follows:

1. Preprocessing of the issue:
   • we throw out everything except the capital and small letters of the English alphabet and a space.
   • we throw out the so-called “stop” words (words that have little effect on the meaning of the sentence).
2. Set the zero vector representation of the question q = 0.
3. We cycle through all the remaining words of the question and add to q the vector representation of each word.
4. We carry out similar calculations of the vector representation of all four answers.
5. Choose the answer, the distance to which from the vector q is the smallest.

This implementation gives 32% accuracy on the validation set.

5vision has another more “classic” implementation ( ck12_wiki_predict.py ), based on the use of the TF-IDF measure. It works like this:

1. Parsing keywords on major topics from www.ck12.org (for example, the topic “Physics”: http://www.ck12.org/physics/ ).
2. We download documents from wikipedia on the collected keywords.
3. We calculate the TF and IDF measures for this collection of documents and words.
4. For each question, select the most relevant article from the collection.
5. For each answer, consider its relevance in relation to this article and choose the one that gives the maximum value.

This implementation gives 36% accuracy. To present the scale of the results, it is worth noting that currently the first place in the rating answers correctly 56% of the questions.

The competition has a number of features ( here you can find a review-squeeze from the competition forum) - for example, the final solution should work without access to the Internet. And in general, reading the forum of the competition can give a lot of useful information.

The schedule of the event is the same as in the past DeepHack.

Sunday: participants gathering, organizational meeting, team building, setting up the environment, introductory lecture, and the first nightly launch of the calculation.

Monday-Friday: discussion of intermediate results, programming of new solutions developed in a team, lectures, launching calculations.

Saturday: summing up, awarding the winners.

All participants will also be provided with accommodation and computing resources . But you need to bring your laptop with you.

In the near future we plan to publish a review of neural network architectures in the field of natural language processing, from which it will be possible to draw ideas for improving current solutions.