Mlbootcamp competition from mail.ru. Briefly about the recipe for second place

Good afternoon, reader! This article will talk about how to get second place in the MLBootCamp III competition. For those who are not up to date - this is the competition for machine learning and data analysis from Mail.Ru Group, held from February 15 to March 15.

The article will briefly describe the history of the construction of the solution, some tips about what made cones and thanks.

So let's go.



Logloss was used as a metric, so it was worth remembering that:

- Many classifiers already know how to minimize logloss "out of the box";
- Logloss "does not forgive" when rounding up answers to 1.0 or 0.0, instead of, for example, 0.97 and 0.03 and you are mistaken (see the chart below).


Therefore, you should not touch the results of the prediction with your hands, and moreover round off.

About solving a problem

The first submission was made almost immediately after the opening of the competition - this is a random forest on the top 10 best, in his opinion, features. After that, a step was taken, which was spent about a week, but it was not effective - quadratic features were generated and the simplest greedy feature selection algorithm was launched based on the result of cross-validation.

The algorithm is as follows: we teach the forest, we take the importance of features from it, and, starting with the first two, we add one feature to our current set. If adding a feature leads to an improvement in the result - leave the feature in the set. If not, we throw it back and never return to it.

Upon completion of this simple algorithm, we use another heuristic - throw out one feature at a time and look at the result when working on all the others. If the result improves - then the feature in the furnace.

While the search for features generated forests and warmed up the apartment, I wanted to train the logistic regression and play around with xgboost, since I have not used it before (yes-yes).

At first I tried the logistic regression on the available features (not forgetting to normalize them, of course), then decided to convert all the features as follows:

- beat all feature values ​​on N intervals of equal size 1 / N on percentiles
- we encode the value of the feature as ohe, i.e. if the value of the feature in the example falls into the interval - we set 1, otherwise 0, so the vector [0 0 ... 0 1 0 ... 0] of length N is obtained from one value

The idea was as follows - a different player level (the number of days he played, the number of stars, etc.) nonlinearly affect the target variable, then you can try to bring this nonlinearity into a linear model in this transformed form (although in such , trimmed, "group" version).

Such a transformation yielded about 0.3836 for cross-validation - not bad for a logistic regression (as it turned out later, it can be better).

Then I started xgboost and quickly got a good solution for cross-validation, about the same as in a random forest at that time (something around 0.3812).

After that, I added the results of three quite good cross-validation classifiers into one, divided it into three, sent it, and ... I was disappointed, because the public leaderboard turned out to be a set of examples in which my ensemble worked very badly. By the way, this situation was observed not only for me, but for all my friends and acquaintances who solved the problem. It was ridiculous - the better the classifier behaved in cross-validation, the lower he dragged me into a public leaderboard. This situation is not weakly motivated, so interest in the task was lost for about a week.

<Week has passed>

One day I opened the leaderboard, and saw how my friend went up the table, and in one (or two) days I was already in the top twenty. “Wow,” I thought, rolled up my sleeves, and again took up the task.

The method of close scrutiny of graphs revealed that there were people with one day of participation in an online game, having a 100+ level, a lot of bonuses and points. This suggested that people actually played for a long time, just a two-week window, in which we consider features, hooked them in just one day. Having broken my head over a couple of evenings over how it would be used, on the basis of each of the maxPlayerLevel, totalStarsCount and totalNumOfAttempts features, I decided to make one new “predicted number of days” that the player plays. The idea is the following: we take the lower values ​​(took everything below the 4th percentile) in each of the days for the selected feature (for example, maxPlayerLevel), we get a rather nice graph that approaches the polynomial quite well (took the fourth degree, although the third would be enough). Having a built polynom, we can now find its roots for the player’s feature value and cost predictions of the number of days the player has played. It looks like this:



On the x axis, the number of days is plotted; on the y axis, the value of the selected features. A broken curve is the same feature values ​​that lie below the 4th percentile, a beautiful smooth curve is a graph of the approximating polynomial.


Having thus obtained three more signs, he normalized the original features (except for the number of days the player spent in the game) on each of the new ones, thus the total number of features increased fourfold. He invented and added two dozen more features, among which, for example, such:

 ... X['positiviness'] = X.totalStarsCount*X.totalBonusScore*X.usefulBoosters X['winDesire'] = X.attemptsPerDay*X.numberOfBoostersUsed*X.attemptsOnTheHighestLevel X['positivinessPerWinDesire'] = X.positiviness/(X.winDesire+1) ... 

Further, due to the proliferation of signs, I did not begin to generate quadratic ones and selected the best features from the available ones, trained the ensemble and turned out to be in 60th place, rising from ~ 120th.

Sklearn's logistic regression was replaced with the stochastic implementation of that in vowpal wabbit (used the Python wrapper of its own writing, about it another time), simultaneously abandoned its idea of ​​using feature coding at intervals and slightly improved the result of the logreg on cross-validation (here there were also a few self-made crutches to get the result of the cross prediction). Since vw has already been involved, why not use the perceptron from it (the --nn flag) - so another classifier was added to the ensemble.

Well, since the vw perceptron was involved, why not use the MLPClassifier.

After that, two more classifiers based on a random forest were added to the ensemble. One of them was trained on a sample with the deviations removed on each of the days according to the feature values ​​(all the examples, the feature values ​​in which are below the 1st percentile and above the 99th did not participate in the classifier training), the second - just on the newly selected features.

In total, by the end of the competition the ensemble consisted of 7 classifiers. How for all this to make a normal scheme of cross-validation and not much to suffer? The construction scheme is as follows:

• Each of the classifiers put in a separate folder in the project.
• Each of the classifiers, instead of cross-validation, makes a cross prediction on the train, the result is put into a file in its own folder, with a definite ending (to make it easier to search for these files).
• In each folder we train one classifier on the entire train, we make a prediction on a test sample, which we also put in a file in our folder with a definite ending.
• In the directory above the classifier folders, we make a common ensemble script that, for cross-validation, loads all cross-predicates from all the necessary classifier folders, converts them according to the rules we need into the result of the ensemble and calculates the metric we need (logloss).
• Having received the results of the ensemble validation, it remains to load the predictions from each folder, and convert them according to the same rules to obtain the ensemble forecast on the test sample.
• Add a few finishing touches.

The last day

On the last day, when the decision was already collected and sent, my best result was in public at 37th place, and for cross-validation I issued logloss 0.37906093. About what took second place, I learned from the message of a friend in Skype. I thought it was a joke, but when I went to the site, I was convinced that it really was true.

Useful Tips

The word "useful" deliberately placed in quotes, because Most of them meet more than once in every story about winning a machine learning competition:

• At the start, you should not draw beautiful graphics all day long, count statistics and do other tricks, but rather make the first submit. Spend an hour or two and do not worry about the place in the table yet. This will allow laziness to win from the very start, check whether you understood the task correctly, whether the submission system works on the side of the organizers of the competition, whether the response file is formed correctly, etc.
• After the first submission, you should look at the data, statistics, graphs and try to understand what the task is about. For example, in this task, for some reason, I first thought that all the people on the basis of which examples were formed in the samples began to play from the very first day, and then someone stopped the road, and only after applying the attentive scrutiny method I realized that it was not this way.
• Every change, every experience and every idea should be checked separately, while making copies of all that is possible. If possible, you should, of course, use a version control system, but sometimes you can do without it. For example, if you are working in ipython notebook, make a copy of your laptop before conducting the next experiment. Hard disk space is much cheaper than nerves spent on finding "exactly that" right option, which gave the desired solution.
• Get yourself a board, notebook, or a program in which you write down the ideas that come to mind. I, for example, use trello, there I have three groups of cards: "New ideas", "Working ideas" and "Non-working ideas". I write down new ideas in the first group, and then after conducting the experiment, it moves either to the second or to the third.
• Cross-validate, and do not forget to make a prediction file for a test of current experience. Also a good approach (if not even better) is to record files with a cross-predicate on the training set. Pluses become apparent when combining models into an ensemble.
  
  
• Believe in the result of your cross-validation.

  
  

Thanks

Many thanks to Mail.ru, as well as Ilya Stytsenko and his team for a wonderful contest and presents. Many thanks to my friends and rivals in the competition (and not only) for the support / opportunity to talk / consult and share opinions, I think it was useful for all of us. Thank you my girlfriend for her patience and patience again.