Black Box Challenge Summary
Hi, Habr! Three months ago, we announced the start of the BlackBox Challenge machine learning competition , and recently it ended. In this post, the organizers of the competition will tell you how it all went.
Inspired by the results of Google DeepMind on reinforcement learning , we realized how great it is when the system does not use human expertise, but is learning to understand the environment. We decided to make a competition in which participants need to create just such a system.
The BlackBox Challenge format is a synthesis of the classic format of machine learning competitions (like on the Kaggle website) and artificial intelligence programming competitions (for example, Russian AI Cup ). The participant was asked to write a bot that plays a game with unknown rules - at each step the bot is given 36 variables describing the state of the environment, and it must perform one of four actions.
On the one hand, the competition turned out to be interactive - it was necessary to write an agent who interacts with the external environment. On the other hand, the laws of this environment were unknown to the participants - this forced them not to use a priori knowledge about the game device, but to apply modern methods of machine learning .
')
The competition lasted for three months, during this time 3347 solutions were downloaded, of which 1459 are non-trivial solutions that do not coincide with the published example (baseline agent).
The number of registered participants is 1360, of which 415 have downloaded at least one solution.
93 participants were able to surpass the baseline at the validation level.
The prize fund was 800,000 rubles:
In the last weeks of the competition in the leaderboard was a fierce struggle, and the fate of the prizes decided a few points.
With a large margin, the winner became an insight participant with a score of 4693 points at the final level.
Participants from second to fifth place - 5vision, alexandrbugaychuk, grmel89 and wrwrwr - go very close to each other. The gap between the results of the 2nd and 5th places is less than 150 points! This is astonishingly small, and in order to understand it, we built the graphics of the best solutions at the validation and final levels (note that the graphics are built for the best decisions at the final level).
It can be seen from them that the decisions of these participants themselves are also very close and the difference in the results is due to the randomness inherent in the game. This time fortune was on the side of 5vision and alexandrbugaychuk, congratulations! Prize 6-8 places were taken by VictorGNC, cosionix and AGilmullin (Kesha), overcoming the base bot by more than 1000 points. This is a great result.
Participants SDil and ottogin close the top ten, also overtaking the base bot by more than 1000 points.
The full table of final results is available here .
In addition to the main set of prizes, we also played out the nomination “The most interesting solution”, in which DCA experts evaluated the elegance and perspectivity of the participants' approaches.
Most of the solutions turned out to be multiparameter models in which the parameters were changed randomly, often with the help of evolutionary algorithms. The quality of the model was determined by the result on one of the game levels. Judging by the results, such approaches proved to be quite effective. Similarly, our linear bot was obtained (baseline).
There were, however, several participants who acted differently and also achieved good results. DCA experts found it difficult to choose the most interesting solution, but in the end, the decision of the 5vision team, who managed to implement an elegant idea using policy iteration, won. The team receives an additional 100,000 rubles.
I would also like to mention guillermobarbadillo solutions - the only one who managed to apply Q-learning, ottogin - for the found method of supervised neural network training and, of course, insight - for an unusual and effective approach to sampling.
We opened a verification system for those who want to solve a black box for their own pleasure and check ideas for which there was not enough time.
Judging by the reviews, many liked this format of the competition, so soon we plan to hold a competition with a new interesting interactive task.
For cooperation, we are available at wow@blackboxchallenge.com
Thanks for participating!
Inspired by the results of Google DeepMind on reinforcement learning , we realized how great it is when the system does not use human expertise, but is learning to understand the environment. We decided to make a competition in which participants need to create just such a system.
What a challenge?
The BlackBox Challenge format is a synthesis of the classic format of machine learning competitions (like on the Kaggle website) and artificial intelligence programming competitions (for example, Russian AI Cup ). The participant was asked to write a bot that plays a game with unknown rules - at each step the bot is given 36 variables describing the state of the environment, and it must perform one of four actions.
On the one hand, the competition turned out to be interactive - it was necessary to write an agent who interacts with the external environment. On the other hand, the laws of this environment were unknown to the participants - this forced them not to use a priori knowledge about the game device, but to apply modern methods of machine learning .
')
Results
The competition lasted for three months, during this time 3347 solutions were downloaded, of which 1459 are non-trivial solutions that do not coincide with the published example (baseline agent).
The number of registered participants is 1360, of which 415 have downloaded at least one solution.
93 participants were able to surpass the baseline at the validation level.
Prizes
The prize fund was 800,000 rubles:
- 1st place: 300,000 rubles
- 2nd place: 175,000 rubles
- 3rd place: 125,000 rubles
- 4-8 places: Xbox One
- Special prize of 100,000 rubles for the most interesting solution chosen by DCA experts
In the last weeks of the competition in the leaderboard was a fierce struggle, and the fate of the prizes decided a few points.
With a large margin, the winner became an insight participant with a score of 4693 points at the final level.
Participants from second to fifth place - 5vision, alexandrbugaychuk, grmel89 and wrwrwr - go very close to each other. The gap between the results of the 2nd and 5th places is less than 150 points! This is astonishingly small, and in order to understand it, we built the graphics of the best solutions at the validation and final levels (note that the graphics are built for the best decisions at the final level).
It can be seen from them that the decisions of these participants themselves are also very close and the difference in the results is due to the randomness inherent in the game. This time fortune was on the side of 5vision and alexandrbugaychuk, congratulations! Prize 6-8 places were taken by VictorGNC, cosionix and AGilmullin (Kesha), overcoming the base bot by more than 1000 points. This is a great result.
Participants SDil and ottogin close the top ten, also overtaking the base bot by more than 1000 points.
The full table of final results is available here .
Nomination "The most interesting decision"
In addition to the main set of prizes, we also played out the nomination “The most interesting solution”, in which DCA experts evaluated the elegance and perspectivity of the participants' approaches.
Most of the solutions turned out to be multiparameter models in which the parameters were changed randomly, often with the help of evolutionary algorithms. The quality of the model was determined by the result on one of the game levels. Judging by the results, such approaches proved to be quite effective. Similarly, our linear bot was obtained (baseline).
There were, however, several participants who acted differently and also achieved good results. DCA experts found it difficult to choose the most interesting solution, but in the end, the decision of the 5vision team, who managed to implement an elegant idea using policy iteration, won. The team receives an additional 100,000 rubles.
I would also like to mention guillermobarbadillo solutions - the only one who managed to apply Q-learning, ottogin - for the found method of supervised neural network training and, of course, insight - for an unusual and effective approach to sampling.
What's next
We opened a verification system for those who want to solve a black box for their own pleasure and check ideas for which there was not enough time.
Judging by the reviews, many liked this format of the competition, so soon we plan to hold a competition with a new interesting interactive task.
For cooperation, we are available at wow@blackboxchallenge.com
Thanks for participating!
Source: https://habr.com/ru/post/303286/
All Articles