Bayes Summer | Deep | Bayes Summer School Materials in In-Depth Learning

In-depth training in recent years has become a key area of ​​research in machine learning. Starting with architectural breakthroughs that allowed to effectively train deep neural networks, it began to spread to other subdomains, providing a set of effective tools where solving a problem requires the approximation of some complex function.

Many modern research articles actively use Bayesian formalism in combination with deep neural networks, coming to interesting results. We - BayesGroup research group with the help of our friends from Skoltech, as well as with the support of the Higher School of Economics, Sberbank, Yandex, Kaspersky Lab, JetBrains and nVidia - decided to share our experience and arrange a summer school on Bayesian methods in in-depth training Deep | Bayes , where you can tell in detail what Bayesian methods are, how to combine them with in-depth training and what can come of it.

The selection for the school turned out to be a very difficult task - we received over 300 applications from strong candidates, but only 100 could accommodate (it’s nice that the participants included not only residents of Moscow and St. Petersburg, but also students from the regions, as well as Russian-speaking guests because of borders). I had to refuse to many strong candidates, so in order to mitigate this regrettable fact, we decided to make available the maximum amount of materials that we want to share with habrauders.

Day 1

The first day consisted entirely of introductory lectures - the students of the school had a different background, so they needed leveling lectures. At a minimum, to define a common terminology.

Day 2

The main material of the second day is Bayesian inference in problems with hidden variables and their scaling to complex tasks.

Day 3

On the third day, we took a look at reinforcement training from a Bayes point of view; we looked at various approaches to attention mechanisms. In addition, they discussed the use of a powerful apparatus of recurrent neural networks for various tasks, as well as tricks without which nothing works. Alas, due to technical problems this lecture was not recorded.

Day 4

The topic of the fourth day was Gaussian processes, their scaling and Bayesian optimization, traditionally used to search for hyperparameters. In addition, we considered scalable Monte Carlo methods as an alternative to variational inference.

Day 5

On the fifth, concluding day, we stocked up the most interesting - Bayesian methods for implicit models, crossing variational auto-encoders with generative competing networks, estimating uncertainty in neural networks, as well as automatic thinning of neural networks with a variational dropout.

In conclusion, I would like to say that this was our first experience of holding summer schools and, judging by the preliminary feedback, successful - many students expressed interest in helping to re-organize the school next year. If you want to keep abreast of the news, subscribe to our channel on twitter @deepbayes_ru (and you can subscribe to @bayesgroup too) and follow the announcements.