How we prepare future big data specialists
This Saturday, our program “Big Data Specialist” starts: it turned out to be so intense that it seems that in the next three months the audience will not have any free time. In this post I will tell you exactly how we will grow Big Data specialists, and how the learning process will be built during the first month.
The central case for this period is the creation of a DMP system. DMP (data management platform) analyzes the Internet logs of users and, based on their behavior on the network, assigns these people different properties or assigns them to certain classes. For example, a properly configured DMP system can determine a person’s gender and age and find out if he is a Gadget man or, say, a fan of luxury fashion brands. We are developing this case together with the Data-Centric Alliance company, they use big data to set up advertising campaigns.
')
Teamwork
At our Open House Day we told that the students will work on cases in teams. Some were unhappy: “Why do we need this! What if I get into the same team with a weakling, and he will pull me back !? ”. Others were worried: “I do not have enough knowledge, what will I do in a team with strong developers?”. Is it logical Yes, but in real life, analyzing big data is not an individual, but a team sport, which is not easy for singles. We set a goal to immerse the listeners in conditions as close as possible to the combat ones.
Teams change once a month, and all students will receive not only valuable team work experience on the project, but also good connections with fellow students. Founders of IT companies, technical directors and experienced developers came to study with us - so these connections will be no less useful than acquaintances with eminent teachers.
Data Management Platform in four steps
During the first 4 weeks, students will create their own DMP system, and each week the group will perform one sub-task, pushing the entire case forward.
So, here is the plan:
Points, rating, badges
We have created a system of points and ratings that will help teachers and employers to navigate how hard the students work.
Each subtask will have at least two levels of difficulty: the first is just to solve it, the second is to solve with additional conditions. For such achievements we will give special badges and additional points. Note to those who set a goal to find a new job: such “achivki” are very interested in our partner employers.
The resulting DMP system will have to calculate the user's portrait with a certain accuracy factor. This is a real challenge to business, and the guys from Data-Centric Alliance are ready to compensate a quarter of the cost of training a team that will create a well-working solution. A team that can show an outstanding solution will receive compensation up to 50%. Increasing the accuracy of classification by 5% increases the return on advertising by 30%, says Alexander Petrov, head of the R & D Data-Centric Alliance department and part-time manager of the first month of the course.
Classroom and practical days
Classes are held three times a week - on Tuesdays, Thursdays and Saturdays. Every Tuesday and Thursday - classroom lessons. They are built in such a way as to maximally involve students in their work (as opposed to the old-fashioned university lecture), everything is based on solving problems and analyzing cases. The purpose of classroom studies is to explain the method of solving problems, to show how all the necessary tools work, and to give a critical look at theory and basic concepts. In our experience, this format provides a deep understanding and good working skills.
This format, by the way, makes it possible to actively engage an online audience! We decided that at this course we will require online participants to work with the camera on. Of course, listeners are shy and even outraged, but the included camera does not leave a chance to work carelessly.
Every Tuesday the team will receive a task for which the week is given. On Saturdays - days of consultations. Digital October doors will be open from 11 am, teams may come to work together to solve the problem. At 4:00 pm, office hours of seminarians begin, who will be able to advise the teams, answer specific questions and help if someone is stumped. The tasks themselves can be checked automatically by downloading the code on a special platform.
Challenges, tests and colloquia
We will be tackling all the tasks on cloud resources, which will be deployed for AWS for each team (we have become partners with Amazon Web Services and therefore will be able to give listeners more computing power). There is 1 week to solve each of the four subtasks, this is “soft deadline”. If the team fails to complete the task within this time limit, it will be possible to surrender another week, but already with a 30% penalty. Anyone who misses several deadlines loses the opportunity to receive a certificate of attendance.
To minimize these unpleasant moments, every two weeks a colloquium will be held on the materials covered. Here you can catch up with the group and get extra points.
As you can see, we do not have a concentration camp, but everything is pretty tough. We want to produce specialists whose competencies we are confident in and whose achievements we can be proud of in the future. You can still sign up for the program , one and a half places left.
The central case for this period is the creation of a DMP system. DMP (data management platform) analyzes the Internet logs of users and, based on their behavior on the network, assigns these people different properties or assigns them to certain classes. For example, a properly configured DMP system can determine a person’s gender and age and find out if he is a Gadget man or, say, a fan of luxury fashion brands. We are developing this case together with the Data-Centric Alliance company, they use big data to set up advertising campaigns.
')
Teamwork
At our Open House Day we told that the students will work on cases in teams. Some were unhappy: “Why do we need this! What if I get into the same team with a weakling, and he will pull me back !? ”. Others were worried: “I do not have enough knowledge, what will I do in a team with strong developers?”. Is it logical Yes, but in real life, analyzing big data is not an individual, but a team sport, which is not easy for singles. We set a goal to immerse the listeners in conditions as close as possible to the combat ones.
Teams change once a month, and all students will receive not only valuable team work experience on the project, but also good connections with fellow students. Founders of IT companies, technical directors and experienced developers came to study with us - so these connections will be no less useful than acquaintances with eminent teachers.
Data Management Platform in four steps
During the first 4 weeks, students will create their own DMP system, and each week the group will perform one sub-task, pushing the entire case forward.
So, here is the plan:
- First week: each listener independently develops a small Hadoop cluster
- Second week: you need to pre-process 1TB of web logs and load them correctly into the HBase table
- Third week: We start analyzing the logs - so far without machine learning. We take predetermined rules and select user classes.
- Week four: Machine learning over Map-Reduce. We build a DMP-system for analyzing web logs!
Points, rating, badges
We have created a system of points and ratings that will help teachers and employers to navigate how hard the students work.
Each subtask will have at least two levels of difficulty: the first is just to solve it, the second is to solve with additional conditions. For such achievements we will give special badges and additional points. Note to those who set a goal to find a new job: such “achivki” are very interested in our partner employers.
The resulting DMP system will have to calculate the user's portrait with a certain accuracy factor. This is a real challenge to business, and the guys from Data-Centric Alliance are ready to compensate a quarter of the cost of training a team that will create a well-working solution. A team that can show an outstanding solution will receive compensation up to 50%. Increasing the accuracy of classification by 5% increases the return on advertising by 30%, says Alexander Petrov, head of the R & D Data-Centric Alliance department and part-time manager of the first month of the course.
Classroom and practical days
Classes are held three times a week - on Tuesdays, Thursdays and Saturdays. Every Tuesday and Thursday - classroom lessons. They are built in such a way as to maximally involve students in their work (as opposed to the old-fashioned university lecture), everything is based on solving problems and analyzing cases. The purpose of classroom studies is to explain the method of solving problems, to show how all the necessary tools work, and to give a critical look at theory and basic concepts. In our experience, this format provides a deep understanding and good working skills.
This format, by the way, makes it possible to actively engage an online audience! We decided that at this course we will require online participants to work with the camera on. Of course, listeners are shy and even outraged, but the included camera does not leave a chance to work carelessly.
Every Tuesday the team will receive a task for which the week is given. On Saturdays - days of consultations. Digital October doors will be open from 11 am, teams may come to work together to solve the problem. At 4:00 pm, office hours of seminarians begin, who will be able to advise the teams, answer specific questions and help if someone is stumped. The tasks themselves can be checked automatically by downloading the code on a special platform.
Challenges, tests and colloquia
We will be tackling all the tasks on cloud resources, which will be deployed for AWS for each team (we have become partners with Amazon Web Services and therefore will be able to give listeners more computing power). There is 1 week to solve each of the four subtasks, this is “soft deadline”. If the team fails to complete the task within this time limit, it will be possible to surrender another week, but already with a 30% penalty. Anyone who misses several deadlines loses the opportunity to receive a certificate of attendance.
To minimize these unpleasant moments, every two weeks a colloquium will be held on the materials covered. Here you can catch up with the group and get extra points.
As you can see, we do not have a concentration camp, but everything is pretty tough. We want to produce specialists whose competencies we are confident in and whose achievements we can be proud of in the future. You can still sign up for the program , one and a half places left.
Source: https://habr.com/ru/post/251675/
All Articles