Technopark portal: how we did it
Hello. This article describes the Technopark portal: its functions and capabilities, development history, and the technical and architectural solutions used. The article is written by the developers of the portal - students of Technopark itself. We hope that the development experience we share will be useful to readers.
Three years ago, Technopark opened its doors - a project of the Mail.Ru Group and MSTU. N. E. Bauman. To build interaction with students, LiveStreet CMS was chosen for the start of the project. The teachers laid out topics with homework, and the schedule for students came in the form of weekly topics. This is how the portal looked like in its first release:
')
The project was developing, the basic functions of LiveStreet began to be lacking, and we began to expand the capabilities of the portal using a plugin system. We have added various modules and functional blocks, such as curriculum, schedule, events, personal offices of teachers and students. However, a year later, we were faced with the specific difficulties of updating the initially forum engine, and decided to completely rewrite it specifically for our needs. We present to your attention what came out of it.
The whole portal is divided into two parts. The open part is visible to everyone and includes several pages of a general description of Technopark, a section with a full description of programs and disciplines, as well as a schedule .
All lectures and classes in the Technopark are recorded, we post the processed videos in the “ Video recordings ” section. For applicants, the FAQ section has been developed, containing answers to most questions about the project.
Every six months, Technopark opens a set for those who want to delve into web development and web application architecture. In order to participate in the recruitment of one of the training programs, registering, applicants fill out a questionnaire.
A detailed questionnaire is needed in order to meet a potential Technopark student, introduce students to each other and get a general idea of ​​recruitment. After registering on the portal, the section “Personal Account of the Applicant” becomes available, where all users can learn about the kits currently open, apply for them or get acquainted with those who are already enrolled in Technopark programs.
Admission to the Technopark includes several stages: registration, online testing, thematic offline interview. After each stage, the results and decisions of the interviewees are recorded in the applicant's card. In the end, by pressing the “Enroll” button, the best of the best acquire a study group, become members of our community and gain access to the internal sections of the portal. They have the opportunity to take part in discussions, raise topics for discussion, creating topics both in general sections and in sections on disciplines.
After successful completion of the curriculum, graduates of the project fall into the section “ Issues ” - on the “honors board” of graduates.
Each graduate is awarded a diploma with a QR code.
With this code you can get directly to the page of the graduate - the owner of the diploma, which serves as confirmation of the official diploma.
The second, most of the portal, which has already been mentioned in passing, is a closed part, intended for students, teachers and administrators of Technopark.
Access to it is obtained only by confirmed members of our large family: students, graduates, teachers, and the administration of the portal. The basis of the basic functionality was, is and remains microblogging. Everything that is connected with life in a project - from classes and homework to solving organizational issues - we discuss here.
Students also get access to a private office where they can observe the grades they receive, their own progress and attendance. For success in training, we issue achievements. The best students - the coolest badges!
At the same time, teachers have a private office. It includes sections: journals for grading, feedback on occupations and attendance reports.
Yes, we ask all students at the end of classes to leave their opinion about him.
This information is stored without disclosing the identity of the person who left the review so that the comments of the students would not affect their further progress. According to the obtained data, we form various graphs.
Moreover, each participant has the opportunity to chat in a chat with other participants or as part of a group. Chat is built on WebSocket technology. All messages come instantly (Instant Messenger). For implementation, we used the development of Alexander Emelin Centrifuge , about which there is a detailed article on Habré .
On all pages, we display a mini-blog with classes that the viewing student or teacher will have on this day. The functionality of this unit includes a mark on the presence of the lesson, and after the lesson is completed, it is possible to leave a review.
In the Technopark in the administrative part, it is possible to form all the training programs, ranging from a general description of the program and ending with a detailed indication of each class, each homework given out and each grade required.
An interface has been developed for scheduling on the basis of a program plan, an existing classroom fund, study groups and available time ranges for conducting classes. There are checks and leveling of erroneous situations, when at the same time the group has two classes in different disciplines.
Ultimately, all students and applicants receive complete and comprehensive information on the entire program of study.
During the semester, students attend classes, receive grades, teachers receive feedback and have the opportunity to adjust the subject or pace of the lecture in the next lesson. At the same time, we monitor not only the implementation of the curriculum by students, but also the work of teachers: how quickly they check their homework, how often they write in blogs.
Inevitably, a session begins - the time to pass the mid-term control, close debts and receive final grades in a discipline. Inevitably comes the end of the semester - the stage of transfers and contributions.
All this functionality is implemented in a closed part accessible to project administrators. You can select individual students and, with the click of a button, translate them into new study groups for future semesters. It is just as easy to expel a student. Our leaders are very sensitive to these buttons and at the end of each semester are asked to once again check the correctness of the work of these functions with the fixation of all changes. Just in case, we have provided the possibility of canceling these changes. The hand trembled, nervous ...
Based on the results of such modifications, notifications of decisions taken are sent to all parties. Students who have successfully completed all the training receive the status of "Graduate" (fall into the section "Issues") and receive the eternal right to remain a member of the community. Expelled students lose access to the inside of the portal.
We also automated the process of selecting students for study programs. Before the start of the set, we configure it in the portal admin panel: choose which stages it will consist of, which external testing systems will be used. When we recruit students for the main program, we traditionally choose online testing and face-to-face interview. As a result, we get a table in which there are all applicants who have applied for this set, successive steps through which applicants can be transferred. If the applicant passes the last stage, it is automatically credited. Moreover, if the stage is carried out in an external system, for example, online testing, then automatic export of applicants to this system and export of their grades is provided.
For testing, we use another development of the Technopark laboratory: the Technotest online testing system, it is used both for admission testing and for some major controls.
The Technopark has optional open courses : training potential Technopark students and general disciplines for all students. Students of elective programs do not get access to all blogs of disciplines, because according to the rules of the project before enrolling in the main program they are not full-fledged students. At the same time, each participant can be a student of several programs, a graduate, and even a teacher at the same time.
Technopark regularly hosts events and invites speakers. In the spring of 2015 there were performances:
To ensure recording of events, monitoring the number of volunteers and collecting feedback, the functions of the events were developed. After the speech, we ask you to leave your feedback about the event in a separate letter to all who participated. Guest reviews are later taken into account by the organizers to improve the quality of future workshops.
The announcement of all upcoming events is also duplicated on the pages for applicants and visitors. Take part in the role of listeners can everyone.
Due to the fact that the number of participants in events increases every year and we can no longer physically accommodate everyone in one student audience, we are broadcasting from the main audience of Technopark, which is available to all authorized users of the portal.
The Technopark community is a large, friendly and active family. We communicate a lot, and many of our friends have a lot of ideas on how to improve the portal. Sometimes some software bugs pop up. So that all ideas and comments could be brought and discussed with us - the developers, we made a special system of bug reports.
Its peculiarity is that under each created topic a ticket is started in the tracker. When changing the status of the ticket (in work, waiting for the layout, ready) the author, all developers and all participants who subscribe to the request receive notifications. In this way, you can always find out what status a particular offer is in.
Our students help us not only in finding bugs, but also in implementing various features. For example, students have developed a mobile application for easy viewing of class schedules, from our side we have implemented only the API.
The application is available in the App Store and Google Play .
A year and a half ago at Moscow State University. Lomonosov launched a similar Technopark project - the Technosphere , the purpose of which is to train qualified programmers - researchers in the field of processing large amounts of data. During the year, students gain knowledge about the technologies of using and building a DBMS, methods for analyzing large amounts of data, multi-threaded programming, and much more.
To ensure the educational process in the Technosphere, we have expanded the functions of our portal, providing the opportunity to serve several educational institutions with one platform. So, we present to your attention the portal Tehnosphere . The functions of the portal Technosphere completely replicate the functions of Technopark, but has its own design and interface.
Already this September, our new educational project was launched in collaboration with MIPT - Tehnotrek , which is preparing mobile developers. Since after the introduction of Technosphere, our educational platform could serve several different training centers, it was not difficult for us to raise a new portal, and in just a couple of weeks, after the introduction of the design, we launched the registration of applicants for the first set in Tehnotrek.
It is time to talk a little about the architecture of the platform and some of the technological solutions used in the development of the portal.
Now the portal is fully implemented in Python 2.6 using the Django 1.6 framework.
For performance and quick return of pages, most objects are cached in memcached, and MySQL v5.4 as a database. Many portal operations are implemented asynchronously using Celery. Among such operations:
RabbitMQ is used as a backend to Celery. We used Sphinx to provide full-text search across all blogs and portal content. All content is indexed every 15 minutes. The Django application also interacts with two external services: Technotest is a system that we use to test applicants during recruitment, and a bug report system that collects all reports from educational platforms.
For video feeds we use the Wowza Streaming Engine . In the audience where the broadcast is conducted, VGADVI Broadcaster is installed, which connects the images from the projector and camera and transmits the picture and sound via RTP to the server from Wowza. In turn, on the client side, we use JW Player.
When designing the portal, we tried to normalize the structure of the database as much as possible, allowing for denormalization in places where it was reasonable. As a result, we have more than 130 tables.
While in the implementation of the portal on LiveStreet the main entities were blogs and topics, now the focus is on the essence of training programs. The structure of the curriculum data in a simplified form is given below.
After the end of the semester, the programs of the Technopark are modified on the basis of feedback from students. Each next semester creates a new version of the curriculum: the new version is a copy of the previous one, with the corrections made. A curriculum version consists of a specific set of disciplines — discipline versions, which in turn consist of a set of classes, homework assignments, and grades. The mark may be given for occupation (test), homework, or for additional activity not covered by the curriculum.
The curriculum versions are attached to the semesters, the semester in this case is the time frame: it indicates in which periods of time a particular program is taught. The study group also has a connection with the semester, thereby determining at what point in time the students were enrolled in one or another course of the curriculum. Students are associated with a study group where they study through a status that takes on four meanings:
It is training in any curriculum that determines access to each blog, while if a student withdraws from the curriculum, he loses access.
The blogs available to the user are denormalized in the database and are stored in a separate table, which allows you to implement the so-called closed blogs that are not included in any training program and are available to a limited number of users. Plus, this solution simplifies the query to the database for issuing available blogs and topics.
Such denormalizations are conveniently supported by the signaling mechanism in Django, especially if the models are in different applications or even libraries. In this case, we use two signals: changes in the connection of M2M blogs with the curriculum version and saving the entity linking the user to the study group, and since updating all available blogs is quite a lengthy operation, we remove it from the celery-task.
We use a similar mechanism for recalculating student ratings and achievements.
When a teacher gives points to a student, a record appears in a separate table, how many points (of the possible scores a given student has received). At the same time, a signal handler is called that starts a task to recalculate the rating. This task, in turn, summarizes the scores of all students of this curriculum and updates the ranking for all. A task is also launched, which is traversed through the list of not yet issued achievements and checks whether the student has scored the required number of points to achieve or not. So, at the mark on the lesson, it is triggered by a signal of a task to miscalculate the achievements in attendance.
Thus, the basis of the Technopark portal is organized, but this is only a small part of it. New ideas are constantly being born here, and we are not standing still. As of October 10, 2015, the backlog tracker includes another 233 unrealized ideas and unfixed defects with a total estimate of 800 hours. 2900 hours of work were spent on development, including 1700 hours of development, 75 hours of testing and 400 hours of work at the combat site. Gitlab tells us about 3,650 commits made since October 2014.
Now we proudly say that we do not have a mini-blog, but a real educational platform to provide an educational process that can be applied to most educational institutions.
How it all began
Three years ago, Technopark opened its doors - a project of the Mail.Ru Group and MSTU. N. E. Bauman. To build interaction with students, LiveStreet CMS was chosen for the start of the project. The teachers laid out topics with homework, and the schedule for students came in the form of weekly topics. This is how the portal looked like in its first release:
')
The project was developing, the basic functions of LiveStreet began to be lacking, and we began to expand the capabilities of the portal using a plugin system. We have added various modules and functional blocks, such as curriculum, schedule, events, personal offices of teachers and students. However, a year later, we were faced with the specific difficulties of updating the initially forum engine, and decided to completely rewrite it specifically for our needs. We present to your attention what came out of it.
Public part of the portal
The whole portal is divided into two parts. The open part is visible to everyone and includes several pages of a general description of Technopark, a section with a full description of programs and disciplines, as well as a schedule .
All lectures and classes in the Technopark are recorded, we post the processed videos in the “ Video recordings ” section. For applicants, the FAQ section has been developed, containing answers to most questions about the project.
Admission to the Technopark
Every six months, Technopark opens a set for those who want to delve into web development and web application architecture. In order to participate in the recruitment of one of the training programs, registering, applicants fill out a questionnaire.
A detailed questionnaire is needed in order to meet a potential Technopark student, introduce students to each other and get a general idea of ​​recruitment. After registering on the portal, the section “Personal Account of the Applicant” becomes available, where all users can learn about the kits currently open, apply for them or get acquainted with those who are already enrolled in Technopark programs.
Admission to the Technopark includes several stages: registration, online testing, thematic offline interview. After each stage, the results and decisions of the interviewees are recorded in the applicant's card. In the end, by pressing the “Enroll” button, the best of the best acquire a study group, become members of our community and gain access to the internal sections of the portal. They have the opportunity to take part in discussions, raise topics for discussion, creating topics both in general sections and in sections on disciplines.
After successful completion of the curriculum, graduates of the project fall into the section “ Issues ” - on the “honors board” of graduates.
Each graduate is awarded a diploma with a QR code.
With this code you can get directly to the page of the graduate - the owner of the diploma, which serves as confirmation of the official diploma.
Private part of the portal
The second, most of the portal, which has already been mentioned in passing, is a closed part, intended for students, teachers and administrators of Technopark.
Access to it is obtained only by confirmed members of our large family: students, graduates, teachers, and the administration of the portal. The basis of the basic functionality was, is and remains microblogging. Everything that is connected with life in a project - from classes and homework to solving organizational issues - we discuss here.
Personal account of the student and teacher
Students also get access to a private office where they can observe the grades they receive, their own progress and attendance. For success in training, we issue achievements. The best students - the coolest badges!
At the same time, teachers have a private office. It includes sections: journals for grading, feedback on occupations and attendance reports.
Yes, we ask all students at the end of classes to leave their opinion about him.
This information is stored without disclosing the identity of the person who left the review so that the comments of the students would not affect their further progress. According to the obtained data, we form various graphs.
Moreover, each participant has the opportunity to chat in a chat with other participants or as part of a group. Chat is built on WebSocket technology. All messages come instantly (Instant Messenger). For implementation, we used the development of Alexander Emelin Centrifuge , about which there is a detailed article on Habré .
On all pages, we display a mini-blog with classes that the viewing student or teacher will have on this day. The functionality of this unit includes a mark on the presence of the lesson, and after the lesson is completed, it is possible to leave a review.
Curriculum management
In the Technopark in the administrative part, it is possible to form all the training programs, ranging from a general description of the program and ending with a detailed indication of each class, each homework given out and each grade required.
An interface has been developed for scheduling on the basis of a program plan, an existing classroom fund, study groups and available time ranges for conducting classes. There are checks and leveling of erroneous situations, when at the same time the group has two classes in different disciplines.
Ultimately, all students and applicants receive complete and comprehensive information on the entire program of study.
During the semester, students attend classes, receive grades, teachers receive feedback and have the opportunity to adjust the subject or pace of the lecture in the next lesson. At the same time, we monitor not only the implementation of the curriculum by students, but also the work of teachers: how quickly they check their homework, how often they write in blogs.
Inevitably, a session begins - the time to pass the mid-term control, close debts and receive final grades in a discipline. Inevitably comes the end of the semester - the stage of transfers and contributions.
All this functionality is implemented in a closed part accessible to project administrators. You can select individual students and, with the click of a button, translate them into new study groups for future semesters. It is just as easy to expel a student. Our leaders are very sensitive to these buttons and at the end of each semester are asked to once again check the correctness of the work of these functions with the fixation of all changes. Just in case, we have provided the possibility of canceling these changes. The hand trembled, nervous ...
Based on the results of such modifications, notifications of decisions taken are sent to all parties. Students who have successfully completed all the training receive the status of "Graduate" (fall into the section "Issues") and receive the eternal right to remain a member of the community. Expelled students lose access to the inside of the portal.
Selection in Technopark
We also automated the process of selecting students for study programs. Before the start of the set, we configure it in the portal admin panel: choose which stages it will consist of, which external testing systems will be used. When we recruit students for the main program, we traditionally choose online testing and face-to-face interview. As a result, we get a table in which there are all applicants who have applied for this set, successive steps through which applicants can be transferred. If the applicant passes the last stage, it is automatically credited. Moreover, if the stage is carried out in an external system, for example, online testing, then automatic export of applicants to this system and export of their grades is provided.
For testing, we use another development of the Technopark laboratory: the Technotest online testing system, it is used both for admission testing and for some major controls.
The Technopark has optional open courses : training potential Technopark students and general disciplines for all students. Students of elective programs do not get access to all blogs of disciplines, because according to the rules of the project before enrolling in the main program they are not full-fledged students. At the same time, each participant can be a student of several programs, a graduate, and even a teacher at the same time.
activity
Technopark regularly hosts events and invites speakers. In the spring of 2015 there were performances:
- Master class D. Kibkalo "Mosigra: the creation of a non-tech business techie"
- Master class by D. Shkolnikov “Application of good old mathematics in modern internet marketing”
- Master class by V. Makishvili "36"
- I. Penyaev's master class "The story of a Bauman, or how to do what you like, and read about your features in the news"
- Master class by I. Sysoyev "Nginx: program and company"
- Master class of N. Chudinov "Modern trends and SEO strategies - successful cases and typical optimization errors"
- A. Kapterev's master class “Why Steve Jobs loved fonts: about the importance and fantastic beauty of small details in working with text”
To ensure recording of events, monitoring the number of volunteers and collecting feedback, the functions of the events were developed. After the speech, we ask you to leave your feedback about the event in a separate letter to all who participated. Guest reviews are later taken into account by the organizers to improve the quality of future workshops.
The announcement of all upcoming events is also duplicated on the pages for applicants and visitors. Take part in the role of listeners can everyone.
Due to the fact that the number of participants in events increases every year and we can no longer physically accommodate everyone in one student audience, we are broadcasting from the main audience of Technopark, which is available to all authorized users of the portal.
Bug reports
The Technopark community is a large, friendly and active family. We communicate a lot, and many of our friends have a lot of ideas on how to improve the portal. Sometimes some software bugs pop up. So that all ideas and comments could be brought and discussed with us - the developers, we made a special system of bug reports.
Its peculiarity is that under each created topic a ticket is started in the tracker. When changing the status of the ticket (in work, waiting for the layout, ready) the author, all developers and all participants who subscribe to the request receive notifications. In this way, you can always find out what status a particular offer is in.
Mobile app
Our students help us not only in finding bugs, but also in implementing various features. For example, students have developed a mobile application for easy viewing of class schedules, from our side we have implemented only the API.
The application is available in the App Store and Google Play .
Technosphere
A year and a half ago at Moscow State University. Lomonosov launched a similar Technopark project - the Technosphere , the purpose of which is to train qualified programmers - researchers in the field of processing large amounts of data. During the year, students gain knowledge about the technologies of using and building a DBMS, methods for analyzing large amounts of data, multi-threaded programming, and much more.
To ensure the educational process in the Technosphere, we have expanded the functions of our portal, providing the opportunity to serve several educational institutions with one platform. So, we present to your attention the portal Tehnosphere . The functions of the portal Technosphere completely replicate the functions of Technopark, but has its own design and interface.
Technotrack
Already this September, our new educational project was launched in collaboration with MIPT - Tehnotrek , which is preparing mobile developers. Since after the introduction of Technosphere, our educational platform could serve several different training centers, it was not difficult for us to raise a new portal, and in just a couple of weeks, after the introduction of the design, we launched the registration of applicants for the first set in Tehnotrek.
Architecture
It is time to talk a little about the architecture of the platform and some of the technological solutions used in the development of the portal.
Now the portal is fully implemented in Python 2.6 using the Django 1.6 framework.
For performance and quick return of pages, most objects are cached in memcached, and MySQL v5.4 as a database. Many portal operations are implemented asynchronously using Celery. Among such operations:
- mailing notifications;
- delivery of achievements to users;
- generating internal reports;
- receiving and sending data to external services;
- calculation of students rating.
RabbitMQ is used as a backend to Celery. We used Sphinx to provide full-text search across all blogs and portal content. All content is indexed every 15 minutes. The Django application also interacts with two external services: Technotest is a system that we use to test applicants during recruitment, and a bug report system that collects all reports from educational platforms.
For video feeds we use the Wowza Streaming Engine . In the audience where the broadcast is conducted, VGADVI Broadcaster is installed, which connects the images from the projector and camera and transmits the picture and sound via RTP to the server from Wowza. In turn, on the client side, we use JW Player.
When designing the portal, we tried to normalize the structure of the database as much as possible, allowing for denormalization in places where it was reasonable. As a result, we have more than 130 tables.
Data structure
While in the implementation of the portal on LiveStreet the main entities were blogs and topics, now the focus is on the essence of training programs. The structure of the curriculum data in a simplified form is given below.
After the end of the semester, the programs of the Technopark are modified on the basis of feedback from students. Each next semester creates a new version of the curriculum: the new version is a copy of the previous one, with the corrections made. A curriculum version consists of a specific set of disciplines — discipline versions, which in turn consist of a set of classes, homework assignments, and grades. The mark may be given for occupation (test), homework, or for additional activity not covered by the curriculum.
The curriculum versions are attached to the semesters, the semester in this case is the time frame: it indicates in which periods of time a particular program is taught. The study group also has a connection with the semester, thereby determining at what point in time the students were enrolled in one or another course of the curriculum. Students are associated with a study group where they study through a status that takes on four meanings:
- is trained;
- graduated;
- deducted;
- on sabbatical.
It is training in any curriculum that determines access to each blog, while if a student withdraws from the curriculum, he loses access.
The blogs available to the user are denormalized in the database and are stored in a separate table, which allows you to implement the so-called closed blogs that are not included in any training program and are available to a limited number of users. Plus, this solution simplifies the query to the database for issuing available blogs and topics.
Such denormalizations are conveniently supported by the signaling mechanism in Django, especially if the models are in different applications or even libraries. In this case, we use two signals: changes in the connection of M2M blogs with the curriculum version and saving the entity linking the user to the study group, and since updating all available blogs is quite a lengthy operation, we remove it from the celery-task.
We use a similar mechanism for recalculating student ratings and achievements.
When a teacher gives points to a student, a record appears in a separate table, how many points (of the possible scores a given student has received). At the same time, a signal handler is called that starts a task to recalculate the rating. This task, in turn, summarizes the scores of all students of this curriculum and updates the ranking for all. A task is also launched, which is traversed through the list of not yet issued achievements and checks whether the student has scored the required number of points to achieve or not. So, at the mark on the lesson, it is triggered by a signal of a task to miscalculate the achievements in attendance.
Finally
Thus, the basis of the Technopark portal is organized, but this is only a small part of it. New ideas are constantly being born here, and we are not standing still. As of October 10, 2015, the backlog tracker includes another 233 unrealized ideas and unfixed defects with a total estimate of 800 hours. 2900 hours of work were spent on development, including 1700 hours of development, 75 hours of testing and 400 hours of work at the combat site. Gitlab tells us about 3,650 commits made since October 2014.
Now we proudly say that we do not have a mini-blog, but a real educational platform to provide an educational process that can be applied to most educational institutions.
Source: https://habr.com/ru/post/271837/
All Articles