An open proctoring system for remote tracking of online exams
Last summer, the ITMO University held a distance examination for entering graduate school, and in January of this year, students of the National Open Education Platform courses wishing to receive a confirmed certificate were given the opportunity to complete final certification for the chosen course under the supervision of a proctor. All this was organized on the basis of an open system of remote supervision, specially designed for this purpose. In this article I will try to review the system, as well as talk about some of the features of its development.
The remote supervision system (hereinafter referred to as VOS) is intended to accompany the process of remotely passing examinations, confirm the identity of the subject and confirm the results of his certification. Of course, this system is not unique, there are foreign services that provide remote assistance services, such as ProctorU , Mettl Online Assessment , ProctorExam or Pearson VUE , but there is a domestic service Examus and ProctorEdu system. However, the developed VOS has a main difference - it is not a service, but open source software that anyone can use for their own purposes, whether it is organizing their own proctoring center, conducting confirmed attestations in their e-learning system or something else . Of course, for the full functioning of the system, people are needed (proctors), workplaces with cameras and audio headsets, as well as server equipment and system administrators for the deployment and maintenance of the system. But when the necessary infrastructure is already there, the question of software remains. And it is this question that the given system answers.
Work with VOS is divided into two main parts: the interface of the subject and the interface of the proctor. First, the subject downloads and launches the application, logs in and sees the list of exams available to him for passing in proctoring mode (the list of exams is created in advance, or automatically loaded from the LMS via API). Then he can schedule the time for passing these exams, for each scheduled exam, the subject is assigned the currently available proctor. Available time is formed on the basis of the schedule that the proctors make up, they have an appropriate interface for this. Then the student connects to the proctor at the time indicated by him and the exam begins. The proctor accompanies the subject throughout the exam, his duties include conducting a procedure for identifying and tracking violations of the exam rules. After completion of the proctoring session, the student sees the conclusion of the proctor for this exam and his commentary.
')
Proctor works with VOS also through the application. The main interface of the proctor is a table of exam schedules, where he sees all the exams to which he is assigned, and their status, which changes in real time. When a student connects to the exam, the proctor sees it, connects to it, and the exam begins. Proctor sees the test through a webcam, and also sees his desktop. Before the exam begins, the proctor must conduct an identification procedure in which the student shows an identity document, and the proctor verifies the specified data in the subject's profile with the photograph of the document and the person tested on the webcam. If the identification is successful, the data and photograph of the document are saved in the exam protocol, and the subject is given access to the exam in the LMS. After that, the subject enters the LMS and proceeds to perform the tasks. Throughout the exam, the proctor can take notes and save screenshots in the exam protocol, which can be useful in resolving controversial issues in the future. At the end of the exam, the proctor signs the protocol and issues a conclusion, which is displayed to the student and transferred to the LMS via API. In case of violation of the rules for passing the exam, the proctor may interrupt the exam indicating the reason for the early completion of the exam. Proctor can simultaneously monitor several subjects, the number of simultaneous sessions is limited by the capabilities of the proctor, his computer's resources and network bandwidth.
The overall VOS architecture is shown below. All used technologies are open and distributed under free licenses.
The system is completely written in JavaScript, the cloud IDE Cloud9 was used as the development environment. The main parts of the system are the server on node.js and express, the MongoDB database with storage of small files in the GridFS, the Kurento media server, the media archive storage (video recording of proctoring sessions) with access via WebDAV protocol and the dextop application on the node-webkit (as an alternative the application can use the Chrome special extension to work from a web browser).
WebRTC and the Kurento media server were chosen as the basis for transmitting video to organize and manage media streams between clients, as well as to record these streams. At the moment, Kurento satisfies all the needs both in terms of functionality and performance, but other similar solutions worth mentioning are Licode and Jitsi Videobridge .
Authorization in VOS is based on the PassportJS module, currently login by password and OAuth is supported. If necessary, you can add an authorization strategy from those already supported in PassportJS or write your own, as was done for authorization through the unified entrance system of the ITMO University (for this purpose, a special strategy called passport-ifmosso was written ).
The client is implemented as a one-page web application that can work in a web browser, but the screen capture function and screenshots are available only after installing a special Chrome extension or through the special node-webkit application, which is available for Windows, Linux and OS X platforms. The application is especially useful if you do not want to bind to the web browser and its versions, where, in addition, third-party extensions may be installed that can affect the operation of the application in the browser, which is undesirable.
The DSS does not contain the functions of a Learning Management System (LMS), but can be associated with it through an API. To interact with the National Open Education Platform (NPOO), a special API was implemented, which can be extended to interact with other LMSs. The figure below shows a diagram of the interaction of system participants using such an API.
Within one session, several video streams are transmitted: the camera of the subject, the camera of the proctor and the desktop of the subject. The parameters of each stream are set in the application settings, where you can set the video and audio source, maximum resolution and frame rate. To check the quality of communication and pictures, there is an appropriate interface that allows you to know in advance about possible technical problems.
All video and audio sessions of the proctoring session are recorded. Session video recordings are stored on a separate storage with access via WebDAV protocol, which allows you to logically separate the storage from the system itself. Thanks to the WebDAV protocol, you can, for example, use Yandex.Disk to store videos. Later, these records and exam records can be viewed through the VOS interface. A special module has been written for transcoding video recordings and uploading them to the server, a description of which can be found in a separate article .
The application uses the exact server time, which does not depend on the computer clock, for ease of perception, the time is adjusted according to the client's time zone.
To access the VOS database, a special JSON REST API with flexible configuration of access rights and data output parameters is implemented. For example, you can configure access by token to a trusted server that will aggregate and process statistics. This API fully supports CRUD (create, read, update, and delete) for working with MongoDB collections based on Mongoose models created in VDS, and also supports the original Mongo query syntax.
At the moment, about 120 proctoring sessions have been conducted through the system, but this year several thousand sessions are planned to be held. The system can be deployed independently, or use a ready-made Vagrant box. Instructions and source codes are posted on GitHub .
The remote supervision system (hereinafter referred to as VOS) is intended to accompany the process of remotely passing examinations, confirm the identity of the subject and confirm the results of his certification. Of course, this system is not unique, there are foreign services that provide remote assistance services, such as ProctorU , Mettl Online Assessment , ProctorExam or Pearson VUE , but there is a domestic service Examus and ProctorEdu system. However, the developed VOS has a main difference - it is not a service, but open source software that anyone can use for their own purposes, whether it is organizing their own proctoring center, conducting confirmed attestations in their e-learning system or something else . Of course, for the full functioning of the system, people are needed (proctors), workplaces with cameras and audio headsets, as well as server equipment and system administrators for the deployment and maintenance of the system. But when the necessary infrastructure is already there, the question of software remains. And it is this question that the given system answers.
Work with VOS is divided into two main parts: the interface of the subject and the interface of the proctor. First, the subject downloads and launches the application, logs in and sees the list of exams available to him for passing in proctoring mode (the list of exams is created in advance, or automatically loaded from the LMS via API). Then he can schedule the time for passing these exams, for each scheduled exam, the subject is assigned the currently available proctor. Available time is formed on the basis of the schedule that the proctors make up, they have an appropriate interface for this. Then the student connects to the proctor at the time indicated by him and the exam begins. The proctor accompanies the subject throughout the exam, his duties include conducting a procedure for identifying and tracking violations of the exam rules. After completion of the proctoring session, the student sees the conclusion of the proctor for this exam and his commentary.
')
The interface of the test before the exam
Proctor works with VOS also through the application. The main interface of the proctor is a table of exam schedules, where he sees all the exams to which he is assigned, and their status, which changes in real time. When a student connects to the exam, the proctor sees it, connects to it, and the exam begins. Proctor sees the test through a webcam, and also sees his desktop. Before the exam begins, the proctor must conduct an identification procedure in which the student shows an identity document, and the proctor verifies the specified data in the subject's profile with the photograph of the document and the person tested on the webcam. If the identification is successful, the data and photograph of the document are saved in the exam protocol, and the subject is given access to the exam in the LMS. After that, the subject enters the LMS and proceeds to perform the tasks. Throughout the exam, the proctor can take notes and save screenshots in the exam protocol, which can be useful in resolving controversial issues in the future. At the end of the exam, the proctor signs the protocol and issues a conclusion, which is displayed to the student and transferred to the LMS via API. In case of violation of the rules for passing the exam, the proctor may interrupt the exam indicating the reason for the early completion of the exam. Proctor can simultaneously monitor several subjects, the number of simultaneous sessions is limited by the capabilities of the proctor, his computer's resources and network bandwidth.
Exam table in the interface of the proctor
Proctor interface, identification procedure
The overall VOS architecture is shown below. All used technologies are open and distributed under free licenses.
The system is completely written in JavaScript, the cloud IDE Cloud9 was used as the development environment. The main parts of the system are the server on node.js and express, the MongoDB database with storage of small files in the GridFS, the Kurento media server, the media archive storage (video recording of proctoring sessions) with access via WebDAV protocol and the dextop application on the node-webkit (as an alternative the application can use the Chrome special extension to work from a web browser).
WebRTC and the Kurento media server were chosen as the basis for transmitting video to organize and manage media streams between clients, as well as to record these streams. At the moment, Kurento satisfies all the needs both in terms of functionality and performance, but other similar solutions worth mentioning are Licode and Jitsi Videobridge .
Authorization in VOS is based on the PassportJS module, currently login by password and OAuth is supported. If necessary, you can add an authorization strategy from those already supported in PassportJS or write your own, as was done for authorization through the unified entrance system of the ITMO University (for this purpose, a special strategy called passport-ifmosso was written ).
The client is implemented as a one-page web application that can work in a web browser, but the screen capture function and screenshots are available only after installing a special Chrome extension or through the special node-webkit application, which is available for Windows, Linux and OS X platforms. The application is especially useful if you do not want to bind to the web browser and its versions, where, in addition, third-party extensions may be installed that can affect the operation of the application in the browser, which is undesirable.
The DSS does not contain the functions of a Learning Management System (LMS), but can be associated with it through an API. To interact with the National Open Education Platform (NPOO), a special API was implemented, which can be extended to interact with other LMSs. The figure below shows a diagram of the interaction of system participants using such an API.
Within one session, several video streams are transmitted: the camera of the subject, the camera of the proctor and the desktop of the subject. The parameters of each stream are set in the application settings, where you can set the video and audio source, maximum resolution and frame rate. To check the quality of communication and pictures, there is an appropriate interface that allows you to know in advance about possible technical problems.
All video and audio sessions of the proctoring session are recorded. Session video recordings are stored on a separate storage with access via WebDAV protocol, which allows you to logically separate the storage from the system itself. Thanks to the WebDAV protocol, you can, for example, use Yandex.Disk to store videos. Later, these records and exam records can be viewed through the VOS interface. A special module has been written for transcoding video recordings and uploading them to the server, a description of which can be found in a separate article .
The application uses the exact server time, which does not depend on the computer clock, for ease of perception, the time is adjusted according to the client's time zone.
To access the VOS database, a special JSON REST API with flexible configuration of access rights and data output parameters is implemented. For example, you can configure access by token to a trusted server that will aggregate and process statistics. This API fully supports CRUD (create, read, update, and delete) for working with MongoDB collections based on Mongoose models created in VDS, and also supports the original Mongo query syntax.
Minimum technical requirements
| Parameter | Minimum requirements |
|---|---|
| Client part | |
| operating system | Windows XP +; OS X 10.7+; Linux |
| CPU | Intel i3 1.2 GHz or equivalent |
| Network connection speed | 1 Mbit / s |
| Free disk space | 100 MB |
| Free RAM | 1 GB |
| Webcam resolution | 640x480 |
| Webcam frame rate | 15 frames / s |
| Monitor screen resolution | 1280x720 |
| Server part | |
| operating system | Ubuntu 14.04 (64 bits) |
| CPU | AMD Six-Core Opteron 2427 2.2 GHz or equivalent |
| Average processor load | 5% / session |
| RAM | 2 GB + 100 MB / session |
| Network connection | 1.5 Mbit / s / session |
| Burn to disc | 150 KB / s / session |
| Disk space | 10 GB + 500 MB / hour / session |
| Video Archiving | 100 MB / hour / session (5 frames / s with sound) |
At the moment, about 120 proctoring sessions have been conducted through the system, but this year several thousand sessions are planned to be held. The system can be deployed independently, or use a ready-made Vagrant box. Instructions and source codes are posted on GitHub .
Media mentions
Source: https://habr.com/ru/post/277147/
All Articles