Experience in developing an automated proctoring system to confirm the results of online exams
In modern education, distance learning is becoming increasingly important. In many countries of the world, this format is widely used and represents a leading trend. Online learning has come to Russia recently and still raises many questions. The convenience of online learning is difficult to overestimate - it allows you to gain knowledge at a convenient time, in a convenient form and with minimal cost. However, when it comes to student certification, issuance of certificates, certificates and diplomas on the results of training and confidence in these results, the distance form, at first glance, is much inferior to full-time study. There are problems of verification of the identity and recognition of dishonest behavior of the subject (student) during testing. An important step in solving this problem is proctoring systems.
In Russia, the word proctoring is still hardly familiar to anyone. There is no clear idea about the functions of this format and how it is organized. In the classical sense, proctoring is a procedure for remotely accompanying online exams and verifying the subject's personality in order to increase the level of confidence in the results when a remote observer (proctor) follows the exam.
Of course, the presence of a qualified observer markedly improves the situation. The proctor can verify the student’s identity, monitor the online testing procedure via a webcam. However, as I will show later, the format of classical proctoring has a number of limitations. The final word in the field of online examinations is the development of systems for the partial or full automation of the functions of the proctor. Automation allows you not only to help the live observer and improve its efficiency, but even in some cases to replace the proctor.
')
I develop and implement an automated system for remote support of online exams at ProctorEdu. The system allows to partially or fully automate proctoring functions and can be integrated with other systems, such as online learning platforms. Over the past year, we have managed to make significant progress in this area and accumulate some experience, which we would like to share.
First, let's take a closer look at the classical scheme, when the Proctor is watching the procedure for remotely taking the exam online. Usually a proctor can accompany up to nine students at a time. The duties of Proctor include:
Such a variant of proctoring is usually called synchronous, since Proctor and student are online at one time. Another option is asynchronous procoring. In this case, the student independently passes the verification of the person and the exam itself, but his behavior during the exam and all his actions on the camera and on the computer are recorded. Then the proctor looks at the video and assesses the degree of confidence in the results obtained in the exam.
Both of these options are quite working, can increase the level of confidence in the results of examinations and protect themselves from personality substitution. But they have a number of drawbacks:
To eliminate these drawbacks, reduce the cost of maintaining the online exams and increase the speed of processing the results allows the automation of proctoring functions. Partial automation is designed to help the proctor handle a large number of exams, indicating the points that cause the most suspicion or require the attention of the proctor. It allows you to speed up the work of proctors and, as a result, reduce their number. Full automation allows you to completely exclude proctors while accompanying exams, but at this stage of technology development, this option still does not reach the quality level of a person’s work. Despite the fact that the accuracy of a fully automated system may be lower than with the participation of Proctor, in some cases this option is fully justified, especially when choosing the price / quality ratio, the advantage is towards reducing the cost and increasing the speed of work.
Proctoring system is technically difficult to implement, so it is not part of most online learning platforms. Proctoring systems developed separately integrates with such platforms, adding the missing functionality to them.
What have you managed to achieve? An examination script was developed on online learning platforms, which does not require the direct participation of Proctor, but does not exclude passive observation and even interference in the process (in text, audio and video format). At first, this scenario was long and contained at least nine stages of interaction between the student (course listener), the online learning platform and the proctoring system. Only in order to start the exam it was necessary to perform manually at least four stages in the proctoring system: verification of communication, photo of the person, photo of the document, verification of the identity of the proctor. In addition, it was necessary to interact alternately with both the online learning system and the proctoring system. All this resembled a quest that every student needed to pass, and caused many questions to the organizers.
Gradually, this path was shortened by automating individual steps and changing the way of integration with online learning platforms. Now the scenario of passing an exam for a student is as follows: the student enters the website of the online learning system, starts the exam, passes the exam in any form (test, task, virtual laboratory, etc.), finishes the exam and gets the result. In other words, for a student, the exam scenario in the online training system does not change, and the proctoring system is automatically connected at the moment of the exam start, monitors the student throughout the exam and finishes his work at the moment of completion of the exam. It looks simple, all the complexity is hidden from the student.
Embedding proctoring functions in an online learning system exam (LMS) scenario
In more detail about each stage:
I would like to note once again that, in this case, the proctoring procedure almost does not change the scenario of passing the exam for a student, all functions work in the background without the need for intervention from the student. The student will only need to give access to the webcam and microphone in the browser during the first interaction with the system.
Now I want to tell in more detail how the subsystem of detecting dishonest behavior during the exam.
The structure of the automatic detection of dishonest behavior
Throughout the exam, data is automatically collected on dynamic indicators, including computer tracking, face and voice detection, and biometrics. The collected data is analyzed and processed, after which an automatic assessment of the quality of passing the exam from 0 to 100% is formed. On the basis of the formed assessment, a conclusion on the session can be automatically made according to predetermined criteria.
The main focus is on the fact that for passing the exam does not require any special equipment, in addition to the fact that there are usually: a webcam, microphone, keyboard, screen. Data is collected on the following events:
Automatic assessment of the quality of the exam is calculated according to the following formula:
,
where N is the number of minutes in the exam, x1, x2, ... xk is the data of the estimated parameters in percent, E is the assessment of the quality of passing the exam in percent. The function f (x) describes the method for selecting the most significant events. Examples of functions f (x):
Thus, the assessment shows the averaged percentage of the most significant recorded events for the entire exam, starting from the moment the student starts the exam and until the end of the exam. If the percentage of events per minute exceeds a certain threshold value, then the violation is fixed.
And here is a video showing the work of the event tracking subsystem:
Each passed exam on the proctoring procedure includes an automatic assessment from 0 to 100%, a photo of the student’s face and an interactive protocol that contains video from a webcam and a screen (optional), minute-by-minute detailing of recorded events and other exam data.
The stored information does not contain personal data, during the exam there is no need to show documents to the camera, and the saved photos of persons do not comply with GOST R ISO / IEC 19794-5-2006 , therefore they cannot be considered biometric data.
When issuing certificates, certificates, diplomas, verification of the identity of only those students who have successfully reached the end of the curriculum is carried out, thereby reducing the total amount of work. To confirm the identity of the student, the school may request the necessary data from its student, and then for each exam compare the photos of the person saved by the proctoring system with a photo in an identity document. The procedure for comparing photographs of a person on documents has yet to be automated.
Summarizing, I would like to add that, following distance learning, technologies for distance passing exams with confirmation of identity and confirmation of the results of such examinations are being developed. I would like our country to develop educational technologies and to be able to compete with foreign world leaders. There is still a lot of work to do.
In Russia, the word proctoring is still hardly familiar to anyone. There is no clear idea about the functions of this format and how it is organized. In the classical sense, proctoring is a procedure for remotely accompanying online exams and verifying the subject's personality in order to increase the level of confidence in the results when a remote observer (proctor) follows the exam.
Of course, the presence of a qualified observer markedly improves the situation. The proctor can verify the student’s identity, monitor the online testing procedure via a webcam. However, as I will show later, the format of classical proctoring has a number of limitations. The final word in the field of online examinations is the development of systems for the partial or full automation of the functions of the proctor. Automation allows you not only to help the live observer and improve its efficiency, but even in some cases to replace the proctor.
')
I develop and implement an automated system for remote support of online exams at ProctorEdu. The system allows to partially or fully automate proctoring functions and can be integrated with other systems, such as online learning platforms. Over the past year, we have managed to make significant progress in this area and accumulate some experience, which we would like to share.
First, let's take a closer look at the classical scheme, when the Proctor is watching the procedure for remotely taking the exam online. Usually a proctor can accompany up to nine students at a time. The duties of Proctor include:
- The subject's identity is usually a visual verification of the face of a person with a photo on an identity document, which the student shows in a webcam;
- Monitoring the student throughout the exam and tracking possible violations of the rules for passing the exam, cheating attempts or other dishonest behavior.
Such a variant of proctoring is usually called synchronous, since Proctor and student are online at one time. Another option is asynchronous procoring. In this case, the student independently passes the verification of the person and the exam itself, but his behavior during the exam and all his actions on the camera and on the computer are recorded. Then the proctor looks at the video and assesses the degree of confidence in the results obtained in the exam.
Both of these options are quite working, can increase the level of confidence in the results of examinations and protect themselves from personality substitution. But they have a number of drawbacks:
- performance (scalability) is directly dependent on the number of proctors who will be online or view videos;
- the time delay when assessing the exam in the case of asynchronous proctor; it takes time to view the video recordings;
- the cost of an hour of work of a proctor is relatively high, the more exams, the more proctors need to be involved in the work;
- the complexity of organizing the work of a large number of proctors;
- the human factor - the bad faith or lack of commitment of a proctor can affect quality.
To eliminate these drawbacks, reduce the cost of maintaining the online exams and increase the speed of processing the results allows the automation of proctoring functions. Partial automation is designed to help the proctor handle a large number of exams, indicating the points that cause the most suspicion or require the attention of the proctor. It allows you to speed up the work of proctors and, as a result, reduce their number. Full automation allows you to completely exclude proctors while accompanying exams, but at this stage of technology development, this option still does not reach the quality level of a person’s work. Despite the fact that the accuracy of a fully automated system may be lower than with the participation of Proctor, in some cases this option is fully justified, especially when choosing the price / quality ratio, the advantage is towards reducing the cost and increasing the speed of work.
Proctoring system is technically difficult to implement, so it is not part of most online learning platforms. Proctoring systems developed separately integrates with such platforms, adding the missing functionality to them.
What have you managed to achieve? An examination script was developed on online learning platforms, which does not require the direct participation of Proctor, but does not exclude passive observation and even interference in the process (in text, audio and video format). At first, this scenario was long and contained at least nine stages of interaction between the student (course listener), the online learning platform and the proctoring system. Only in order to start the exam it was necessary to perform manually at least four stages in the proctoring system: verification of communication, photo of the person, photo of the document, verification of the identity of the proctor. In addition, it was necessary to interact alternately with both the online learning system and the proctoring system. All this resembled a quest that every student needed to pass, and caused many questions to the organizers.
Gradually, this path was shortened by automating individual steps and changing the way of integration with online learning platforms. Now the scenario of passing an exam for a student is as follows: the student enters the website of the online learning system, starts the exam, passes the exam in any form (test, task, virtual laboratory, etc.), finishes the exam and gets the result. In other words, for a student, the exam scenario in the online training system does not change, and the proctoring system is automatically connected at the moment of the exam start, monitors the student throughout the exam and finishes his work at the moment of completion of the exam. It looks simple, all the complexity is hidden from the student.
Embedding proctoring functions in an online learning system exam (LMS) scenario
In more detail about each stage:
- The exam that the student opens at the LMS is automatically registered in the proctoring system and associated with that student's account at the LMS. Since the student is already authorized in the LMS, the authorization in the proctoring system occurs automatically on the token imperceptible to the student.
- Before the exam, the user's computer and network are automatically checked for compliance with the minimum system requirements. The procedure takes a few seconds, if no problems are detected, then the stage passes unnoticed by the student. For the first time, the student is asked for permission to access the webcam and microphone.
- After checking the connection, the background observation of the student is launched, continuous data collection and analysis is carried out. In case of detection of deviations in the student's behavior from the norm, a corresponding notice may be issued so that the student can correct.
- In the first minute of the exam, learning of the subsystem of continuous verification of the person on the face using facial recognition methods takes place. After that, the system automatically monitors the student's face in order to track the fact of identity substitution at the computer during the exam. In addition to this, identity verification is carried out using keyboard handwriting. The algorithm works in such a way that it learns the set of the first 100 characters, and then it gets trained on the new data. This verification method can be used in tasks where text input is required as an answer, for example, in programming tasks. Text can be entered in any language, the algorithm does not take into account the input language.
- After training the subsystem verification of identity of a person who passes the first minute of the exam, a photo of the student’s face is saved and attached to the current exam. The algorithm itself chooses the best photo from the video stream, which allows you to always get the best quality photo of a person without the direct participation of the student.
- After completion of the exam, the results are available immediately after processing within a few minutes.
I would like to note once again that, in this case, the proctoring procedure almost does not change the scenario of passing the exam for a student, all functions work in the background without the need for intervention from the student. The student will only need to give access to the webcam and microphone in the browser during the first interaction with the system.
Now I want to tell in more detail how the subsystem of detecting dishonest behavior during the exam.
The structure of the automatic detection of dishonest behavior
Throughout the exam, data is automatically collected on dynamic indicators, including computer tracking, face and voice detection, and biometrics. The collected data is analyzed and processed, after which an automatic assessment of the quality of passing the exam from 0 to 100% is formed. On the basis of the formed assessment, a conclusion on the session can be automatically made according to predetermined criteria.
The main focus is on the fact that for passing the exam does not require any special equipment, in addition to the fact that there are usually: a webcam, microphone, keyboard, screen. Data is collected on the following events:
- switched focus to a third-party application or tab (b1);
- Exam page not full screen (b2);
- no face in front of the camera (c1);
- unauthorized persons in front of the camera (c2);
- unidentified face in front of the camera (c3);
- microphone is turned off or low volume (m1);
- conversation or noise in the background (m2);
- keyboard style not recognized (k1);
- student not in session (n1).
Automatic assessment of the quality of the exam is calculated according to the following formula:
,where N is the number of minutes in the exam, x1, x2, ... xk is the data of the estimated parameters in percent, E is the assessment of the quality of passing the exam in percent. The function f (x) describes the method for selecting the most significant events. Examples of functions f (x):
- maximum percentage of events:
; - weighted average event percentage:
.
Thus, the assessment shows the averaged percentage of the most significant recorded events for the entire exam, starting from the moment the student starts the exam and until the end of the exam. If the percentage of events per minute exceeds a certain threshold value, then the violation is fixed.
And here is a video showing the work of the event tracking subsystem:
Each passed exam on the proctoring procedure includes an automatic assessment from 0 to 100%, a photo of the student’s face and an interactive protocol that contains video from a webcam and a screen (optional), minute-by-minute detailing of recorded events and other exam data.
The stored information does not contain personal data, during the exam there is no need to show documents to the camera, and the saved photos of persons do not comply with GOST R ISO / IEC 19794-5-2006 , therefore they cannot be considered biometric data.
When issuing certificates, certificates, diplomas, verification of the identity of only those students who have successfully reached the end of the curriculum is carried out, thereby reducing the total amount of work. To confirm the identity of the student, the school may request the necessary data from its student, and then for each exam compare the photos of the person saved by the proctoring system with a photo in an identity document. The procedure for comparing photographs of a person on documents has yet to be automated.
Summarizing, I would like to add that, following distance learning, technologies for distance passing exams with confirmation of identity and confirmation of the results of such examinations are being developed. I would like our country to develop educational technologies and to be able to compete with foreign world leaders. There is still a lot of work to do.
Source: https://habr.com/ru/post/325296/
All Articles