Experience in the development of a simulator program for the practical work of students
Faced with the need to check six dozen student work (6 practical work * 10 students), I realized that this process should be automated. Not to mention the difficulty of checking handwritten solutions, we must somehow solve the problem of cheating. Even when I was a student, I had the opportunity to take internships in one of the disciplines on a computer with a specially developed control and training simulator (under DOS). Now I wanted to repeat this experience.
I’ll clarify that this is a simulator , and not a testing system . I would distinguish the difference between them as follows: the testing system receives a set of tests and a set of answers to them, and in the course of work gives the student a test and checks the coincidence with the answer. On the contrary, the simulator contains algorithms for solving each task, so each time a unique task is generated and its solution is carried out, and then the student’s answer is checked against the system’s response. The advantage of this option is that you can not bypass the system by creating a database of questions and answers. Each newly generated task has to be solved from scratch.
The question of the choice of language and technology of development of the simulator was not. It is necessary to provide external students and those lagging behind the opportunity to take work without leaving home. Therefore, only a web application, and therefore PHP as the most convenient language for me. DBMS - MySQL.
The discipline for which I needed to check the practical work is called Information Retrieval. The tasks are from the book by Christopher D. Manning, Prabhakar Raghavan, Heinrich Schutz. Introduction to information retrieval . The need for natural language processing has complicated the matter.
')
First of all, it was necessary to create a collection of documents from which tasks will be generated. I started with a dictionary of Russian proverbs, but I had to abandon this idea because of the complexity of the morphological analysis of the Russian language and take the same English-language dictionary, specifying in the assignment that the different morphological forms of the word are considered different words, and indicating the regular expression, according to which the testing script will break sentences into words.
For many tasks, the generation of source data has also proved to be a difficult task. For example, building an inverted index of a collection of documents requires that proverbs used as such “documents” have several common words among themselves. Otherwise, the task will not be interesting to solve. Therefore, we had to come up with a whole algorithm: choose a random word, find all the proverbs in which it is used, compare them in pairs, count for each pair the number of common words and write the result of those proverbs where this number exceeds the specified threshold. If the resulting list of items is smaller than another predetermined threshold, then start over. It is clear that it is difficult to generate such things on the fly, so we had to index all proverbs at the same time, breaking them up by words and writing the result into the database. Complicated? And this algorithm had to come up with almost every job!
The form of recording the answer is also not always a trivial question. Sometimes it is enough to write down a number or a string, but these numbers or strings are easily calculated by eye and do not require the use of any formulas. And it is necessary to check the knowledge of formulas! Therefore, the form of the response record varied. Somewhere usual textbox, somewhere textarea. But for the problem of calculating the Levenshtein distance between two words, we had to derive a whole matrix, since it is important to check that the student understood the calculation algorithm.
Regarding the procedure for working with the simulator , registration is required to gain access to the tasks. A logged-in user sees a list of all jobs and all tasks, and those tasks that he passed even appear without a hyperlink - access to them is denied.
Error corrections are not foreseen - each time the task is accessed, the script generates it again.
I will tell you now about how students perceived such a system of control of their knowledge. At first, a lot of nerves caused the need to re-solve the problem again with the slightest error. Soon, two ways were found to circumvent this circumstance. The first is to use the Opera browser, in which when you click the Back button, the entire page was restored along with the old task. The second is to replace the post-request, in which not only the student’s answer, but also the task, in hidden elements of the html form, was sent for verification. After a couple of classes, students decided that it was possible to use a post-request substitution to send not their old task, but someone else's. Thus learned to write off even here.
However, not everything was written off, and in any case one solution must definitely be made, albeit for a few people. Therefore, the type of computer class in our practices was a wonderful picture of the intense mental work of all students.
To view the progress I made a script that displayed a list of passed tasks, sorted either by descending date-time or by students.
Now you can open this list on Sunday and see that while you are resting, your students continue to take the tasks.
What conclusions can be drawn from this experience? Of course, the simulator needs to be improved. It is necessary to limit the number of attempts to surrender the task and prevent the possibility of replacing the original data. I decided that this semester I will only observe the process of putting / walking / hacking (whether it is not enough) of the system and will not fix anything. In the future, I think to adapt this simulator for other disciplines - wherever there are tasks.
Unfortunately, it is unlikely to check the program code written by students in laboratory work in the same way. Only the black box method could be used here, as in the ACM / ICPC programming contests. But then you need to look for ways to check the work on plagiarism. In addition, no program will replace a human teacher.
That's all. Thanks to those who read the article to the end. Waiting for comments, comments and tips.
You can test the simulator here: science-search.ru/ir_students Login: test , password: test .
I’ll clarify that this is a simulator , and not a testing system . I would distinguish the difference between them as follows: the testing system receives a set of tests and a set of answers to them, and in the course of work gives the student a test and checks the coincidence with the answer. On the contrary, the simulator contains algorithms for solving each task, so each time a unique task is generated and its solution is carried out, and then the student’s answer is checked against the system’s response. The advantage of this option is that you can not bypass the system by creating a database of questions and answers. Each newly generated task has to be solved from scratch.
The question of the choice of language and technology of development of the simulator was not. It is necessary to provide external students and those lagging behind the opportunity to take work without leaving home. Therefore, only a web application, and therefore PHP as the most convenient language for me. DBMS - MySQL.
The discipline for which I needed to check the practical work is called Information Retrieval. The tasks are from the book by Christopher D. Manning, Prabhakar Raghavan, Heinrich Schutz. Introduction to information retrieval . The need for natural language processing has complicated the matter.
')
First of all, it was necessary to create a collection of documents from which tasks will be generated. I started with a dictionary of Russian proverbs, but I had to abandon this idea because of the complexity of the morphological analysis of the Russian language and take the same English-language dictionary, specifying in the assignment that the different morphological forms of the word are considered different words, and indicating the regular expression, according to which the testing script will break sentences into words.
For many tasks, the generation of source data has also proved to be a difficult task. For example, building an inverted index of a collection of documents requires that proverbs used as such “documents” have several common words among themselves. Otherwise, the task will not be interesting to solve. Therefore, we had to come up with a whole algorithm: choose a random word, find all the proverbs in which it is used, compare them in pairs, count for each pair the number of common words and write the result of those proverbs where this number exceeds the specified threshold. If the resulting list of items is smaller than another predetermined threshold, then start over. It is clear that it is difficult to generate such things on the fly, so we had to index all proverbs at the same time, breaking them up by words and writing the result into the database. Complicated? And this algorithm had to come up with almost every job!
The form of recording the answer is also not always a trivial question. Sometimes it is enough to write down a number or a string, but these numbers or strings are easily calculated by eye and do not require the use of any formulas. And it is necessary to check the knowledge of formulas! Therefore, the form of the response record varied. Somewhere usual textbox, somewhere textarea. But for the problem of calculating the Levenshtein distance between two words, we had to derive a whole matrix, since it is important to check that the student understood the calculation algorithm.
Regarding the procedure for working with the simulator , registration is required to gain access to the tasks. A logged-in user sees a list of all jobs and all tasks, and those tasks that he passed even appear without a hyperlink - access to them is denied.
Error corrections are not foreseen - each time the task is accessed, the script generates it again.
I will tell you now about how students perceived such a system of control of their knowledge. At first, a lot of nerves caused the need to re-solve the problem again with the slightest error. Soon, two ways were found to circumvent this circumstance. The first is to use the Opera browser, in which when you click the Back button, the entire page was restored along with the old task. The second is to replace the post-request, in which not only the student’s answer, but also the task, in hidden elements of the html form, was sent for verification. After a couple of classes, students decided that it was possible to use a post-request substitution to send not their old task, but someone else's. Thus learned to write off even here.
However, not everything was written off, and in any case one solution must definitely be made, albeit for a few people. Therefore, the type of computer class in our practices was a wonderful picture of the intense mental work of all students.
To view the progress I made a script that displayed a list of passed tasks, sorted either by descending date-time or by students.
Now you can open this list on Sunday and see that while you are resting, your students continue to take the tasks.
What conclusions can be drawn from this experience? Of course, the simulator needs to be improved. It is necessary to limit the number of attempts to surrender the task and prevent the possibility of replacing the original data. I decided that this semester I will only observe the process of putting / walking / hacking (whether it is not enough) of the system and will not fix anything. In the future, I think to adapt this simulator for other disciplines - wherever there are tasks.
Unfortunately, it is unlikely to check the program code written by students in laboratory work in the same way. Only the black box method could be used here, as in the ACM / ICPC programming contests. But then you need to look for ways to check the work on plagiarism. In addition, no program will replace a human teacher.
That's all. Thanks to those who read the article to the end. Waiting for comments, comments and tips.
You can test the simulator here: science-search.ru/ir_students Login: test , password: test .
Source: https://habr.com/ru/post/268363/
All Articles