Exam: as it was

On the exam a lot of copies broken. At the same time, however, on Habré, basically, there was an analysis of comical or incorrect assignments and only one view from the inside through the eyes of the admin , to whom there was an increased interest, which is not surprising. Since our company has been in this project for a long time, we also have something to tell. ABBYY participated:

• in the design of the examination results processing architecture using the ABBYY FormReader Developer Edition product (now called ABBYY FlexiCapture Engine ),
• in the training of regional specialists,
• in implementation,
• technical support.

So, continue to look at the exam from the inside.

Story

The history of the Unified State Exam begins in 1999, when the Center for Strategic Research was tasked to analyze the possibility of holding a unified state exam in Russia. The basic requirements were that the exam was to be massive, independent, objective, comparable.
')
Alexey Kalinin, lead manager for corporate projects at ABBYY Russia, notes that the planning was quite literate: the main scenarios and the overwhelming majority of exceptional projects were described from the very beginning, which helped to avoid additional confusion in an already difficult project. But even here it was not without various problems, about which (and not only about them) you can read below.

In 2001, 5 regions and 30 thousand graduates joined in testing the USE. At that time, the Unified State Exam is not quite the Unified State Exam, but in fact “centralized testing” conducted by the Federal Testing Center (FTC), because for the true way, the Unified State Examination has not yet purchased hardware or software. There is only a cunning plan :)

In general, the scheme initially looked quite simple: the operator monitors the scanner at the scanning station, the scans are transmitted to the recognition station, where you guess what happens. After the end of this mysterious automatic process, the result is transmitted to the verification station, where the operator has the opportunity to compare the result of recognition with the original (in electronic form, of course).

In 2002 there were already 16 regions, 300,000 participants gathered in them. And it was already a real exam. Forms that are amenable to automatic processing were processed automatically, but not without problems: the “scan-recognition-verification” chain was arranged in such a way that when a human verifier was distracted, the performance of the entire chain fell. Why should he be distracted? The fact is that the filling of forms is fairly rigidly formalized. Therefore, when any mark “not according to the charter” is found, for example, the barcode on one of the forms is shaded (the student is bored, what can you do), the verifier is obliged to involve specialists who will fix the problem. At the same time, such actions could often be accompanied by the collection of the commission and the completion of acts. In the same 2002 another problem emerges with the second part of the forms, which only experts can solve. It consisted in the fact that the automatic distribution of work among the experts checking the work was not implemented and, as a result, some chaos arose.

In 2003, the majority of graduates took part in the unified exam. To eliminate the problems, the mechanisms of automatic distribution of tasks of the second part between experts are implemented (for them, an area with the answers of the “C” forms is automatically printed), and also introduced into the “station of the senior verifier” scheme. It was the time when those specialists worked, to whom the usual verifier had to “walk with his feet” earlier. Now, in the event of problems, the ordinary verifier simply redirected the task to the elder and the “pipeline” did not slow down.

2004. Suddenly it became clear that a huge number of empty documents were being printed in vain, because 50-80% of students do not even begin to fill out the form with the questions of group “C”. But there is a problem: according to the rules, a blank page is necessarily crossed out with the letter “Z” at the end of the exam, so that it is impossible to finish writing something later. Therefore, the system was taught to recognize pages that were crossed out in this way, and they began to be defined as empty. The number of printed materials has decreased significantly.

It would seem that the forests are saved, the conveyor is adjusted - what more could you want?

By 2007, some negligence on the part of the verifiers began to be observed - they had participated in the project for more than a year, got used and relaxed. It became clear that it is necessary to stimulate people to return to a normal state - a state of extreme attentiveness, because the matter is extremely serious - literally the future of children is at stake. Then the system was added to the mechanism of statistical evaluation of the probability of error. If the field had a rare value, the form got to the station of additional verification control. The operator had to make sure that the verifiers correctly checked everything and release the peace card further or make “a-ta-ta” to the verifier and return the document to the verification step. The presence of such control returned the quality level of verification.
So, the document processing scheme has found its final form.


In 2008, the USE "covers" the whole country. In order to reach inaccessible areas, USE forms are sent in advance, and tasks are transmitted in encrypted form. An hour before the exam passwords are transmitted. After the exam, the completed forms are scanned on any scanner and sent to the regional information processing center. The process is designed so that the qualification of the operator can be low (“physical education teacher”), i.e. A person who is practically not familiar with a computer can cope with work.

Instruments

Such a project duration naturally affected the tools: they started with VB, MS SQL 2000, continued on C #, MS SQL 2008. C ++ was also used for complex mathematical operations. "OSes": Windows XP, 2000. Say "old"? In general, even now, when they are already so many years, you need to think very deeply before dragging everything under modern Windows 7.

Training

The training of technical specialists was held with the participation of ABBYY. From each region, 2 people came to the 3-day courses, where they learned how to set up the infrastructure, and also tried their strength as operators of each station to ask questions if they arise. These specialists left with the application distribution kit, auto-tuning scripts and step-by-step instructions for each workplace. Practice has shown that such a training format is quite effective, so it remains that way today.

Support

In technical support, in addition to standard means (e-mail, helpdesk), a closed forum was used in which employees participating in the organization of the EGE could consult with each other. The forum turned out to be very useful, since the overwhelming majority of questions appear, of course, during the exams, i.e. for 5 days, questions should have time and arise and be resolved.

What happened

According to our data, among those applications for appeal that were granted in 2006, 0.3% were satisfied “for technical reasons”. Is it a lot or a little? It says here that <2% of graduates are appealing, and approximately every third of them are satisfied, i.e. about 0.6% of the total. Those. It turns out that technical errors are detected in 0.0018% of the work.