USE inside (through the eyes of the admin)

Hot topic, a lot of topics and Habré ( here and here )
I am absolutely not going to dwell on the legality of the experiment, the content of the CMM (control measurement materials) and so on. I will try to concentrate directly on the technological aspect of preparing and conducting the exam.
Why I can be trusted - experience of 2004–2007 at the RCSEI of the Moscow Region (system administration, data collection, coordination with developers and regions, conducting trainings, consultations, etc.). All this time we have been working on the CROC technology (see below), in 2009 the technology changed in the form of an order for DH. So, in order

Technology

Until recently, there were two technologies for organizing and conducting the Unified State Examination: the Federal Testing Center (hereinafter referred to as CT, www.rustest.ru ) and CROC (CROC CJSC www.croc.ru , then simply CROC).
Technologies differ fundamentally. I will dwell on the CROC technology in as much detail as possible and at the end I will point out the points why it was better than DH (my personal professional opinion, colleagues and management).
CROC - MS SQL 2000 as a database server and WIN 2000 (2003) as an application server. Servers are distributed on different machines, on WIN 2000 there was an additional AD for the internal local network.
On the application server there is a server part of the software package (hereinafter AIS), on client machines, respectively, the client part, the user logs in via AD to the domain, work through the browser interface (IE6 is required). According to the rules of security, no other software can be put on a client (I put another office, rar and radmin are password-protected and only for the group of administrators). All software is 100% licensed (Windows, offices, server programs and the like).
As an addition to the comments on the volumes on the disks: 7 gigs uncompressed dump database (full) and 90 gigs archive of pictures of all exams. The total number of people \ exams about 120,000

The entire local network with a database and any exam data is physically disconnected from the Internet (in order to avoid data leakage).
')
At the AIS level, there is a distribution of user rights for access to the system (it is configured very flexibly, the necessary privileges for a group of users are checked by checkboxes). Accordingly, any client station can be authorized via AD with the necessary rights and work as required.

The network structure for the exam for 50,000 people (we had this maximum load, the data needed to be processed and passed to the test for a maximum of 4 days after the exam) consisted of:
1 - 3 servers (databases, AIS server, file archive of scanned images)
2 - 12-17 scanning stations
3 - 20-25 verification stations
2 - 1-2 admin stations for continuous monitoring of bottlenecks.

During the rest of the year, the client scanning / verification stations operate normally for other users. At the time of the exams, an image with a ready-made system was rolled in and withdrawn for our use.
Scan stations - high-speed scanners DS-300 from LOMO, PANASONIC type KV-S3105C and low-power Fujitsu. The DSs worked via the network, KV and Fujitsu via Skype. The speed of the first two cars is impressive (pervosti) - 90-120 sheets per minute in black and white format of about 200 dpi. When scanning the form is cut into pieces (by reference marks), instantly passes recognition and files are transferred to the file server. All software is developed by CROC (did not intervene in their subcontracts).

Data collection.

Exam preparation begins in October (approximately) and lasts the entire year. It includes the following steps:
1 - reconciliation of information on educational institutions, administrative directories, clarification of those responsible for conducting examinations (usually one person from the Moscow region, but if the area is large and / or people are a little computer savvy - two, one techie, one manager). Municipal coordinators are generally a separate topic, they would put a monument to them. Usually these are women from the governing bodies of the entities.

2 - collection of data on graduates of the OS (the most complete card, name, passport, planned exams). The collection is carried out by specialized client software (program without installation) with a very intuitive interface. The capabilities of the software are much broader - in addition to basic data collection, this is the collection of analytical data for the stat. Analysis. For example, data on a teacher (experience, specialization, date of courses), annual grades on the subject, a textbook, and much more. These data are filled throughout the year (for a very large amount of information must be collected and entered).

3 - verification of passport data of graduates (on paper media signed by the head of the educational institution)

4 - with the publication of the exact dates of the examinations, the final collection of data on registration for the exams and subsequent reconciliation (on paper, signed by the head of the OU and the student's signature alongside his own passport data and selected exams). The period of this stage is very compressed, because there is very little time from the order with the dates of examinations and the deadline for submitting data to higher authorities. Why so strictly - to print the required number of KIM

With reconciliations in general, very interesting things were happening. Pupils throughout the year should see lists with their data at least 2-3 times and, if necessary, correct them (full name, passport). Not only did each adjustment bring about 5–10% of changes to the database (this is all done by the operators of the center), and on the exams there were many forms with correction. Children are completely inattentive to their data, and because of this, and they may have problems with obtaining results (correction of data passes through CTT and it is very long and difficult for us).

So, all the data is collected, verified, go to the exam.

Exam

A few days before the exam, we get the ordered amount of CMM from the FTS (with a certain limit in case of marriage or under-complement of the audience). Everything is very serious - sealed, taken by inventory by the head of the center.

Printing Exam Materials
The center prints kits for each audience and PES (point of the exam). They printed on HP in wild streaming mode, from the second year they got the hang of simultaneously printing from two clients to two printers at once. The printed set is removed from the printer, signed by a statement and sealed in the so-called sekyurpak. After the pack, they are packaged for each area \ and stored in a special room (with a combination lock). Cartridges change every 3-4 hours, dust is a pillar, lungs slowly die from 3-4 simultaneously working HP-shnik.

A day before the exam, the municipal coordinators arrive - according to the inventory, the packages are issued, the number and general data are checked. Hours 18-19 last package issued, on brandy and sleep, for the next 3-4 days like a bad dream.

Exam day

From 8 am (opening time for packages in the field) we were in the center with the head - the answers to all possible questions during the exam and the nervous expectation of marriage at the CIM (it was such a marriage at the print stage, the child has to make a change of the option), errors with registration of children for the exam (several times the municipal coordinators forgot to enter the student for the exam, but he calmly did not sign on the reconciliation statements). Some calm after 10 am (exam went).

Approximately on the 14th day, the delivery of materials from PES begins. Acceptance is as follows:
1 - reconciliation of packages by the head of the center

Reception and scanning after the exam
2 - the municipal coordinator and the operator of the scanning station are “secluded” in the hall and are scanned per packet, after processing each package, the audience verifies the data (machine processing) with the statements, no form should be lost. At this stage, several times there were cases that the form was forgotten in the PES. In this case, the reception of the area is suspended until clarification. At the client level, there is a strict calculation of the number of forms issued and scanned for each audience. Substitution is not possible. Forms registered and with a unique code, when scanning is a reconciliation of all fields.

There was a curious case when a student went into the toilet with an unresolved part C and washed the blank into the toilet (didn’t flush it well). When we received it, we found it, sent a messenger to a point, he fished out a form in the toilet and brought our nose, we had to process it.

3 - I, as an administrator, in the intervals between the solution of current issues, check the processed audience, if the number of issued / accepted forms agreed - I transfer the audience for verification.

Documents from a large area, Odintsovsky district (colleagues recalled), waiting for the queue to scan, in this pack of forms 2500 participants
4 - after this, the forms with the works are verified. Verification is a delicate matter - each client machine with a verification base is self-learning. If the operator confirms the value of one field 2 times - it enters the dictionary and with a probability of 99% will be automatically worked. Lesson is very tedious, because usually ends well after scanning (2-6 o'clock at night). It is at this stage that the processing of "random" errors occurs. If the child put part A of the answer in the 5th cell (it is reserved), the extra \ service fields are filled in - all this gets to the verifier.

5 - administrator - conducts general monitoring of the system, corrects any errors in the statements and so on. Cases of the “student did not show up” tag and the completed form are monitored (the system shows such cases), otherwise the work will not be counted. Lost forms - the system could not recognize the scanned number and did not attach the form to the audience.

6 - after verification is complete, you can print packages for the work of experts. They start at 10 am the next day of the exam. Therefore, at 8–9 pm, at least 2 stations of printing works are launched. C. Extradition is strictly random - blank forms, the expert does not know whom he is checking, each work is given to two experts, if the difference in their estimates is higher than the set mark, the work is shown as requiring additional verification and printed out for the "third expert." The third expert cannot be one of the two who have already checked; it is manually assigned from the instructions of the chairman of the inspection commission.

Check part C lasts 3-4 days. Once all the data has been processed and no form has been lost, a CSV is formed with the results. This file is encrypted on the server (Crypto Pro, the keys are stored on a separate media in a safe and issued for signature) and are sent via an encrypted channel through a special program to the FTT for evaluation.

I will not tell about scaling, who will be interested in finding the calculations. A rather complex mathematical model is built in the FTsT when obtaining results from all over Russia.

After 3-5 days, the processed results with the estimates come from the FTsT to the CSV, are loaded into the database and the protocols are printed out (each signed by the seal of the regional minister of education) and issued to each district. Recent years have begun to allow the publication of data through websites, this year we on the Mosoblast website have done the same functionality (which caused an increase in attendance and laid the server of the hoster)).
In principle, in general, everything. In real-time mode, when exams take place with an interval of 3-5 days and / or 2-3 exams in one day, this results in 2-3 weeks 100% of your presence at work seven days a week, 25 hours a day. But when you know that there are 50,000 children who pass the exam behind us and happy parents are worth it.

Appeals

I don’t remember appeals to conduct an exam (although it would sometimes be worth it, but children are tired and do not immediately understand the situation). According to the results - about 500 units per 50,000.

A special commission is going to appeal, printed forms are recognized, copies of originals are checked, machine recognition errors are checked and experts on objects with the child and the parent go through all the tasks step by step. Appeals are divided into categories: rejected and reviewed. 99% of the appeals reviewed pass with an increase in the score (rarely justified, usually flowers, candy, tears). A couple of times especially "brazen" lowered the score. About 40% remain with the same result, that is, the appeal is rejected. Separately, I want to say excellent students - if you don’t have 1-2 points in a higher education institution, be sure to file an appeal, such cases (there are few of them) are always considered positively.

Bottlenecks of the whole procedure:

1 - banal inattention at the level of districts and schools. Coordinators lose / forget children, children do not pay attention to their passport details and selected exam dates. It creates a lot of problems for us when reconciling data and children in case of complications.

2 - low computer literacy coordinators. In no case do I want to blame them, but the fact remains. Not every district can find a specialist to solve technical issues (usually consist in working with e-mail, excel, word, software for collecting information). From there, and my anecdote about the "red and blue Internet."

3 - the absence of the Internet in education authorities. Many work through a modem, send data to us from home computers. Pro schools just keep quiet - rarely rarely there is a network, data exchange through floppy disks is usually (!!!!)

4 - complete chaos in the relationship with FCT. A couple of examples. They write and approve a database format with representatives of CROC and the Ministry. In fact, we find a discrepancy - we approved one, unloaded the data into CSV, and they do not load into the FT database - the order of the fields has changed, the format of the data, and so on. There is swearing, Krok convulsively writes a hack for his base, in order to fit the allegedly approved format to the new requirements (not expressed anywhere else). Further, the results are output, for some reason our file is not accepted by them (the format is erroneous or otherwise) - the silence is complete on their part, only by a call (oh, how difficult it is to call them) it turns out that there is a cant. Again this time, the nerves of us and the children / parents. When communicating creates the impression of a classic "sovok" shop from jokes. Very, very annoying such facts, it seems like we are doing a common thing, but here are such tricks.

5 - even more chaos at a higher level. Discrepancy of orders of reality, one ruling contradicts the other and the like. Eternal glory to our boss, who did not regret his gray hair, would settle such questions as quickly as possible (although not at all our level)

6 - and, perhaps most importantly - INSIGHTNESS and UNKNOWNESS of the rules when conducting the exam in paragraph. We conduct year-round training for specialists in each PES, and we are ready to constantly answer any question. After it turns out that they do not know which side to open the package and issue a list.

Positive aspects of technology

1 - a very, very benevolent software (not without sins, but compared to FTsT software - heaven on earth). Implementation specialists answered almost any question. A lot of own developments (sinful, wrote a couple of pribluds on php for speeding up the same type of procedures). A single complex, flexible and extensible, developed by one office. The software of the same CT was assembled from different pieces of different developers, based on ABBYY. The work is unintuitive, inconvenient, and in my opinion just a curve.

2 - convenience of the final configuration of the system in the center, technology debugged literally to the thread. When the center was transferred to the FTsT, the labor costs during the conduction increased significantly (as well as the level of risks)

3 - high data security. Not a single drop of information was lost from the center and from the PES. In addition to the administrator, no one physically can make changes to the database, at the client level everything is monitored and saved. Only the admin has the opportunity to make "byaku". A couple of times the truth came to me personally with a quiet money offer :) but without success. As far as I know, at the district level, similar cases did not pass either.

4 - almost 100% convenience for the end user - student. A personalized form, indicating the school, TES, only signing and solving problems (for the CFT, you must also enter your full name - an extra reason for error)

5 - great opportunities for statistical analysis not only of the unified state examination, but also of the entire educational system. The data array allowed it (on condition of 100% filling in the field). We used SQL Analyze Server with web components (analytics module - development of the same CRIC). Well, directly from the database, requests of particularly interesting indicators. You can compare almost anything - an estimate of the duration of the study of the subject in conjunction with the textbook, the conformity of the evaluation of the Unified State Examination and ready-made indicators on the growth curve of the total score from grade 5. Many wonderful things like this have never been found before. In practice, it was covered with a copper basin, although for statistical work there was a separate center attached to the ministry.

Results (by experience) - the percentage of machine processing errors is 0.1%, the percentage of verifier errors is 2 cases per 50,000 person / exams. That is, from our side everything is clean.

I repeat - everything that has been written is my personal opinion on the experience of working with this technology and the experience of the solidarity colleagues of our center during the period of work until 2007. And sorry for the language of presentation - the Chukchi is not a writer, the Chukchi reader.

UPD information from the leading developer of the system

1 - what it says: in C # (ASP.Net) - app. server part, T-SQL - client part, Active-X ( ++, VB6, C #) + VBScript - client.
2 - how is the delivery of options for the audience: a rather complicated algorithm that evenly spreads the KIM options for the PES audiences, trying not to give the same options to the neighbors in the "cross" (left-right-up-down)

If you are interested in separate, undisclosed, aspects of technology - write, I will try to add. Well, if there is no account on Habré - write to the comments in the blog
Workplace admin))