State tests through the eyes of a programmer

Introduction

I was fortunate enough to recently visit the State Testing of Products in the city of Kolomna. We worked on it for a year and a half, and then came the most crucial moment of putting it to the customer.

This short article describes my initial understanding of the trials and how they actually went. It will not contain references to GOSTs and other documents regulating such procedures.



The tests will be described by the eyes of a simple programmer who developed a piece of software for the simulator. The product of military subjects (simulator), but not a secret, so the description will be present military. The simulator is a hardware and software, has a full-sized frame (4x3x2 m) filled with various devices and devices, it implements all the necessary for the operation of the device in the gland.

The first day

We arrived in Kolomna, after three hours of waiting at the checkpoint of the military unit, we were given passes, we handed over several instruments for installation and went to the hotel to settle without heating.



Settled, left things and went to the part to our already mounted simulator. Also they don’t drown them - they felt a little sad ... Our task (two programmers and an electronics engineer) on this trip was to set up open source software (Special Software) and various iron devices.

')

Until the end of the day we worked hard, set up everything that could be. Several times spent a full cycle of work. It turned out that the backup devices do not function correctly, and with them some of the important devices, without which it is impossible to fully work. Commission members have not yet been. For your information, the defective work of the product is a 100% failure test.

Second day

All the second day we showed the commissions a simulator with a software imitation of non-working units. They were more interested in the picture as a whole, whether it is possible to effectively learn on the simulator, how it is implemented, and how it all works. A part of the commission checked documents, a part investigated a product. There were several frivolous comments and suggestions that were recorded in the journal and noted.



The theme of the day was the cable, the usual Ethernet cable. The fact is that in the process of developing the simulator, we, in accordance with the TOR, realized mutual work with another simulator (by the way, it did not provide for interaction with us at all). Our management as agreed and we realized the interaction, of course, over the network. The correctness of the implemented interaction was tested on preliminary tests at the site of that simulator.



But we did not include a cable for the connection with that simulator and did not describe the setup procedure. That was a dispute with the commission. This dispute overshadowed even the importance of the TK item that the interaction of simulators should be carried out at the GI (State Testing).



They have GOSTs, and we have common sense. That simulator is more stationary than ours, with its servers, database and network. He, as I understand it, there are only two copies. At the preliminary tests, we came to them, connected to their local network. They told us all the necessary data about the database and the network. Naturally, only having two specialists from two sides can do the docking.



The hotel gave heating, in the room with a simulator bought a heater. Life has become more fun.

Test plan

The result of the work of the commission at the end of the second day was a detailed plan for the IG. Programmers needed to solve the problem with improperly functioning devices and train one leader, 8 lieutenants and 18 soldiers to work on the simulator and, accordingly, on the combat vehicle.

According to the plan, two equally trained groups of trainees were supposed to undergo training on a simulator and on a combat vehicle. And at the end of the test, the commission evaluated the knowledge gained. In full, this did not happen. Either those who were engaged in the combat vehicle were already trained, or the quantitative composition was not equal, but at the end the assessment was set subjectively by the training supervisor. The leader was a military officer who knew his business perfectly well, so no one doubted his assessment.



The active members of the commission all quickly took pictures of their soap dishes and left. They had the next day a report to the Minister of Defense.

The main thing is the process

For several days we worked with the head of training from morning to late evening. They showed him the capabilities of the product, wrote down the identified comments.



In parallel with the academic work, we were looking for a technical solution for non-working devices. It was very important to correct all the problems before the commencement of training of lieutenants and soldiers, so that for them there would be no difference between a combat vehicle and a simulator.



Commission members came and went, read the documents, watched us work with the simulator, asked questions. All without fuss, thoroughly and clearly, as is customary in the military.



People in the commission - the military, from various institutions and departments. There were people whose goal was to fill up the GI. And the problem was in one document. They offered our company to create this document for a certain amount, but ours found some company that performed work at a higher level for a much smaller amount. It’s natural not to complain about the document, so they with particular diligence searched for and revealed all our mistakes in other documents and products.



They fed everyone in the soldiers' canteen. There is now outsourcing - in the dining room run by civilians. They feed for slaughter, after lunch the soldiers sleep for 1 hour. All the same age, but for three weeks I was not persuaded to serve.

Job

We found a solution to our problems with non-working units. Coordinated all changes with the chief designer of the product and set to work. Worked in the part and in the hotel. It was possible to exclude non-critical blocks from the system (their faults were recorded as comments). Everything else began to work stably, fulfilled all the functions of the simulator, the clouds dispersed over us.

And again the process

A week and a half after the test began, the lieutenants came to study. They should work on the product as billing commanders. Prior to that, they were trained only by posters and did not work on a combat model. The training took place as intended. The lieutenants studied the product, pressed the buttons and twisted the handle.



Then a new technical specialist metrologist came to us. In addition to metrology, he checked the reliability of the simulator. Not paying attention to the simulator, he was digging through documents all day. And in the end, he said that everything is bad with metrology, and that he will make his decision in a week. It even seemed strange to me - a metrologist and without a caliper. A week later, he brought a positive conclusion.



Then, in the middle of the day, a colonel came to us, who fought in the military complex, and knows him firsthand. No documents immediately got into the simulator. There were two lieutenants who were trained only two days. He is not small, not very small, the voice is like that of a company of soldiers, says to this lieutenant: “we are working!”. The calculation strongly pressed into the chair, they are still just learning, do not know how to work on the product. Then the colonel began to say what buttons to press, what to watch. On this day, we received the most valuable and constructive comments and additions for the entire period of GI.



Another programmer came. I checked various keyboard shortcuts, asked about information security, wondered if we checked checksums when loading our libraries, how we clean the RAM after the work was completed. To this he was told that we do not have certified SZI (information security tools) for TK and the simulator is not a combat product. In conclusion, he wrote to implement the integrity check of all downloaded free software, register entry and exit from the system, block the possibility of unauthorized access (unauthorized access) to the OS interface, and also limit the minimum password length for the input to six characters.



Evaluated our work and ergonomist. A young guy arrived, looked at our software. Asked how it interacts with each other, asked to show some modes of operation and sat down to write a conclusion. For all the work he had had a few hours, after which he quickly retreated. Do you know GOST software ergonomics?

Our problems

For three weeks I received a lot of lessons, understood how to pass tests, saw the whole process with my own eyes and formed my opinion.



All revealed remarks could be eliminated before the GI. But they were either ignored, or they lacked time and resources. In the process of creating the product, the customer did not take a serious part (only the actual paper work according to the State Standards), therefore, on the face of some commission members, there was some surprise as to why certain things were done the way they were done.



The customer did not see the product at work before the preliminary tests, so on the simulator, for example, it is inconvenient to train 10 people at a time. Some TK items were written as for a combat sample (maybe taken from its documentation) and were not suitable for our simulator in any way. Later they were excluded from the additional TZ.



It can be concluded: all the problems that have arisen in the process of GI are associated with insufficient interaction between the product developer and the customer. The customer, unfortunately, was not particularly eager to work together.

About the work of the commission

The commission worked professionally, revealed all possible comments. But such disputes and disagreements about the words and phrases in the documentation, I have not heard anywhere else. They diligently read the documentation and not so much attention was paid to the product itself. As I understand, their statements are documents, so there was so much attention to them. In some matters they insisted on their own, but in terms of technical solutions they listened to specialists.



I understand that the commissions are different. It all depends on what solution everyone needs. I will not raise questions of bribes, but I think it has not affected us. The commission worked hard. The teaching officer at the meeting of the commission said that the simulator was necessary for them “like air”, and after signing the acts of the State Institute, the commission members said that they liked the simulator.

Conclusion

Participation in conducting GI gives invaluable experience. Tests reveal all the weaknesses of the product. People are beginning to understand better what products to create, what further fate awaits them. This is direct communication with users of the system, understanding how their desires are satisfied by the product.



Someone will seem my narrative naive, someone will remember his old project, and someone will start to work better, but I hope this experience will be useful not only for me.