The use of machine learning methods in the task of assessing the actions of a pilot at the landing stage

INTRODUCTION

Accident statistics show that the landing is still the most dangerous part of the flight. And in most cases, accidents occur due to the fault of the personnel (Fig. 1). Thus, the task of evaluating the actions of the pilot to control the aircraft during the landing stage is relevant for both civil and military aviation, since it allows for improved flight safety.


Fig. 1. Diagrams of the distribution of accidents by stages of flight and by types of violations of personnel

The analysis of existing methods for assessing the quality of piloting on the basis of information from on-board registration devices (BUR) showed their insufficient efficiency.
In accordance with the Course of Combat Training (UBC), the assessment of the quality of the landing maneuver performance is determined on the basis of the data of on-board and ground-based objective monitoring tools, as well as the observations of the instructor and the flight management team (Fig. 2).
')

Fig. 2. The structure of the assessment

According to the BSC, the piloting technique is evaluated at the landing stage based on the on-board SOK materials by analyzing the flight parameters in the 4 control sections of the glide path (Fig. 3): entering the glide path, passing the long-distance radio beacon (DPRM), passing the near-driving radio beacon (BPRM), touch the runway.


Fig. 3. Reduction of aircraft on glide path

At the same time, the flight data are not fully used: out of the 14 regulatory parameters established by the BCP for assessing the quality of landing performance, only 5 parameters are determined based on the materials of the drills, which is only 35.7%. The remaining parameters are determined on the basis of observations of the instructor and persons of the flight management group, which makes a significant share of subjectivity in the final assessment.
Thus, there is a need to supplement the PCU methodology with parameters that take into account the nature of aircraft piloting during the entire descent phase of the glide slope and registered by regular BUR.

MODIFICATION OF THE EXISTING METHOD FOR EVALUATING THE ACTIONS OF THE PILOT AT THE STAGE OF THE LANDING

In the works [1-3], it was proved that the structure of the movement of the aircraft control stick on landing is a characteristic of the piloting quality (it shows how confidently the pilot controls the plane) . Based on these works, it can be concluded that, as an informative attribute, it is advisable to use the parameter "Pitch deflection angle which is registered on all modern aircraft by a regular onboard registration device (Fig. 4). This method allows you to evaluate the actions of the pilot throughout the landing stage, in addition there is no subjectivity. A typical aircraft control system in the longitudinal channel is shown in Figure 5.


Fig. 4. Graphs of the change of the parameter “Deviation of the engagement in pitch” at the landing stage for pilots of various class qualifications; a) class 1 pilot; b) a pilot without a class; c) transfer of control


Fig. 5. The control system of the aircraft in the longitudinal channel (MU-615A - potentiometric sensor of angular movements, RP - steering gear)

The evaluation method described above is proposed to be integrated into the well-known methodology for estimating piloting techniques according to CBP. Thus, the percentage of objectivity of the final assessment will be increased by introducing into the final assessment an additional objective indicator characterizing the actions of the pilot at the entire stage of descending along the glide path.
On the basis of [4, 5], it is proposed to use a signal spectrum to estimate the fit , and to build a model of dependence between a given spectrum and a pilot’s level of training (or an assessment made by the flight commander) , characterized by his classroom qualifications, use methods of machine learning with a teacher.
As input features in the implementation of the proposed method, samples of the normalized smoothed spectrum were used. investigated discrete signal , and as the output attribute - the class number of the pilot.
Input feature vector was formed using the following transformations:

1. Calculate the spectrum F of a discrete signal N lengths
   ;
   
2. Signal amplitudes whose frequencies are less than 0.5 Hz are zeroed out to eliminate the information component of the signal associated with the natural frequency of the aircraft in terms of angle of attack;
3. We bring the spectrum to the form from 0 to 2.5 Hz and change the size of the obtained feature vector to a fixed value using bicubic interpolation (we obtained the vector );
4. Smooth the spectrum using the sliding window method to obtain more homogeneous data and normalize in amplitude:
   ,
   ,
   where m is the size of the window (in the implementation of m = 10 ), and K is a constant number that exceeds all values (in the implementation of K = 250 ).
   

Building a model of dependence between the signal spectrum and the class number of the pilot was performed using two machine learning algorithms with a teacher: the support vector machine (SVM) method with a radial basis function as the core and the tree gradient boosting method (GBT).
In the implementation, GBT was used with the following parameters:

• type of loss function - abnormal loss;
• the number of iterations of the boosting - 3000;
• the regularization parameter is 0.0008;
• subsample batch - 0.0001;
• maximum depth of decision trees - 1;
• using surrogates.

REFINEMENT OF PREDICTED LETTER CLASS NUMBER

To identify the weak points of the algorithms, tests were carried out and error distribution diagrams were calculated. One of these diagrams is shown in Figure 6.
The description of the base used for testing is given in the “Results and Discussion” section.


Fig. 6. Error distribution diagram for the GBT method

The first three columns of the diagram (Fig. 6) correspond to the cases of an incorrectly detected 1st class (including snipers), while the red column corresponds to the cases when the 1st class is defined as 2nd class, the green column corresponds to the cases when the 1st the class is defined as the 3rd class and the blue column corresponds to the cases when the 1st class is defined as without a class. Similarly, for the 2nd, 3rd and 4th triples of columns, which correspond to the 2nd, 3rd class and “zero” class (without class).
The analysis of the error distribution diagram showed that in most cases the algorithms are wrong for one class, therefore it is reasonable to carry out a separate analysis for each pair of the next classes in the future. For example, specify the predicted class number of the pilot already in the binary classifier.
The implementation used the following binary classifiers:

• if the predicted class is “3rd class”, then the SVM method is used to classify pilots of the 2nd and 3rd classes;
• if the predicted class is “without class”, then the GBT method is used to classify pilots of 3rd class and without class.

RESULTS AND DISCUSSION

According to the standard estimation scheme of the generalizing ability of the classification algorithm, all available data are divided into training and test samples. However, due to the large number of input features, as many examples as possible in the training set are required, otherwise the classification algorithm may not have enough information to build a dependency model. Also requires a large number of examples in the test sample to assess the quality of the classification with high accuracy.
In our case, the amount of available data is limited, so the cross-validation method is used to estimate the generalizing ability of the classification algorithm.
All available data were divided into 5 non-intersecting blocks, each of which contains 48 samples of 12 samples for each pilot's class: no class, 3 class, 2 class and 1 class (including pilots snipers). The data was obtained from the onboard devices for registration of maneuverable aircraft (the name of the aircraft I can not say) . The results are presented in the table below.

Test results of machine learning algorithms

Type of algorithm	Error E , units
	overall accuracy	1st class and pilots snipers	2 class	3 class	without class
SVM method	0.5526	0.5667	0.6167	0.3167	0.7152
GBT method	0.5566	0.6167	0.3667	0.5167	0.7318

The final assessment of the effectiveness of the algorithm:

calculated by the following formula:

Where - the number of test cases in which an error was made, - the number of test cases in the test sample.
Analysis of the results showed that the GBT method showed the best overall accuracy. In addition, the refinement of the predicted class number of a pilot in binary classifiers increased the overall accuracy of the SVM method by 2%, and that of the GBT method by 3%.

CONCLUSION

The use of machine learning methods made it possible to experimentally confirm the correlation of the parameter “Pitch deflection angle of the aircraft control stick” at the stage of reducing the plane on the glide path with the pilot's level of training, expressed in points in accordance with his classiness. The obtained accuracy of the support vector method and the gradient boosting of trees is 55%. Thus, the signal spectrum “Pitch deflection angle of the aircraft control stick” can be used as an additional parameter to evaluate the actions of the pilot during the landing stage, and building a system that assesses this spectrum can be performed using machine learning algorithms with a teacher: support vectors and gradient boosting of trees.




UPD: link to the project and database .