Machine learning in an investment company: we classify calls to technical support
In theory, the use of machine learning (ML) helps to reduce human participation in processes and operations, reallocate resources and reduce costs. How does it work in a particular company and field of activity? As our experience shows, it works.
At a certain stage of development, we at VTB Capital faced with the urgent need to reduce the time for processing requests for technical support. After analyzing possible options, it was decided to apply ML technology to categorize calls from business users of Calypso, the company's key investment platform. Fast processing of such requests is essential for high quality IT services. To help in this task, we asked our key partners - the company EPAM .
So, inquiries to the support service are received by e-mail and transformed into tickets in Jira. Then, support specialists manually classify them, set priorities, add additional data (for example, from which department and location the request was received, to which functional block of the system it belongs) and designate the executors. Total used about 10 categories of requests. This, for example, may be a request to analyze any data and provide the author of the request with information, add a new user, etc. Moreover, actions can be both standard and non-standard, so it is very important to immediately determine the type of request and assign execution to the right specialist.
')
It is important to note: “VTB Capital” wanted not only to develop an applied technological solution, but also to evaluate the capabilities of various tools and technologies on the market. One task, two different approaches, two technological platforms and three and a half weeks: what was the result?
The basis for the development of the prototype was the approach proposed by the EPAM team, and historical data - about 10,000 tickets from Jira. The focus was on 3 mandatory fields, which each such ticket contains: Issue Type (type of problem), Summary ("header" of the letter or subject line) and Description (description). As part of the project, it was planned to solve the problem of analyzing text from the Summary and Description fields and automatically determine the type of query based on its results.
It is the features of the text in these two fields of the ticket that became the main technical difficulty in analyzing data and developing ML models. Thus, the Summary field may contain a fairly “clean” text, but including specific words and terms (for example, CWS reports not running). The Description field, on the contrary, is characterized by a “dirtier” text with an abundance of special characters, symbols, backslashes and remnants of non-text elements:
In addition, the text often combines several languages ​​(mainly, of course, Russian and English), there may be business terminology, runglish and programmer slang. And of course, since requests are often written in a hurry, in both cases typos and spelling errors are not excluded.
The technologies chosen by the EPAM team included Python 3.5 for prototype development, NLTK + Gensim + Re for text processing, Pandas + Sklearn for data analysis and model development, and Keras + Tensorflow as a deep learning framework and backend.
Taking into account the possible features of the initial data, three representations were built for extracting features from the Summary field: at the level of symbols, combinations of symbols and individual words. Each of the representations was used as an input to a recurrent neural network.
In turn, as a representation for the Description field, service character statistics were chosen (important for text processing using exclamation marks, slashes, etc.) and average values ​​of strings after filtering service characters and garbage (for compact saving the text structure) , as well as the presentation at the level of words after filtering stop words. Each representation acted as an entrance to the neural network: statistics in fully connected, line-by-line and word-level in recurrent.
In this scheme, a neural network consisting of a bidirectional GRU layer with a recurrent and regular dropout, pooling the hidden states of the recurrent network using the GlobalMaxPool1D layer and a fully connected (Dense) layer with a dropout was used as a recurrent network. For each of the inputs, its own “head” of the neural network was built, and then they were combined through concatenation and locked to the target variable.
To obtain the final result, the combined neural network returned the probabilities that a particular query belongs to each of the types. The data were divided into five blocks without intersections: the model was based on four of them and was tested on the fifth. Since each request can be assigned only one type of request, the rule for making a decision was simple - according to the maximum probability value.
The second prototype, for which the proposal prepared by the VTB Capital team, was taken as a base, is a Microsoft .NET Core application with Microsoft.ML libraries for implementing machine learning algorithms and the Atlassian.Net SDK for interacting with Jira via the REST API. The basis for the construction of ML-models are also historical data - 50,000 Jira-tickets. As in the first case, machine learning covered the Summary and Description fields. Both fields were also “cleared” before use. Greetings, signatures, history of correspondence and non-text elements (for example, images) were deleted from the user's letter. In addition, with the help of the Microsoft ML built-in functionality, stop words that have no meaning for processing and text analysis were cleaned out of the English text.
Averaged Perceptron (binary classification) was chosen as the machine learning algorithm, which is complemented by the One Versus All method to provide multi-class classification
No ML model can (perhaps, so far) ensure 100% accuracy of the result.
Prototype Algorithm №1 provides the share of correct classification (Accuracy), equal to 0.8003 of the total number of requests, or 80%. At the same time, the value of a similar metric in a situation where it is assumed that the correct answer will be chosen by a person from the two presented by the solution reaches 0.901, or 90%. Of course, there are cases where the developed solution works worse or cannot give the correct answer - as a rule, due to a very short set of words or the specificity of the information in the request itself. Plays a role and is still not a large amount of the data used in the learning process. According to preliminary estimates, an increase in the amount of information processed will make it possible to increase the classification accuracy by another 0.01–0.03 points.
The results of the best model in the metrics of accuracy (Precision) and completeness (Recall) are evaluated as follows:
If we evaluate the quality of the model as a whole for different types of queries using ROC-AUC curves, the results are as follows.
Requests for action (Action Request) and analysis of information (Analysis / Task Request)
Requests for Business Data Request and Change Request
Development Request and Inquiry Request
Requests to create a new object (New Object Request) and add a new user (New User Request)
Production Request and Request Related to UAT / DEV Support (UAT / Dev Support Request)
Examples of correct and erroneous classification for some types of queries are given below:
Request for Information (Inquiry Request)
Change Request
Correct classification
Misclassification
Action request
Correct classification
Misclassification
Production Issue
Correct Classification
Misclassification
The second prototype showed good results: ML approximately in 75% of cases correctly determines the type of query (Accuracy metric). The opportunity to improve the indicator is connected with the improvement of the quality of the initial data, in particular, the elimination of cases when the same requests were assigned to different types.
Each of the implemented prototypes showed its effectiveness, and now a combination of the two developed prototypes has been launched into pilot production at VTB Capital. A small experiment with ML in less than a month and with minimal cost gave the company the opportunity to get acquainted with the machine learning toolkit and solve an important applied task of classifying user requests.
The experience gained by the developers of EPAM and VTB Capital - in addition to applying for the further development of the implemented algorithms for processing user requests - can be reused in solving a variety of tasks related to streaming information processing. Movement in small iterations and coverage of one process after another allows gradually mastering and combining various tools and technologies, choosing well-shown options and discarding less efficient ones. This is interesting for the IT team and at the same time helps to get results that are important for management and business.
At a certain stage of development, we at VTB Capital faced with the urgent need to reduce the time for processing requests for technical support. After analyzing possible options, it was decided to apply ML technology to categorize calls from business users of Calypso, the company's key investment platform. Fast processing of such requests is essential for high quality IT services. To help in this task, we asked our key partners - the company EPAM .
So, inquiries to the support service are received by e-mail and transformed into tickets in Jira. Then, support specialists manually classify them, set priorities, add additional data (for example, from which department and location the request was received, to which functional block of the system it belongs) and designate the executors. Total used about 10 categories of requests. This, for example, may be a request to analyze any data and provide the author of the request with information, add a new user, etc. Moreover, actions can be both standard and non-standard, so it is very important to immediately determine the type of request and assign execution to the right specialist.
')
It is important to note: “VTB Capital” wanted not only to develop an applied technological solution, but also to evaluate the capabilities of various tools and technologies on the market. One task, two different approaches, two technological platforms and three and a half weeks: what was the result?
Prototype # 1: Technologies and Models
The basis for the development of the prototype was the approach proposed by the EPAM team, and historical data - about 10,000 tickets from Jira. The focus was on 3 mandatory fields, which each such ticket contains: Issue Type (type of problem), Summary ("header" of the letter or subject line) and Description (description). As part of the project, it was planned to solve the problem of analyzing text from the Summary and Description fields and automatically determine the type of query based on its results.
It is the features of the text in these two fields of the ticket that became the main technical difficulty in analyzing data and developing ML models. Thus, the Summary field may contain a fairly “clean” text, but including specific words and terms (for example, CWS reports not running). The Description field, on the contrary, is characterized by a “dirtier” text with an abundance of special characters, symbols, backslashes and remnants of non-text elements:
Dera colleagues,
FX_Opt_delta_all and FX_Opt_delta_cash risk measures?
! 01D39C59.62374C90_image001.png! )
In addition, the text often combines several languages ​​(mainly, of course, Russian and English), there may be business terminology, runglish and programmer slang. And of course, since requests are often written in a hurry, in both cases typos and spelling errors are not excluded.
The technologies chosen by the EPAM team included Python 3.5 for prototype development, NLTK + Gensim + Re for text processing, Pandas + Sklearn for data analysis and model development, and Keras + Tensorflow as a deep learning framework and backend.
Taking into account the possible features of the initial data, three representations were built for extracting features from the Summary field: at the level of symbols, combinations of symbols and individual words. Each of the representations was used as an input to a recurrent neural network.
In turn, as a representation for the Description field, service character statistics were chosen (important for text processing using exclamation marks, slashes, etc.) and average values ​​of strings after filtering service characters and garbage (for compact saving the text structure) , as well as the presentation at the level of words after filtering stop words. Each representation acted as an entrance to the neural network: statistics in fully connected, line-by-line and word-level in recurrent.
In this scheme, a neural network consisting of a bidirectional GRU layer with a recurrent and regular dropout, pooling the hidden states of the recurrent network using the GlobalMaxPool1D layer and a fully connected (Dense) layer with a dropout was used as a recurrent network. For each of the inputs, its own “head” of the neural network was built, and then they were combined through concatenation and locked to the target variable.
To obtain the final result, the combined neural network returned the probabilities that a particular query belongs to each of the types. The data were divided into five blocks without intersections: the model was based on four of them and was tested on the fifth. Since each request can be assigned only one type of request, the rule for making a decision was simple - according to the maximum probability value.
Prototype number 2: algorithms and principles of operation
The second prototype, for which the proposal prepared by the VTB Capital team, was taken as a base, is a Microsoft .NET Core application with Microsoft.ML libraries for implementing machine learning algorithms and the Atlassian.Net SDK for interacting with Jira via the REST API. The basis for the construction of ML-models are also historical data - 50,000 Jira-tickets. As in the first case, machine learning covered the Summary and Description fields. Both fields were also “cleared” before use. Greetings, signatures, history of correspondence and non-text elements (for example, images) were deleted from the user's letter. In addition, with the help of the Microsoft ML built-in functionality, stop words that have no meaning for processing and text analysis were cleaned out of the English text.
Averaged Perceptron (binary classification) was chosen as the machine learning algorithm, which is complemented by the One Versus All method to provide multi-class classification
Evaluation of results
No ML model can (perhaps, so far) ensure 100% accuracy of the result.
Prototype Algorithm №1 provides the share of correct classification (Accuracy), equal to 0.8003 of the total number of requests, or 80%. At the same time, the value of a similar metric in a situation where it is assumed that the correct answer will be chosen by a person from the two presented by the solution reaches 0.901, or 90%. Of course, there are cases where the developed solution works worse or cannot give the correct answer - as a rule, due to a very short set of words or the specificity of the information in the request itself. Plays a role and is still not a large amount of the data used in the learning process. According to preliminary estimates, an increase in the amount of information processed will make it possible to increase the classification accuracy by another 0.01–0.03 points.
The results of the best model in the metrics of accuracy (Precision) and completeness (Recall) are evaluated as follows:
If we evaluate the quality of the model as a whole for different types of queries using ROC-AUC curves, the results are as follows.
Requests for action (Action Request) and analysis of information (Analysis / Task Request)
Requests for Business Data Request and Change Request
Development Request and Inquiry Request
Requests to create a new object (New Object Request) and add a new user (New User Request)
Production Request and Request Related to UAT / DEV Support (UAT / Dev Support Request)
Examples of correct and erroneous classification for some types of queries are given below:
Request for Information (Inquiry Request)
Change Request
Correct classification
Misclassification
Action request
Correct classification
Misclassification
Production Issue
Correct Classification
Misclassification
The second prototype showed good results: ML approximately in 75% of cases correctly determines the type of query (Accuracy metric). The opportunity to improve the indicator is connected with the improvement of the quality of the initial data, in particular, the elimination of cases when the same requests were assigned to different types.
Summing up
Each of the implemented prototypes showed its effectiveness, and now a combination of the two developed prototypes has been launched into pilot production at VTB Capital. A small experiment with ML in less than a month and with minimal cost gave the company the opportunity to get acquainted with the machine learning toolkit and solve an important applied task of classifying user requests.
The experience gained by the developers of EPAM and VTB Capital - in addition to applying for the further development of the implemented algorithms for processing user requests - can be reused in solving a variety of tasks related to streaming information processing. Movement in small iterations and coverage of one process after another allows gradually mastering and combining various tools and technologies, choosing well-shown options and discarding less efficient ones. This is interesting for the IT team and at the same time helps to get results that are important for management and business.
Source: https://habr.com/ru/post/455740/
All Articles