Mail.Ru Mail Support: Past, Present, Future
In this post we would like to talk about how the support service of the most well-known and heavily loaded service of our group of companies Mail.Ru works and how it looks today after the recent redesign of the structure and efficiency.
Now the support service employs more than 50 employees. The support works 7 days a week on a shift schedule, on weekdays it is 2 shifts of 6 hours each (from 9 to 15 and from 15 to 21), on holidays and weekends - one 8-hour shift (this is due to the smaller flow of applications in weekends). This schedule is convenient for students and allows you to combine work with full-time education.
The schedule is flexible, it is made taking into account the possibilities and the desire to work of each employee :) Payment depends on the number of shifts. The logic is simple: all shifts above the minimum wage of the employee, are paid at an increased rate - this encourages people to work more than the minimum level.
')
We use the OTRS application processing system (Open-source Ticket Request System, http://www.otrs.org ), a popular open source project that runs on almost all platforms. OTRS is used in many large companies, for example, in Opera, MySQL AB, SuSE Linux AG, and it is also for us. Unfortunately, the out-of-box system (at the time of implementation it was version 2.2.6) was not suitable for working with large amounts of data (thousands of incoming emails per day and hundreds of thousands of tickets in the database, parallel work of tens of people), therefore, within our Support applies a modified version.
Changes affected the acceleration of the system, user interface, search system, integration with internal tools and information systems support services. A detailed story about our modernization OTRS deserves a separate post - we plan to talk about the advantages of the system over other solutions and our implementation experience in the next publication.
Profits from the implementation of OTRS:
In the process of thinking about how to improve efficiency and optimize costs, we came to the idea of the need to transfer support from Moscow to the regions. Now the whole technical support service is located in Nizhny Novgorod.
The final choice of the city was preceded by an analysis of a large number of options, including, for example, Voronezh and Ivanovo. The key parameters for us were the possibility of recruiting the required number of professional staff (in particular, the quality and number of universities available - after all, most of the workers will be students), the cost of the Internet and rent, proximity to an office in Moscow, the transport situation in the city and even the location of the office itself. For example, a problem could be if the office is located in one part of the city, and most of the universities or residential areas in the other, and parts of the city are poorly connected by transport. This may make adjustments to the allowable start or end time for the shift.
The Support Service has a single entry point for all user requests.
To contact the support service, the user can send a request to one of the support service mailboxes (for example, support@corp.mail.ru or abuse@corp.mail.ru) or fill in a special contact form ( http: //help.mail. com / mail-support ). The form (seriously reworked) is a “problem solving wizard” and is designed to help the user find a solution before contacting Customer Support.
There are cases that the user writes his request to the personal corporate box of one of the employees of the company. But even here the corporate policy is such that such user requests are sent to the mailbox support@corp.mail.ru.
Emails from all sources are collected at OTRS. When a new application is received, OTRS forms a ticket in the system and sends an automatic response to the user indicating the ticket number. Such a ticket enters the queue of the first line.
We use a two-tier scheme of work, the support service consists of two “lines”. Each line has its own access rights to tickets (rights define, for example, possible actions with a ticket and a set of queues available to an employee), as well as its own set of internal tools. At the same time, no one has access rights to the user's mailbox, as well as reading the correspondence.
In order not to return to this anymore, let us immediately say what happens with repeated applications. In general, when responding to a user, the ticket is “closed”, i.e. remains in the system, but ceases to constantly "loom" in front of the supportrs in the foreground. When a letter arrives on an existing ticket, it “opens” and becomes visible again. Repeated requests do not go through the first line - the ticket is opened in the queue from which the previous answer was sent.
The entire stream of new applications falls exactly here. On the first line, the most simple, typical problems are solved, applications are distributed to the corresponding queues (for example, there are such queues as "Password recovery", "Problems of mail programs", etc.). One of the main tasks of the first line is to obtain from the user information sufficient to solve the problem on the second line. Also, the first line is a kind of rapid response group, since is at the forefront of receiving feedback from the audience, the first to catch any signs of possible problems and problems.
On the first line, all possible actions with tickets are combined into “chains”, i.e. “In one click” the employee can select the desired sequence of actions:
There is a set of rules that form hints in the interface - which topic or problem this ticket belongs to. This is determined by various parameters of the letter - labels in the body of the letters, the address of the sender and the recipient, the subject, individual headers, etc. The first-line ticket handling interface is optimized for the fastest processing of one ticket. We use several principles:
The purpose of the first line is to respond as quickly as possible to the user's letter. On the day, about 7 - 10 thousand new applications fall on the first line, of which about 40% is transferred to the second line. The average “waiting time” for a ticket on the first line does not exceed an hour; a high processing speed on the first line allows you to drastically reduce the average “lifetime” of a ticket for the Support Service as a whole. Of course, when we talk about the speed of response to an application, this does not mean “speed at any cost”. There are metrics for the quality of processing applications - both individually and for the group as a whole.
Suppose further the application falls on the second line. It is important to note that before getting into the hands of second-line specialists, all problems are distributed in queues, which makes life easier for those who will deal with them in the future, and also allows you to receive important information - in which areas more difficulties arise, and track the overall flow of applications.
The second line consists of four groups and works with its own set of queues:
The second line consists of the most experienced and qualified employees who have received special training. Working in this group people tend to more broadly consider the problem. Staff of the second line has to work with each user individually, in order to give not only a competent, but also an exhaustive answer, which will not require a second contact of the user to the support.
It should be noted that if the first line is focused on mass and speed of responses, then the second line approaches each user's problem individually. The main criterion of the second line is how quickly the user will receive an answer that will satisfy him. All answers that are sent by second-line help desk workers are written directly to the problem of a specific user.
All lines and groups are constantly in close contact with each other, there is a constant flow of information. In particular, there are special conferences in Mail.Ru Agent, in which updates and innovations are announced, the first line consults with the second on operational issues, signals alarming "bells" that can be indicators of potentially major problems. In addition, within these conferences, product managers report on “calculations” of new features, which allows support to prepare for an influx of requests of one type or another, as well as provide operational feedback to the project.
Also within the support there is its own internal knowledge base, built on the Wiki-platform. It not only collects and constantly updates internal instructions, regulations, useful information for new employees (all this radically simplifies the period of adaptation of newly arrived employees and helps them to become most effective in a short time), but cases are constantly accumulating, methods for solving typical problems are added various how-to`s, etc. For example, we have allocated an individual person who is responsible for the relevance of the documentation on the functional features of the project.
Since we have raised the issue of the arrival of new employees, all newcomers go through four stages of training:
Each stage is regulated in time, the coach decides whether or not a person can move from stage to stage. If a person has been studying for too long, a decision may be made about professional incompetence. Ultimately, the certification is conducted by the team leader. By the way, tuition coaches also receive bonuses, i.e. additionally motivated for the development of "growing" personnel.
There is also the so-called “rotational training”, when people shift between different lines and groups in several shifts, which contribute to a better understanding of the difficulties encountered in each other’s work, improves understanding of the problem-solving process - and, of course, increases team spirit.
Speaking of motivation. There are certain sets of indicators by which the work of specialists in support is evaluated (“Stakhanovites”, of course, receive bonuses) - both positive and negative (there are disadvantages for errors, for example, incorrect forward problems, which can seriously slow down the user’s receiving solutions).
A significant share of the “Habra” audience is made up of people who work on Internet projects in one way or another. Surely for many of you the issues of interaction with the target audience working with mailboxes on mail .ru are highly relevant.
All emails from webmasters, mailers, site owners, etc. - all those who send mail of a “non-personal nature” to mail.ru are handled by a specially selected person. It is focused on qualitatively processing all the letters arriving at the specially intended address abuse@corp.mail.ru, to solve every problem.
Sometimes misunderstanding arises from the fact that people who address a problem to this address provide too little technical information needed to resolve the issue. And for a qualitative solution of the problem, we really need a lot of it - error codes, original letters, etc. Please treat with understanding;)
If you still have insurmountable difficulties in dealing with abuse - do not be silent, and write about it in the comments to this post. I personally investigate all particularly difficult cases.
The achieved results allow us to say that we are moving in the right direction. In addition to helping users, the support gives a constant stream of feedback for processing, improving and expanding the Help section ( http://help.mail.ru/mail-help ), and in general - the product as such as a whole. We understand that we still have unsolved problems, the work of a number of processes is far from ideal. But we are systematically following the way of their solution.
Already, the services of other projects, primarily My World, Agent, Search, ICQ, etc., are being transferred to the new support structure.
It is planned to revise the structure of KPI and make them even more ambitious. The speed of response to a user request is not the only important parameter that you should focus on In assessing the effectiveness of the service, we strive to focus not so much on the quantity and quality of letters sent to users, but on the fact of a real solution to the problem (which prompted the user to write to us). Based on this idea, we are building a system of balanced indicators and service performance metrics, including relying on industry practices such as ITIL and COBIT.
PS No, this is not all support staff
PPS Boys in support too;)
I will be glad to answer your questions.
Kondratov Nikolay,
Head of Mail.Ru Support Services
Support today
Now the support service employs more than 50 employees. The support works 7 days a week on a shift schedule, on weekdays it is 2 shifts of 6 hours each (from 9 to 15 and from 15 to 21), on holidays and weekends - one 8-hour shift (this is due to the smaller flow of applications in weekends). This schedule is convenient for students and allows you to combine work with full-time education.
The schedule is flexible, it is made taking into account the possibilities and the desire to work of each employee :) Payment depends on the number of shifts. The logic is simple: all shifts above the minimum wage of the employee, are paid at an increased rate - this encourages people to work more than the minimum level.
')
We use the OTRS application processing system (Open-source Ticket Request System, http://www.otrs.org ), a popular open source project that runs on almost all platforms. OTRS is used in many large companies, for example, in Opera, MySQL AB, SuSE Linux AG, and it is also for us. Unfortunately, the out-of-box system (at the time of implementation it was version 2.2.6) was not suitable for working with large amounts of data (thousands of incoming emails per day and hundreds of thousands of tickets in the database, parallel work of tens of people), therefore, within our Support applies a modified version.
Changes affected the acceleration of the system, user interface, search system, integration with internal tools and information systems support services. A detailed story about our modernization OTRS deserves a separate post - we plan to talk about the advantages of the system over other solutions and our implementation experience in the next publication.
Profits from the implementation of OTRS:
- Work with the system occurs through a web interface - employees are not “tied” to a specific workplace
- All correspondence with the user for one problem is collected in a chain (in terms of application processing systems, this is called a “ticket”). In the ticket you can see all the user's letters, all the answers, comments from support staff, status changes, etc.
- Support for "queues" for various types of tickets, automatic filtering of incoming letters (for example, by their content and headers). A queue is an analogue of a folder in an email client.
- custom response templates
- customizable reports and monitoring for a variety of different system parameters — such as ticket age, emails and responses, individual employee numbers, and more
- support of several projects within one system - with its employees, access rights, schedules, queues, filtering settings, etc.
- integration with grocery task tracker and internal databases of support
- ensuring reliable storage of applications and their history
In the process of thinking about how to improve efficiency and optimize costs, we came to the idea of the need to transfer support from Moscow to the regions. Now the whole technical support service is located in Nizhny Novgorod.
The final choice of the city was preceded by an analysis of a large number of options, including, for example, Voronezh and Ivanovo. The key parameters for us were the possibility of recruiting the required number of professional staff (in particular, the quality and number of universities available - after all, most of the workers will be students), the cost of the Internet and rent, proximity to an office in Moscow, the transport situation in the city and even the location of the office itself. For example, a problem could be if the office is located in one part of the city, and most of the universities or residential areas in the other, and parts of the city are poorly connected by transport. This may make adjustments to the allowable start or end time for the shift.
Support structure or user application life
The Support Service has a single entry point for all user requests.
To contact the support service, the user can send a request to one of the support service mailboxes (for example, support@corp.mail.ru or abuse@corp.mail.ru) or fill in a special contact form ( http: //help.mail. com / mail-support ). The form (seriously reworked) is a “problem solving wizard” and is designed to help the user find a solution before contacting Customer Support.
There are cases that the user writes his request to the personal corporate box of one of the employees of the company. But even here the corporate policy is such that such user requests are sent to the mailbox support@corp.mail.ru.
Emails from all sources are collected at OTRS. When a new application is received, OTRS forms a ticket in the system and sends an automatic response to the user indicating the ticket number. Such a ticket enters the queue of the first line.
We use a two-tier scheme of work, the support service consists of two “lines”. Each line has its own access rights to tickets (rights define, for example, possible actions with a ticket and a set of queues available to an employee), as well as its own set of internal tools. At the same time, no one has access rights to the user's mailbox, as well as reading the correspondence.
In order not to return to this anymore, let us immediately say what happens with repeated applications. In general, when responding to a user, the ticket is “closed”, i.e. remains in the system, but ceases to constantly "loom" in front of the supportrs in the foreground. When a letter arrives on an existing ticket, it “opens” and becomes visible again. Repeated requests do not go through the first line - the ticket is opened in the queue from which the previous answer was sent.
First line
The entire stream of new applications falls exactly here. On the first line, the most simple, typical problems are solved, applications are distributed to the corresponding queues (for example, there are such queues as "Password recovery", "Problems of mail programs", etc.). One of the main tasks of the first line is to obtain from the user information sufficient to solve the problem on the second line. Also, the first line is a kind of rapid response group, since is at the forefront of receiving feedback from the audience, the first to catch any signs of possible problems and problems.
On the first line, all possible actions with tickets are combined into “chains”, i.e. “In one click” the employee can select the desired sequence of actions:
- sends a clarifying question to the user
- sends the answer to a simple question
- closes the ticket (if it is, for example, spam)
- sends the problem to the second line in the desired queue
There is a set of rules that form hints in the interface - which topic or problem this ticket belongs to. This is determined by various parameters of the letter - labels in the body of the letters, the address of the sender and the recipient, the subject, individual headers, etc. The first-line ticket handling interface is optimized for the fastest processing of one ticket. We use several principles:
- only one-click actions
- There are few options for possible actions (about 50). The fewer they are - the faster the ticket is processed, the lower the probability of an error
- the most frequent actions should be as accessible as possible
- tips
The purpose of the first line is to respond as quickly as possible to the user's letter. On the day, about 7 - 10 thousand new applications fall on the first line, of which about 40% is transferred to the second line. The average “waiting time” for a ticket on the first line does not exceed an hour; a high processing speed on the first line allows you to drastically reduce the average “lifetime” of a ticket for the Support Service as a whole. Of course, when we talk about the speed of response to an application, this does not mean “speed at any cost”. There are metrics for the quality of processing applications - both individually and for the group as a whole.
Suppose further the application falls on the second line. It is important to note that before getting into the hands of second-line specialists, all problems are distributed in queues, which makes life easier for those who will deal with them in the future, and also allows you to receive important information - in which areas more difficulties arise, and track the overall flow of applications.
Second line
The second line consists of four groups and works with its own set of queues:
- Group "Passwords" : works with problems accessing accounts. There is a special interface through which user data is verified. It is worth noting that all personal information of users is hidden from employees, the system only shows the coincidence or discrepancy of the data sent with the real. Based on the accuracy of the data sent, the history of requests and changes in the mailbox, and a decision is made to restore access to the account.
- The Web group (historically a name that has arisen, although the group deals with a wider range of problems): questions about the web interface, mail program settings, and the operation of services — the whole range of issues, except for problems with access to accounts. Sometimes the problem is not well described or not reproduced - in this case, the ticket is transferred to the “Solvers”.
- Solvers group : solving the most complex and atypical issues that require not only detailed communication with the user, but also interaction with the technical department. Employees possess deeper technical knowledge. The tasks of the technical department are set by this group.
- Abuse group : working with webmasters' inquiries, phishing complaints, spam, mail delivery issues, etc. The “Abuse” group not only works with user requests, but also closes scam boxes, blocks phishing links, and interacts with spam analysts. Letters sent to security@corp.mail.ru with reports of potential vulnerabilities are highlighted in a separate priority stream.
The second line consists of the most experienced and qualified employees who have received special training. Working in this group people tend to more broadly consider the problem. Staff of the second line has to work with each user individually, in order to give not only a competent, but also an exhaustive answer, which will not require a second contact of the user to the support.
It should be noted that if the first line is focused on mass and speed of responses, then the second line approaches each user's problem individually. The main criterion of the second line is how quickly the user will receive an answer that will satisfy him. All answers that are sent by second-line help desk workers are written directly to the problem of a specific user.
We learn, we instruct, we develop
All lines and groups are constantly in close contact with each other, there is a constant flow of information. In particular, there are special conferences in Mail.Ru Agent, in which updates and innovations are announced, the first line consults with the second on operational issues, signals alarming "bells" that can be indicators of potentially major problems. In addition, within these conferences, product managers report on “calculations” of new features, which allows support to prepare for an influx of requests of one type or another, as well as provide operational feedback to the project.
Also within the support there is its own internal knowledge base, built on the Wiki-platform. It not only collects and constantly updates internal instructions, regulations, useful information for new employees (all this radically simplifies the period of adaptation of newly arrived employees and helps them to become most effective in a short time), but cases are constantly accumulating, methods for solving typical problems are added various how-to`s, etc. For example, we have allocated an individual person who is responsible for the relevance of the documentation on the functional features of the project.
Since we have raised the issue of the arrival of new employees, all newcomers go through four stages of training:
- A person is issued documentation - he reads it, formulates a list of questions.
- Trained is allocated to the trainer. They sit down together, the trainer works - the newbie is watching. His task at this stage is to try to understand what the ticket is about and what to do with it (he is not obliged to remember a specific answer or a specific action, his task is understanding).
- The beginner tries to answer with the coach - the coach adjusts his actions if necessary.
- Responds under the supervision of a coach.
Each stage is regulated in time, the coach decides whether or not a person can move from stage to stage. If a person has been studying for too long, a decision may be made about professional incompetence. Ultimately, the certification is conducted by the team leader. By the way, tuition coaches also receive bonuses, i.e. additionally motivated for the development of "growing" personnel.
There is also the so-called “rotational training”, when people shift between different lines and groups in several shifts, which contribute to a better understanding of the difficulties encountered in each other’s work, improves understanding of the problem-solving process - and, of course, increases team spirit.
Speaking of motivation. There are certain sets of indicators by which the work of specialists in support is evaluated (“Stakhanovites”, of course, receive bonuses) - both positive and negative (there are disadvantages for errors, for example, incorrect forward problems, which can seriously slow down the user’s receiving solutions).
A few words about Abuse
A significant share of the “Habra” audience is made up of people who work on Internet projects in one way or another. Surely for many of you the issues of interaction with the target audience working with mailboxes on mail .ru are highly relevant.
All emails from webmasters, mailers, site owners, etc. - all those who send mail of a “non-personal nature” to mail.ru are handled by a specially selected person. It is focused on qualitatively processing all the letters arriving at the specially intended address abuse@corp.mail.ru, to solve every problem.
Sometimes misunderstanding arises from the fact that people who address a problem to this address provide too little technical information needed to resolve the issue. And for a qualitative solution of the problem, we really need a lot of it - error codes, original letters, etc. Please treat with understanding;)
If you still have insurmountable difficulties in dealing with abuse - do not be silent, and write about it in the comments to this post. I personally investigate all particularly difficult cases.
What will be tomorrow
The achieved results allow us to say that we are moving in the right direction. In addition to helping users, the support gives a constant stream of feedback for processing, improving and expanding the Help section ( http://help.mail.ru/mail-help ), and in general - the product as such as a whole. We understand that we still have unsolved problems, the work of a number of processes is far from ideal. But we are systematically following the way of their solution.
Already, the services of other projects, primarily My World, Agent, Search, ICQ, etc., are being transferred to the new support structure.
It is planned to revise the structure of KPI and make them even more ambitious. The speed of response to a user request is not the only important parameter that you should focus on In assessing the effectiveness of the service, we strive to focus not so much on the quantity and quality of letters sent to users, but on the fact of a real solution to the problem (which prompted the user to write to us). Based on this idea, we are building a system of balanced indicators and service performance metrics, including relying on industry practices such as ITIL and COBIT.
Perfect half of support
PS No, this is not all support staff
PPS Boys in support too;)
I will be glad to answer your questions.
Kondratov Nikolay,
Head of Mail.Ru Support Services
Source: https://habr.com/ru/post/128445/
All Articles