A day in the life of technical support
Is it good to work in technical support? Well, it depends on what needs to be supported! Today, we will talk about the tasks that support companies in Virtuozzo have to solve, and they will share their secrets - why they came to work specifically for these positions.
So, technical support is a specific job associated with constant communication with customers. No matter how hard we try to make the program code better, there are always enough problems. They can be associated with the next release, and with an atypical scenario of using software, and with unforeseen situations in a personal server client. In our case, all these troubles have to be solved by a support team in which 10 people work - and their number is planned to increase.
')
Leading Technical Support Specialists at Virtuozzo Maria Antonova and Anna Vinokurova
- I have been working in technical support for a long time, and I really like this work from the point of view of language self-improvement. English is the language of international communication, for many of our clients it is not native, and this cannot but leave its imprint on their manner of speech and perception of what you say in response. For those who love communication and linguistics in general, but see themselves in a technical profile, it seems to me that working in such an environment is very useful for improving language skills and career development, ”says Anna Vinokurova.
The practice of the technical support service of Virtuozzo has derived several rules for itself that must be followed when communicating with customers from different regions. In this case, it is easier to work, and results are achieved faster. Here are some of them:
• Clients from the Asian region should not write too long and detailed letters; of all the letters, only the first couple of paragraphs will most likely be read.
• When verbally communicating with US customers, it is necessary several times to give confirmation that they understood you correctly.
• Germans and Austrians can issue quite comprehensive instructions consisting of many points, and they will apply them exactly as it was written
In general, the specificity of the work of supporters is a very high degree of spontaneity. You can never guess at what point something goes wrong and engineers should be, as they say, “always ready.” Clients sometimes need urgent help for various reasons, but basically any urgency is associated with obligations to end users - after all, most of the Virtuozzo clients are hosters, and non-working virtual machines or containers are someone's inaccessible sites or databases.
However, there are predictable load waves, they are usually associated with the release of new products, the release of major updates, or the discovery of another Linux vulnerability.
“When a regular security advisory bulletin is published on the Internet, anxious clients begin to come to the support team who want to know which versions are affected by the vulnerability, and how soon we will release an update,” notes Anna Vinokurova.
- I really like working in technical support, because this work combines the tasks of a software expert, a programmer and a detective. Sometimes the problem is so disguised that its solution is not at all obvious. And each time, when it is possible to solve the riddle, “why nothing works,” a feeling of deep satisfaction and even celebration comes! - says Maria Antonova.
What does the Virtuozzo support do?
For those who are wondering whether to go to work for supportrs, we will tell a few stories from the real practice of our technical support.
1. Memory is not added - CPU error
One night, the client reports support for an error in an attempt to increase RAM for one of the virtual machines. However, the error for some reason, "swears" on the CPU, not RAM.
At first, this situation was very surprised by everyone, but after a while we figured out that the error returned was not directly related to the action being performed - it only revealed a failure that hid much deeper inside the system.
The reason turned out to be quite serious: on the physical server, the CPU socket was knocked out, and it ceased to be detected, and with it all virtual cores within the socket went offline. Naturally, when changing the memory limits for a virtual machine, validation of the future configuration did not take place simply as a whole, because the number of cores began to exceed the number of available cores on the server node.
The fact is that in the process of validating the configuration file, the NUMA arrays, and the CPUs that belong to them, are processed. On a physically “healthy” server, no NUMA should be empty, and the situation arose because no one had previously assumed that a server that was damaged in this way would be able to work for a long time.
Decision
On the client server, a correction was made to the python-component of the product, which made it possible to continue correct operation, despite the existence of NUMA nodes consisting entirely of offline-cores. After that, we were able to set the number of CPUs within the real “surviving” cores for the virtual machine, which means that further configuration changes were possible.
However, more global improvements followed: in the product code, the definition of CPU statuses was improved on an ongoing basis: an empty array with virtual CPUs belonging to the same NUMA group no longer causes errors in the code and is considered a legal situation.
“In fact, the client was very lucky that his server remained accessible for quite a long time, despite such a serious hardware failure. This allowed virtual machines to migrate to other hosts while they were still available, ”said Maria Antonova.
2. Lost network connection in virtual machines
The client noticed a periodic loss of access to the network in the virtual machines. However, if such a virtual machine is migrated to another physical server or restarted on the current one, the network works again.
To investigate the problem, it was necessary to get a virtual machine when the problem was reproduced, but the services inside the virtual machines would have to work without interruption, and our online research would cause problems for a large number of end users. Therefore, the client provided us with a copy of the virtual machine, on which we were able to reproduce the problem and investigate it in our environment without inconvenience to users. The bug was found in the memory allocation mechanism for the virtual-socket buffer of the virtual device virtio-net - part of the kernel module.
Decision
Alas, there was simply no quick solution to the problem, without changing the kernel modules, it turned out that the client simply did not upgrade to the latest version of Virtuozzo Linux. At the time of the appeal, an update has already been released, and in the ReadyKernel format. This allowed to apply the update and solve the problem in a few minutes, even without restarting the services.
3. User has deleted all containers.
One client decided to directly change the path to the private area of ​​the containers, but due to an error in the shell script prepared by him, the path for all containers began to point to the same place — the common storage of all containers. Then, as expected, one of the end users decided to remove their own container. The deletion led to the disappearance of all containers in the storage, as the updated configuration pointed this way.
Decision
To prevent further data loss, the client control panel was temporarily disabled for end users so that they could not stop the containers, as this would lead to further data loss. With the containers completely removed, there was nothing to do, so the task was to recover the metadata for those containers that remained running.
Since the affected containers were about 400 pieces, the task was quite creative - it was necessary to write a script that would make similar, but not the same type of manipulation for all the victims. As a deeper inspection showed, for 200 containers, the data were completely lost, since the containers were at that moment in a stopped state. But for running containers, the deletion did not complete completely, since their virtual disks were locked. In such containers all data was lost except for the virtual disks themselves - metadata, configuration files and other service files.
Physically, the virtual disk files were transferred to a temporary directory for deleting files, and stopping such a container would result in the final deletion of data.
Since metadata varies from container to container, they need to be created from scratch individually. Configuration files were restored on the basis of billing information and tariff plans of end users - this was done by the client himself, since our support service does not have access to the billing information of the client. But we, in turn, provided something like a “skeleton” of the configuration, in which you had to enter IP, host name, and resource limits. To restore the disk metadata, the Virtuozzo supporters created a script that generated the correct metadata based on information about the mounted containers from the kernel, and also calculated the virtual disk parameters, such as the number of cylinders, heads and sectors.
By the way, after this call to the container removal mechanism, a check was turned on whether the deleted directory is a container - this will avoid such problems in the future, even if the user runs the script with an error.
And a detective, and a linguist, and a programmer
Thus, we can confidently say that our support service is a very interesting team, ready to solve many-sided problems and even engage in the finalization of the product. And if you want to try yourself in this kind of work, you can work at night and are not afraid of Asian English, we can open new vacancies!
So, technical support is a specific job associated with constant communication with customers. No matter how hard we try to make the program code better, there are always enough problems. They can be associated with the next release, and with an atypical scenario of using software, and with unforeseen situations in a personal server client. In our case, all these troubles have to be solved by a support team in which 10 people work - and their number is planned to increase.
')
Leading Technical Support Specialists at Virtuozzo Maria Antonova and Anna Vinokurova
- I have been working in technical support for a long time, and I really like this work from the point of view of language self-improvement. English is the language of international communication, for many of our clients it is not native, and this cannot but leave its imprint on their manner of speech and perception of what you say in response. For those who love communication and linguistics in general, but see themselves in a technical profile, it seems to me that working in such an environment is very useful for improving language skills and career development, ”says Anna Vinokurova.
The practice of the technical support service of Virtuozzo has derived several rules for itself that must be followed when communicating with customers from different regions. In this case, it is easier to work, and results are achieved faster. Here are some of them:
• Clients from the Asian region should not write too long and detailed letters; of all the letters, only the first couple of paragraphs will most likely be read.
• When verbally communicating with US customers, it is necessary several times to give confirmation that they understood you correctly.
• Germans and Austrians can issue quite comprehensive instructions consisting of many points, and they will apply them exactly as it was written
In general, the specificity of the work of supporters is a very high degree of spontaneity. You can never guess at what point something goes wrong and engineers should be, as they say, “always ready.” Clients sometimes need urgent help for various reasons, but basically any urgency is associated with obligations to end users - after all, most of the Virtuozzo clients are hosters, and non-working virtual machines or containers are someone's inaccessible sites or databases.
However, there are predictable load waves, they are usually associated with the release of new products, the release of major updates, or the discovery of another Linux vulnerability.
“When a regular security advisory bulletin is published on the Internet, anxious clients begin to come to the support team who want to know which versions are affected by the vulnerability, and how soon we will release an update,” notes Anna Vinokurova.
- I really like working in technical support, because this work combines the tasks of a software expert, a programmer and a detective. Sometimes the problem is so disguised that its solution is not at all obvious. And each time, when it is possible to solve the riddle, “why nothing works,” a feeling of deep satisfaction and even celebration comes! - says Maria Antonova.
What does the Virtuozzo support do?
For those who are wondering whether to go to work for supportrs, we will tell a few stories from the real practice of our technical support.
1. Memory is not added - CPU error
One night, the client reports support for an error in an attempt to increase RAM for one of the virtual machines. However, the error for some reason, "swears" on the CPU, not RAM.
At first, this situation was very surprised by everyone, but after a while we figured out that the error returned was not directly related to the action being performed - it only revealed a failure that hid much deeper inside the system.
The reason turned out to be quite serious: on the physical server, the CPU socket was knocked out, and it ceased to be detected, and with it all virtual cores within the socket went offline. Naturally, when changing the memory limits for a virtual machine, validation of the future configuration did not take place simply as a whole, because the number of cores began to exceed the number of available cores on the server node.
The fact is that in the process of validating the configuration file, the NUMA arrays, and the CPUs that belong to them, are processed. On a physically “healthy” server, no NUMA should be empty, and the situation arose because no one had previously assumed that a server that was damaged in this way would be able to work for a long time.
Decision
On the client server, a correction was made to the python-component of the product, which made it possible to continue correct operation, despite the existence of NUMA nodes consisting entirely of offline-cores. After that, we were able to set the number of CPUs within the real “surviving” cores for the virtual machine, which means that further configuration changes were possible.
However, more global improvements followed: in the product code, the definition of CPU statuses was improved on an ongoing basis: an empty array with virtual CPUs belonging to the same NUMA group no longer causes errors in the code and is considered a legal situation.
“In fact, the client was very lucky that his server remained accessible for quite a long time, despite such a serious hardware failure. This allowed virtual machines to migrate to other hosts while they were still available, ”said Maria Antonova.
2. Lost network connection in virtual machines
The client noticed a periodic loss of access to the network in the virtual machines. However, if such a virtual machine is migrated to another physical server or restarted on the current one, the network works again.
To investigate the problem, it was necessary to get a virtual machine when the problem was reproduced, but the services inside the virtual machines would have to work without interruption, and our online research would cause problems for a large number of end users. Therefore, the client provided us with a copy of the virtual machine, on which we were able to reproduce the problem and investigate it in our environment without inconvenience to users. The bug was found in the memory allocation mechanism for the virtual-socket buffer of the virtual device virtio-net - part of the kernel module.
Decision
Alas, there was simply no quick solution to the problem, without changing the kernel modules, it turned out that the client simply did not upgrade to the latest version of Virtuozzo Linux. At the time of the appeal, an update has already been released, and in the ReadyKernel format. This allowed to apply the update and solve the problem in a few minutes, even without restarting the services.
3. User has deleted all containers.
One client decided to directly change the path to the private area of ​​the containers, but due to an error in the shell script prepared by him, the path for all containers began to point to the same place — the common storage of all containers. Then, as expected, one of the end users decided to remove their own container. The deletion led to the disappearance of all containers in the storage, as the updated configuration pointed this way.
Decision
To prevent further data loss, the client control panel was temporarily disabled for end users so that they could not stop the containers, as this would lead to further data loss. With the containers completely removed, there was nothing to do, so the task was to recover the metadata for those containers that remained running.
Since the affected containers were about 400 pieces, the task was quite creative - it was necessary to write a script that would make similar, but not the same type of manipulation for all the victims. As a deeper inspection showed, for 200 containers, the data were completely lost, since the containers were at that moment in a stopped state. But for running containers, the deletion did not complete completely, since their virtual disks were locked. In such containers all data was lost except for the virtual disks themselves - metadata, configuration files and other service files.
Physically, the virtual disk files were transferred to a temporary directory for deleting files, and stopping such a container would result in the final deletion of data.
Since metadata varies from container to container, they need to be created from scratch individually. Configuration files were restored on the basis of billing information and tariff plans of end users - this was done by the client himself, since our support service does not have access to the billing information of the client. But we, in turn, provided something like a “skeleton” of the configuration, in which you had to enter IP, host name, and resource limits. To restore the disk metadata, the Virtuozzo supporters created a script that generated the correct metadata based on information about the mounted containers from the kernel, and also calculated the virtual disk parameters, such as the number of cylinders, heads and sectors.
By the way, after this call to the container removal mechanism, a check was turned on whether the deleted directory is a container - this will avoid such problems in the future, even if the user runs the script with an error.
And a detective, and a linguist, and a programmer
Thus, we can confidently say that our support service is a very interesting team, ready to solve many-sided problems and even engage in the finalization of the product. And if you want to try yourself in this kind of work, you can work at night and are not afraid of Asian English, we can open new vacancies!
Source: https://habr.com/ru/post/329950/
All Articles