Incredibly true stories from the life of tech support
Today we continue the series of posts about the work of the technical support team, begun at one time by Loxmatiymamont in an article on Veeam support .
It would seem, what is Technical Support? You sit for yourself, solve technical problems, you are the smartest, most knowledgeable, you are the same Engineer, to whom the Scared Users come. They bring their fears, pains, problems, and you decide, help, advise, and, ultimately, the User leaves you not Frightened, but Winged.
Have you already felt the full significance of this work, the deep philosophical and pedagogical principles contained in it?
')
So, everything is a little different. Technical support is, first of all, Support, and then Technical, and therefore all the work is 99% percent about people and communication with them, so we will have something to do with textbook misanthropes and pathological introverts, but it will be difficult this time, and two - people are not always predictable, and therefore, working in Technical Support, you can learn, see and hear a lot of interesting and unusual. Under the cut, I will share with readers a couple of such stories.
The background is as follows: the client had its NTP server (the one that is responsible for synchronizing the time on the network), and on a variety of hosts, the time jumped is completely unpredictable. Of course, this does not apply to Veeam, and the client, as a technically literate specialist, decided everything himself, but: the client using Veeam backed up their MariaDB servers, and did a separate script as a base dump and backup binary logs. Everyday.
Having dealt with NTP, the client checked the backups made by our software and saw a terrible thing: 12 days of backup of the binary logs disappeared somewhere, and the task report showed success. Everyday.
It is to solve this mystery called us.
A quick investigation led us to the main culprit, who turned out to be the very NTP server. How?
This is how: while jumping in time, the NTP server generously put on the server the first days of September of the peasant uprising in Normandy, the subordination of Kashmir by the Afghans and the beginning of the construction of the Tobolo-Ishim fortified line in the Russian Empire - that is, the year 1752 of Christmas. Due to a strange whim of history, it was in September of this year that the UK and its North American colonies decided to switch to the Gregorian calendar, and therefore the month looks like this:
So both the base dumps and the backup logs were actually made every day, only these days in 1752 turned out to be exactly 12 less than usual, and the logs for September 3 were dated fourteenth, the fourth - fifteenth, and so on.
The same problem manifested itself when users used the Hijri calendar - in this case, backups went without a hitch and without a hitch, but when restoring data, the service stubbornly went crazy trying to work with the database created around 1400.
But not always problems with time are software problems: once our engineer had to prove the client for two weeks that the second day of the week was Tuesday, but not Thursday, which the client, a completely respectable European bourgeois, completely refused to believe.
The story itself is causing, if not a smile, then puzzled twitching with one eyebrow for sure.
The client suffered from problems in all his backup tasks for a long time: they died time after time, as did the main service that monitors their work. The engineer who dealt with the problem quickly found out: before each episode, the time in our logs jumped one hour into the past, then into the future. The reason was found out simply by asking the client.
Believing an explanation and suppressing the question “why?” Turned out to be much more difficult.
Imagine: there is a ship somewhere in the Atlantic Ocean. Our software is installed on it, and a small script that tracks the GPS coordinates of the ship is spinning alongside. Every time the ship crosses the regular time zone, the script changes the time on all servers and services, including ours, and moreover, to be sure that everyone is working with the right time, it also reloads some of the services. A good example of a creative solution in the style of "we solved one problem at the cost of the appearance of the other two").
PS In the process of working on this article, I was reminded of the Dovecot mail server and its method of handling similar situations:
Fatal: Time just moved backwards by 7 seconds. This might cause a lot of problems. (Catastrophic error: there was a transfer of time to 7 seconds ago. This can lead to a lot of problems, so I'm just eliminating myself.)
The US comrade who called us suffered very long from the hangs of his Veeam service and also did not want to try the reboot for mystic reasons (the uptime of the machine had been calculated for years), until finally he gave up and explained the reason for his resistance:
“You see, on this machine with Windows 7, not only Veeam is spinning, but also the Smart Home controller: all cameras, sensors, lighting, alarms and everything. The last time we restarted it, our boiler exploded. ”
We have Surebackup technology, which allows you to run backups in an isolated environment and check how successful they are not only in the report, but also in reality (and did they turn into a pumpkin with the last percentage of the task). Good technology, fairly reliable, built on the use of several verification scenarios, including checking for accessibility over the network (for example, checking the port of one or another application).
In order to work with an isolated network, we run a separate applaens (small virtual machine) that lives as a gateway and as the main working tool of all tests - and here something went wrong.
The Applaens was launched successfully, but we could not connect to it ourselves - once the network port was unavailable. Cyclic scanning of IP-port ports quickly showed that sometimes our port is, and sometimes it is not, but port 544 TCP appears from somewhere, which should not be even in theory. We try other addresses - the script repeats, we check arp - mac-addresses are different.
In complete confusion, we open the address with a web browser and are surprised to see the login of some Chinese web camera. Once again we change the IP address for the applause, and we get exactly the same picture - for some reason, all the IP addresses on the network will redirect to this camera, which is what we and the client understood.
The riddle remained unresolved.
Do you think these are isolated examples? Anyone accidentally taken by an engineer can tell you about a dozen of similar stories, without really straining and never repeating.
This is exactly what gives our work a taste and drive. Such stories help to remember that not all of our work comes down to a dry solution of technical problems, it contains comedy and drama, a mystery and a challenge. This is our resource. This is what helps us to meet every call not with a tired “well, what do you want again?”, But with a joyful anticipation, ”and how can you please me today?”
Boasting the words of one of the vice-presidents of our company, said about the support:
But this is a completely different story.
And what about you, dear readers? Have something to share in the comments?
Yes, if you recognized yourself in the previous paragraph, take a look at careers.veeam.ru/departments/support , perhaps there is a vacancy for you.
It would seem, what is Technical Support? You sit for yourself, solve technical problems, you are the smartest, most knowledgeable, you are the same Engineer, to whom the Scared Users come. They bring their fears, pains, problems, and you decide, help, advise, and, ultimately, the User leaves you not Frightened, but Winged.
Have you already felt the full significance of this work, the deep philosophical and pedagogical principles contained in it?
')
So, everything is a little different. Technical support is, first of all, Support, and then Technical, and therefore all the work is 99% percent about people and communication with them, so we will have something to do with textbook misanthropes and pathological introverts, but it will be difficult this time, and two - people are not always predictable, and therefore, working in Technical Support, you can learn, see and hear a lot of interesting and unusual. Under the cut, I will share with readers a couple of such stories.
Story One, Detective: Veeam Support and a Dozen Lost Days
The background is as follows: the client had its NTP server (the one that is responsible for synchronizing the time on the network), and on a variety of hosts, the time jumped is completely unpredictable. Of course, this does not apply to Veeam, and the client, as a technically literate specialist, decided everything himself, but: the client using Veeam backed up their MariaDB servers, and did a separate script as a base dump and backup binary logs. Everyday.
Having dealt with NTP, the client checked the backups made by our software and saw a terrible thing: 12 days of backup of the binary logs disappeared somewhere, and the task report showed success. Everyday.
It is to solve this mystery called us.
A quick investigation led us to the main culprit, who turned out to be the very NTP server. How?
This is how: while jumping in time, the NTP server generously put on the server the first days of September of the peasant uprising in Normandy, the subordination of Kashmir by the Afghans and the beginning of the construction of the Tobolo-Ishim fortified line in the Russian Empire - that is, the year 1752 of Christmas. Due to a strange whim of history, it was in September of this year that the UK and its North American colonies decided to switch to the Gregorian calendar, and therefore the month looks like this:
So both the base dumps and the backup logs were actually made every day, only these days in 1752 turned out to be exactly 12 less than usual, and the logs for September 3 were dated fourteenth, the fourth - fifteenth, and so on.
The same problem manifested itself when users used the Hijri calendar - in this case, backups went without a hitch and without a hitch, but when restoring data, the service stubbornly went crazy trying to work with the database created around 1400.
But not always problems with time are software problems: once our engineer had to prove the client for two weeks that the second day of the week was Tuesday, but not Thursday, which the client, a completely respectable European bourgeois, completely refused to believe.
Second Story - Comedy Geographical Location: Veeam Support and 20,000 Leagues Over the Water
The story itself is causing, if not a smile, then puzzled twitching with one eyebrow for sure.
The client suffered from problems in all his backup tasks for a long time: they died time after time, as did the main service that monitors their work. The engineer who dealt with the problem quickly found out: before each episode, the time in our logs jumped one hour into the past, then into the future. The reason was found out simply by asking the client.
Believing an explanation and suppressing the question “why?” Turned out to be much more difficult.
Imagine: there is a ship somewhere in the Atlantic Ocean. Our software is installed on it, and a small script that tracks the GPS coordinates of the ship is spinning alongside. Every time the ship crosses the regular time zone, the script changes the time on all servers and services, including ours, and moreover, to be sure that everyone is working with the right time, it also reloads some of the services. A good example of a creative solution in the style of "we solved one problem at the cost of the appearance of the other two").
PS In the process of working on this article, I was reminded of the Dovecot mail server and its method of handling similar situations:
Fatal: Time just moved backwards by 7 seconds. This might cause a lot of problems. (Catastrophic error: there was a transfer of time to 7 seconds ago. This can lead to a lot of problems, so I'm just eliminating myself.)
Story Three - “horror”: Veeam Support and an exploding boiler
The US comrade who called us suffered very long from the hangs of his Veeam service and also did not want to try the reboot for mystic reasons (the uptime of the machine had been calculated for years), until finally he gave up and explained the reason for his resistance:
“You see, on this machine with Windows 7, not only Veeam is spinning, but also the Smart Home controller: all cameras, sensors, lighting, alarms and everything. The last time we restarted it, our boiler exploded. ”
The fourth story, the mystical: Veeam Support and the eye of heaven
We have Surebackup technology, which allows you to run backups in an isolated environment and check how successful they are not only in the report, but also in reality (and did they turn into a pumpkin with the last percentage of the task). Good technology, fairly reliable, built on the use of several verification scenarios, including checking for accessibility over the network (for example, checking the port of one or another application).
In order to work with an isolated network, we run a separate applaens (small virtual machine) that lives as a gateway and as the main working tool of all tests - and here something went wrong.
The Applaens was launched successfully, but we could not connect to it ourselves - once the network port was unavailable. Cyclic scanning of IP-port ports quickly showed that sometimes our port is, and sometimes it is not, but port 544 TCP appears from somewhere, which should not be even in theory. We try other addresses - the script repeats, we check arp - mac-addresses are different.
In complete confusion, we open the address with a web browser and are surprised to see the login of some Chinese web camera. Once again we change the IP address for the applause, and we get exactly the same picture - for some reason, all the IP addresses on the network will redirect to this camera, which is what we and the client understood.
The riddle remained unresolved.
***
Do you think these are isolated examples? Anyone accidentally taken by an engineer can tell you about a dozen of similar stories, without really straining and never repeating.
This is exactly what gives our work a taste and drive. Such stories help to remember that not all of our work comes down to a dry solution of technical problems, it contains comedy and drama, a mystery and a challenge. This is our resource. This is what helps us to meet every call not with a tired “well, what do you want again?”, But with a joyful anticipation, ”and how can you please me today?”
Boasting the words of one of the vice-presidents of our company, said about the support:
“ Technical Support - they are monster! They (technical support are some unrealistic guys! They not only know how to solve technical problems, but also know how to properly conduct a dialogue with a client. ”)And we really can, moreover, we teach this: a person with a good language (especially the second or third, besides English), who knows how to communicate with customers and understands why this is necessary, we introduce IT in general and our product in particular (you would know how many talented graduates of language universities work for us - and in fact they started with almost zero knowledge!). A good technical specialist can improve English and help develop soft-skills.
But this is a completely different story.
***
And what about you, dear readers? Have something to share in the comments?
Yes, if you recognized yourself in the previous paragraph, take a look at careers.veeam.ru/departments/support , perhaps there is a vacancy for you.
Source: https://habr.com/ru/post/460058/
All Articles