How we helped to return to the client 60 thousand dollars accrued for international communication
This story happened to one large brand, operating in the domestic market of not the smallest southern country of the former USSR, let's call it simply - BRAND.
- The network is built on Cisco with small patches of switches and servers from HP. VPNs to headquarters and remote offices;
- Internet on its own fiber from one of the leading providers of the capital;
- ISDN-telephony from another very well-known international company. An advertisement is attached to these numbers, so it’s impossible to refuse them!
- Own staff of IT specialists at BRAND is small and constantly involved in solving current problems, therefore, it actively uses the services of IT outsourcing;
- Monthly expenses of the BRAND only for communication services amount to about $ 3000, which is a very good indicator for us and leads to the VIP, and everything seems to be good, but all of a sudden the bill comes first, at $ 20K, and then at $ 40K.
And then suddenly an understanding began to come that the evening was already no longer languid, and events began to unfold according to the classical scheme:
1. Search for the guilty
2. Punishment of the innocent
3. And rewarding uncomplicated
')
About a couple of months before the story, due to the crisis and in order to optimize costs, it was decided to abandon the monthly support of the IT-outsourcer, and go to pay on call.
In principle, the solution is cost-effective, but suddenly one day, the telephony provider, (having received akick from the international switch of the state telecom), informs BRAND that there are unusually many calls to previously non-specific destinations (Cuba, Nigeria and etc.)
The IT department of BRAND, checking the billing on its internal PBX, did not find any records and, with a shrug, said that everything was fine with them. The provider manager, yawning, went on to pursue his routine.
And everything calmed down again for about a month, until the bills arrived, which were mentioned at the beginning of the story.
The IT specialists of BRAND initially stated thatI was not me and the cow was not mine “since there are no records in the internal billing, it means we didn’t make calls and would not pay,” but then changed the position that once among the conditions for obtaining a telecom license there is a mandatory point of the fight against fraud (which is true in our country), then the telephony provider itself is to blame for the origin of the account, so let him pay for it himself.
The supplier reasonably objected that the client’s equipment was not responsible for the condition, the conversations were recorded not only by him, but also by the international communications center and were unambiguously carried out by the client, so the bill must be paid to the client.
The IT outsourcing company conducted its research and concluded that “the client’s equipment was hacked” and, although by whom and when this unlawful act was committed, they cannot be installed “due to the absence of logs”, but the attackers, after seizing the rights, “poured” customer equipment voice international VoIP traffic.
As a solution, the BRAND is an urgent need! to buy additional equipment from them, services for the N-th sum, to hire another security specialist and work out rules for detecting such situations, etc. etc.
In general, as always, everyone tried to shift the blame from himself and, if possible, chop up the dough.
Having experience, none of the parties wanted to go to law enforcement agencies.
Realizing that everything is in the same boat, the provider agreed to write off part of the amount for telephone conversations, eliminating all of their profit, but the client insisted on the division and the residual amount equally. It came to inviting the team of "martial" lawyers, who, in turn, invited their technical specialists (that is, us) to negotiate.
Several meetings of all interested parties made it possible to find out that, in fact, competent monitoring of events is absent both from the client and from service providers. Ticker system at the outsourcer prematurely repaired. Discussion of SLA from any side causedthoughts of suicide as a persistent feeling that none of those present, including us, understood how to ensure and control this.
Well, we will work with the fact that in stock.
As is well known from time immemorial, Cisco has VoIP implementations, by default there is a bug that allows you to send VoIP traffic to the PSTN simply by specifying the telephone number to call @ address number and this must be closed by the appropriate rules via the access-list.
Without becoming much to go into legal aspects, just to be convinced of the opposite, we wanted to familiarize ourselves with the current configuration of the network infrastructure, and in the process of familiarization, despite assurances from the IT outsourcer in the opposite, it turned out that the logs and configuration files were collected on the storage server data.
Now fine.
First of all, the outsourcers themselves were asked to provide us with the logs and configuration files for the hacked equipment that had been attacked for analysis.
They began to look at the subject of errors and everything seemed to be true, but then we noticed that the dates did not fit between the later and earlier files. Let's say it looked like this:
Early file:
Late file:
In addition, both files have absolutely the same value in the properties of the time of creation and the last change, which in principle is impossible without external intervention.
Having suspected something was wrong, we got access to the files directly through the head of the IT department and on his laptop, we opened the data storage server.
SUSE operating system (this is important!)
The logs are polished, but the analysis of the command history showed that the user, under the name of an outsourcer's employee, made changes to them before transferring files to us for research. And also changed the values ​​of “creation time” and “last modification time” of the specified files, replacing further the originals on the server with their own alterations. Then, apparently, in order to erase the traces of his actions, he deleted everything from the / var / log / directory.
SUSE has one interesting feature about saving temporary files before sending them to a TFTP server. I will not describe everything, who knows and understands what is meant, I recommend others to contact Google. So, with very little effort, we got the originals of all the old configuration files, which revealed the following picture to us:
The IT outsourcing company had about a year as the main support engineer of the client changed, and the client just at that time decided to change the Internet provider. In the process of setting up a new interface, the employee of the outsourcer for some reason left this hole open.
Hmm ... for some reason from the very beginning it seemed so to us. Further, after the bill had already arrived and the company's own mistake became clear, the employees began to “play the fool” and tried to deceive the client. Sad
The expert opinion of our company with technical and legal substantiations, as well as the server disk replica, with materials for the possibility of opening a criminal case, were issued to the BRAND management. Further, the client’s management invited the management of an IT-outsourcer, where they were told what had happened and what they could be for.
At first, of course, they made surprised people, but after 2 days, asking to sign the NDA, they promised to pay for everything.
Since the leadership of the BRAND, to their credit, from the very beginning stated that they do not aim at criminal prosecution, but only want to find out what happened and what needs to be done to prevent this in the future, this incident is now considered to be settled.
It was not the first and not the most significant experience of our participation in the investigation of IT crimes in monetary terms, but it is worth remembering that everywhere, where there is no order in accounting, there is a huge field for theft and deception.
Indeed, in principle, if the outsourcer had the task of imitating a crime, and having organized sabotage, to impose his services, then with competent organization they could well succeed. Therefore, we consider it necessary to collect logs of actions of all administrators and keep them in a place inaccessible to them. For example, the head of his own Security Service. Not because you do not trust, but in order to be sure in the event of an incident that it was not someone of your people who tried to “throw” you. It's like a safe in the house to know for sure that your children have nothing to do with when the money was gone. Or "black box" on the plane. Not because we do not trust the pilot, but simply because he is hacking!
We recommended to the client:
»Deploy a server for collecting logs of actions of IT administrators with access only to the head of the Security Service.
»Event monitoring server for equipment inventory, status monitoring and quality control of received services (by SLA).
»Configure the methods and rules of automatic notification on triggers of unusual situations.
But more about this in the following articles, if you are interested.
- The network is built on Cisco with small patches of switches and servers from HP. VPNs to headquarters and remote offices;
- Internet on its own fiber from one of the leading providers of the capital;
- ISDN-telephony from another very well-known international company. An advertisement is attached to these numbers, so it’s impossible to refuse them!
- Own staff of IT specialists at BRAND is small and constantly involved in solving current problems, therefore, it actively uses the services of IT outsourcing;
- Monthly expenses of the BRAND only for communication services amount to about $ 3000, which is a very good indicator for us and leads to the VIP, and everything seems to be good, but all of a sudden the bill comes first, at $ 20K, and then at $ 40K.
And then suddenly an understanding began to come that the evening was already no longer languid, and events began to unfold according to the classical scheme:
1. Search for the guilty
2. Punishment of the innocent
3. And rewarding uncomplicated
')
The story itself
About a couple of months before the story, due to the crisis and in order to optimize costs, it was decided to abandon the monthly support of the IT-outsourcer, and go to pay on call.
In principle, the solution is cost-effective, but suddenly one day, the telephony provider, (having received a
The IT department of BRAND, checking the billing on its internal PBX, did not find any records and, with a shrug, said that everything was fine with them. The provider manager, yawning, went on to pursue his routine.
And everything calmed down again for about a month, until the bills arrived, which were mentioned at the beginning of the story.
The IT specialists of BRAND initially stated that
The supplier reasonably objected that the client’s equipment was not responsible for the condition, the conversations were recorded not only by him, but also by the international communications center and were unambiguously carried out by the client, so the bill must be paid to the client.
The IT outsourcing company conducted its research and concluded that “the client’s equipment was hacked” and, although by whom and when this unlawful act was committed, they cannot be installed “due to the absence of logs”, but the attackers, after seizing the rights, “poured” customer equipment voice international VoIP traffic.
As a solution, the BRAND is an urgent need! to buy additional equipment from them, services for the N-th sum, to hire another security specialist and work out rules for detecting such situations, etc. etc.
In general, as always, everyone tried to shift the blame from himself and, if possible, chop up the dough.
Having experience, none of the parties wanted to go to law enforcement agencies.
Here it is necessary to clarify: taking into account the practice that has developed in the field, usually in such cases we find out which article of the Criminal Code is violated and who is responsible for this field of activity of the victim. Sentence is rendered to the specialist / head of the injured party, suppliers and persons involved, then the budget is replenished with the amount of fines issued and the case is considered successfully solved. Of course, neither the money, nor the more, no one practically tries to look for the intruders themselves.
Realizing that everything is in the same boat, the provider agreed to write off part of the amount for telephone conversations, eliminating all of their profit, but the client insisted on the division and the residual amount equally. It came to inviting the team of "martial" lawyers, who, in turn, invited their technical specialists (that is, us) to negotiate.
Several meetings of all interested parties made it possible to find out that, in fact, competent monitoring of events is absent both from the client and from service providers. Ticker system at the outsourcer prematurely repaired. Discussion of SLA from any side caused
Well, we will work with the fact that in stock.
As is well known from time immemorial, Cisco has VoIP implementations, by default there is a bug that allows you to send VoIP traffic to the PSTN simply by specifying the telephone number to call @ address number and this must be closed by the appropriate rules via the access-list.
Without becoming much to go into legal aspects, just to be convinced of the opposite, we wanted to familiarize ourselves with the current configuration of the network infrastructure, and in the process of familiarization, despite assurances from the IT outsourcer in the opposite, it turned out that the logs and configuration files were collected on the storage server data.
Now fine.
First of all, the outsourcers themselves were asked to provide us with the logs and configuration files for the hacked equipment that had been attacked for analysis.
They began to look at the subject of errors and everything seemed to be true, but then we noticed that the dates did not fit between the later and earlier files. Let's say it looked like this:
Early file:
! Last configuration change at 18:54:05 Tue Feb 10 2015 by it @ outsourcer
! NVRAM config last updated at 19:02:08 Tue Feb 10 2015 by it @ outsourcer
Late file:
! Last configuration change at 13:38:58 Fri Jan 15 2016 by it @ outsourcer
! NVRAM config last updated at 13:38:59 Fri Jan 15 2016 by it @ outsourcer
In addition, both files have absolutely the same value in the properties of the time of creation and the last change, which in principle is impossible without external intervention.
Having suspected something was wrong, we got access to the files directly through the head of the IT department and on his laptop, we opened the data storage server.
SUSE operating system (this is important!)
The logs are polished, but the analysis of the command history showed that the user, under the name of an outsourcer's employee, made changes to them before transferring files to us for research. And also changed the values ​​of “creation time” and “last modification time” of the specified files, replacing further the originals on the server with their own alterations. Then, apparently, in order to erase the traces of his actions, he deleted everything from the / var / log / directory.
SUSE has one interesting feature about saving temporary files before sending them to a TFTP server. I will not describe everything, who knows and understands what is meant, I recommend others to contact Google. So, with very little effort, we got the originals of all the old configuration files, which revealed the following picture to us:
The IT outsourcing company had about a year as the main support engineer of the client changed, and the client just at that time decided to change the Internet provider. In the process of setting up a new interface, the employee of the outsourcer for some reason left this hole open.
Hmm ... for some reason from the very beginning it seemed so to us. Further, after the bill had already arrived and the company's own mistake became clear, the employees began to “play the fool” and tried to deceive the client. Sad
Conclusion
The expert opinion of our company with technical and legal substantiations, as well as the server disk replica, with materials for the possibility of opening a criminal case, were issued to the BRAND management. Further, the client’s management invited the management of an IT-outsourcer, where they were told what had happened and what they could be for.
At first, of course, they made surprised people, but after 2 days, asking to sign the NDA, they promised to pay for everything.
Since the leadership of the BRAND, to their credit, from the very beginning stated that they do not aim at criminal prosecution, but only want to find out what happened and what needs to be done to prevent this in the future, this incident is now considered to be settled.
Afterword
It was not the first and not the most significant experience of our participation in the investigation of IT crimes in monetary terms, but it is worth remembering that everywhere, where there is no order in accounting, there is a huge field for theft and deception.
Indeed, in principle, if the outsourcer had the task of imitating a crime, and having organized sabotage, to impose his services, then with competent organization they could well succeed. Therefore, we consider it necessary to collect logs of actions of all administrators and keep them in a place inaccessible to them. For example, the head of his own Security Service. Not because you do not trust, but in order to be sure in the event of an incident that it was not someone of your people who tried to “throw” you. It's like a safe in the house to know for sure that your children have nothing to do with when the money was gone. Or "black box" on the plane. Not because we do not trust the pilot, but simply because he is hacking!
We recommended to the client:
»Deploy a server for collecting logs of actions of IT administrators with access only to the head of the Security Service.
»Event monitoring server for equipment inventory, status monitoring and quality control of received services (by SLA).
»Configure the methods and rules of automatic notification on triggers of unusual situations.
But more about this in the following articles, if you are interested.
Source: https://habr.com/ru/post/309970/
All Articles