How a startup can fly out of the index overnight
Hello, Habr!
A year ago we ran into a problem. In a nutshell, we were hacked. Hacked creatively, so that we did not immediately understand what was going on. In the process of solving the problem, it turned out to acquire useful skills: to make abuse calls and effectively communicate with hosters, incl. - foreign, more pleased about the safety of our resource, constantly monitor the availability of our site in the index of Yandex and Google, and be attentive to the little things. In the described case, the usual symbol “-” (hyphen, minus, dash, dash, wand, hyphenosis) played a cruel joke with us.
We faced the problem of hacking for the first time, perhaps some things were obvious from the very beginning, but we did not see them, or our steps in the search for a solution would cause a smile, or a misunderstanding of the pros. Or maybe our experience will be useful for those who are faced with problems in the field of information security.
So, our main resource is located at: www.ist-budget.ru : it has its own permanent audience, an array of indexed pages, for example, a directory of organizations: www.ist-budget.ru/customers.php and www.ist-budget. ru / suppliers.php , which in total contains more than 5 million pages; different interactivity, development perspective and so on. To optimize the resource, development and experimentation, we use a sandbox site with the same name, but on a different domain: www.ist-budget.com . The sandbox is closed from “outside” visits, incl. and from search robots that index pages.
')
The first alarming signals began to come in June 2013, when, according to the queries in the Google search results, our pages began to appear, but not belonging to the main resource, but rather like a sandbox. We checked the robots.txt com site (sandbox), made sure once again that it was closed from indexing, and assumed that Google could speed up some of the pages at the time of the short-term removal of restrictions.
A few days later, there was a reason for more serious concern - the attendance of the main resource on the RU domain sharply decreased, as a result, the number of incoming calls from site visitors dropped. And again the search site surfaced in the search engine sandbox, but with the circumstance from the awareness of which we were simultaneously thrashed with cold sweat, we looked closely and found that there is no dash sign in the name of the sandbox, i.e. instead of the address www.ist-budget.com was the address www.istbudget.com , it is surprising that earlier we paid no attention to this thing. Services such as whois predictably showed that the site was registered just a couple of weeks ago, for a private person, the American domain registrar CLOUDFLARE and the Moldovan hoster Trabia-Network completed the picture in such a way that even the most inexperienced person in our company became extremely clear - this site is not “ sandbox ”and has nothing to do with us .
Detailed analysis showed that the site-clone Moldavashka fully preserved our original design, structure and content of the main domain and subdomain pages. The contact details (address, phone numbers, and details) also remained unchanged, except for the Skype login indicated on the main page: instead of ist-budget.ru, istbudget.com appeared, although this login flatly refused to know Skype (i.e., this login did not was registered).
At the same time, our main domain began to slow down, mainly due to the fact that the clone continued to parse its new pages and updated information in the main sections. Each “freeze” was accompanied by SMS from the useful utility Ping Admin (this utility eventually played almost the most important role in solving the problem), constantly monitoring the performance of the site, the number of “alarming” SMS from it about hanging up was then measured in hundreds.
As it turned out, the clone proxied all requests to us, and in the responses replaced all the links to themselves, adding advertising to the page code. The problem was solved by setting the .htaccess file, namely adding lines
SetEnvIfNoCase REMOTE_ADDR "^ xxx \ .xxx \. [0-9] + \. [0-9] + $" REDIR = toredir
RewriteCond% {REDIR} ^ toredir $ [NC]
RewriteRule ^ (. *) $ Www.ist-budget.com/redir.php?page= $ 1 [R = 301, L]
who sent all the proxy requests from the pool of addresses of a phishing site to a special page, the text of which stated that you are on a phishing site and can find us in Google search by company name. I had to resort to such tricks because the code that proxied all requests to our site automatically replaced all links to the phishing domain.
Also, a large number of requests from a phishing proxy highlighted the problem with optimizing SQL queries and some functions. Requests and functionality were completely reworked, and we used Xdebug and Webgrind as an analysis tool.
The speed was relatively restored, but two problems remained:
1. In the index of search engines still hung a site-clone, not us.
2. On the clone site, there were still updates synchronously with updates on our domain, but without obvious hangs, which added even more bewilderment to us.
And there were more questions than answers. Here is one of them: if search engines have identified a full double of the two sites - why do they eject from the index a site that is clearly older (the domain is registered three years earlier than that of the clone), and not a freshly duplicate? There were a lot of delusional answers, for example, we assumed that the priority for search engines (in particular, Google) is always the site in the domain zone * .com, which is why our main RU-domain “went away”. And there was also a correspondence with Google technical support. Although it’s difficult to call the one-way flow of emails to Google support by correspondence, we simply tried to find out if any sanctions were imposed on our website, which caused the departure from the index. There were no response letters. And then it turned out that we were looking for a solution not at all there.
The idea to write a letter to the provider, where the clone site was hosted, became the brightest thought in the ongoing chaos. Having previously phoned with Moldavian technical support (by the way, they speak Russian fairly tolerably), we made a complaint in Russian and English (such a requirement) languages ​​and sent by ordinary e-mail.
_________________________________________
The letter began like this:
Good day.
On June 13, 2013, the phishing site www.istbudget.com was registered .
The data confirms that it is hosted by you:
IP 178.175.138.11
Host: 178-175-138-11.ip.as43289.net
City: Chisinau
Country: Moldova, Republic of
IP range: 178.175.128.0 - 178.175.255.255
Provider Name: ICS Trabia-Network SRL
Whois Server Version 2.0
Domain Name: ISTBUDGET.COM
Registrar: SHANGHAI YOVOLE NETWORKS INC.
Whois Server: whois.yovole.com
Referral URL: www.yovole.com
Name Server: ERIC.NS.CLOUDFLARE.COM
Name Server: RUTH.NS.CLOUDFLARE.COM
Status: clientTransferProhibited
Updated Date: 13-jun-2013
Creation Date: 13-jun-2013
Expiration Date: 13-jun-2014
...
_________________________________________
The answer did not take long to wait, for two hours without any bureaucratic delays the site-clone was blocked, while trying to go to the site of the clone site Chrome reported that the site address does not exist. Unfortunately, the request to provide information about the owner of the clone site received a refusal and an offer to obtain this data through an appeal to local security agencies. We celebrated the victory and went home, and in the morning ...
And in the morning, to our displeasure, the site-clone was again in service. Fortunately for us, this time he was hosted not at somewhere in China for dubious proxies, but directly at his domain registrar - in America. We already went according to a well-established scheme - we sent a complaint against the abuse of a domain registrar and within a few hours the site was again blocked, this time - finally.
In general, the quick reaction of the provider and the domain registrar pleased me pleasantly. Without additional questions and difficulties, we got the desired result. And then there was an expectation that it would take 2-3 days after the “death” of the clone and our site would return to the search engines index, however, this did not happen, the pages of the clone site were still in the issue, and Chrome regularly reported to that the resource does not exist at this address. And again we wrote letters to the support of Yandex and Google. It was impossible to create a letter in Yandex webmaster, therefore, they simply wrote to tech support in free form.
The solution to the problem was found by chance and not at all where efforts were made: the Ping Admin utility stopped sending SMS when the site was hanging. We paid attention to this, went into the personal account of the service, checked the balance (it was positive), saw that Ping Admin defined our site as inaccessible for several days now. They wrote a letter to the support service, reported an error, and in response received a letter, where tech support reported that our primary domain gives the 301st redirect for search robots to the address of the Komov parasite site, respectively, the robots received the following when entering the main resource signal that the domain is a minor mirror of the new clone site.
Thanks to the 301 redirect, the “mirror man” of Yandex successfully glued our main domain to the new parasitic domain, which is why the site-clone began to be indexed and appeared in the results of search results instead of the main resource.
We found and fixed the vulnerability, having been glad that we got off easily, deleted the redirects that were written by others in our code. By the way, the parasitic code was found quite easily by calling eval (base64_encode ()).
Within 30 minutes, our primary domain returned to the Google index, and a little later, to Yandex. And that's another story.
A year ago we ran into a problem. In a nutshell, we were hacked. Hacked creatively, so that we did not immediately understand what was going on. In the process of solving the problem, it turned out to acquire useful skills: to make abuse calls and effectively communicate with hosters, incl. - foreign, more pleased about the safety of our resource, constantly monitor the availability of our site in the index of Yandex and Google, and be attentive to the little things. In the described case, the usual symbol “-” (hyphen, minus, dash, dash, wand, hyphenosis) played a cruel joke with us.
We faced the problem of hacking for the first time, perhaps some things were obvious from the very beginning, but we did not see them, or our steps in the search for a solution would cause a smile, or a misunderstanding of the pros. Or maybe our experience will be useful for those who are faced with problems in the field of information security.
So, our main resource is located at: www.ist-budget.ru : it has its own permanent audience, an array of indexed pages, for example, a directory of organizations: www.ist-budget.ru/customers.php and www.ist-budget. ru / suppliers.php , which in total contains more than 5 million pages; different interactivity, development perspective and so on. To optimize the resource, development and experimentation, we use a sandbox site with the same name, but on a different domain: www.ist-budget.com . The sandbox is closed from “outside” visits, incl. and from search robots that index pages.
')
The first alarming signals began to come in June 2013, when, according to the queries in the Google search results, our pages began to appear, but not belonging to the main resource, but rather like a sandbox. We checked the robots.txt com site (sandbox), made sure once again that it was closed from indexing, and assumed that Google could speed up some of the pages at the time of the short-term removal of restrictions.
A few days later, there was a reason for more serious concern - the attendance of the main resource on the RU domain sharply decreased, as a result, the number of incoming calls from site visitors dropped. And again the search site surfaced in the search engine sandbox, but with the circumstance from the awareness of which we were simultaneously thrashed with cold sweat, we looked closely and found that there is no dash sign in the name of the sandbox, i.e. instead of the address www.ist-budget.com was the address www.istbudget.com , it is surprising that earlier we paid no attention to this thing. Services such as whois predictably showed that the site was registered just a couple of weeks ago, for a private person, the American domain registrar CLOUDFLARE and the Moldovan hoster Trabia-Network completed the picture in such a way that even the most inexperienced person in our company became extremely clear - this site is not “ sandbox ”and has nothing to do with us .
Detailed analysis showed that the site-clone Moldavashka fully preserved our original design, structure and content of the main domain and subdomain pages. The contact details (address, phone numbers, and details) also remained unchanged, except for the Skype login indicated on the main page: instead of ist-budget.ru, istbudget.com appeared, although this login flatly refused to know Skype (i.e., this login did not was registered).
At the same time, our main domain began to slow down, mainly due to the fact that the clone continued to parse its new pages and updated information in the main sections. Each “freeze” was accompanied by SMS from the useful utility Ping Admin (this utility eventually played almost the most important role in solving the problem), constantly monitoring the performance of the site, the number of “alarming” SMS from it about hanging up was then measured in hundreds.
As it turned out, the clone proxied all requests to us, and in the responses replaced all the links to themselves, adding advertising to the page code. The problem was solved by setting the .htaccess file, namely adding lines
SetEnvIfNoCase REMOTE_ADDR "^ xxx \ .xxx \. [0-9] + \. [0-9] + $" REDIR = toredir
RewriteCond% {REDIR} ^ toredir $ [NC]
RewriteRule ^ (. *) $ Www.ist-budget.com/redir.php?page= $ 1 [R = 301, L]
who sent all the proxy requests from the pool of addresses of a phishing site to a special page, the text of which stated that you are on a phishing site and can find us in Google search by company name. I had to resort to such tricks because the code that proxied all requests to our site automatically replaced all links to the phishing domain.
Also, a large number of requests from a phishing proxy highlighted the problem with optimizing SQL queries and some functions. Requests and functionality were completely reworked, and we used Xdebug and Webgrind as an analysis tool.
The speed was relatively restored, but two problems remained:
1. In the index of search engines still hung a site-clone, not us.
2. On the clone site, there were still updates synchronously with updates on our domain, but without obvious hangs, which added even more bewilderment to us.
And there were more questions than answers. Here is one of them: if search engines have identified a full double of the two sites - why do they eject from the index a site that is clearly older (the domain is registered three years earlier than that of the clone), and not a freshly duplicate? There were a lot of delusional answers, for example, we assumed that the priority for search engines (in particular, Google) is always the site in the domain zone * .com, which is why our main RU-domain “went away”. And there was also a correspondence with Google technical support. Although it’s difficult to call the one-way flow of emails to Google support by correspondence, we simply tried to find out if any sanctions were imposed on our website, which caused the departure from the index. There were no response letters. And then it turned out that we were looking for a solution not at all there.
The idea to write a letter to the provider, where the clone site was hosted, became the brightest thought in the ongoing chaos. Having previously phoned with Moldavian technical support (by the way, they speak Russian fairly tolerably), we made a complaint in Russian and English (such a requirement) languages ​​and sent by ordinary e-mail.
_________________________________________
The letter began like this:
Good day.
On June 13, 2013, the phishing site www.istbudget.com was registered .
The data confirms that it is hosted by you:
IP 178.175.138.11
Host: 178-175-138-11.ip.as43289.net
City: Chisinau
Country: Moldova, Republic of
IP range: 178.175.128.0 - 178.175.255.255
Provider Name: ICS Trabia-Network SRL
Whois Server Version 2.0
Domain Name: ISTBUDGET.COM
Registrar: SHANGHAI YOVOLE NETWORKS INC.
Whois Server: whois.yovole.com
Referral URL: www.yovole.com
Name Server: ERIC.NS.CLOUDFLARE.COM
Name Server: RUTH.NS.CLOUDFLARE.COM
Status: clientTransferProhibited
Updated Date: 13-jun-2013
Creation Date: 13-jun-2013
Expiration Date: 13-jun-2014
...
_________________________________________
The answer did not take long to wait, for two hours without any bureaucratic delays the site-clone was blocked, while trying to go to the site of the clone site Chrome reported that the site address does not exist. Unfortunately, the request to provide information about the owner of the clone site received a refusal and an offer to obtain this data through an appeal to local security agencies. We celebrated the victory and went home, and in the morning ...
And in the morning, to our displeasure, the site-clone was again in service. Fortunately for us, this time he was hosted not at somewhere in China for dubious proxies, but directly at his domain registrar - in America. We already went according to a well-established scheme - we sent a complaint against the abuse of a domain registrar and within a few hours the site was again blocked, this time - finally.
In general, the quick reaction of the provider and the domain registrar pleased me pleasantly. Without additional questions and difficulties, we got the desired result. And then there was an expectation that it would take 2-3 days after the “death” of the clone and our site would return to the search engines index, however, this did not happen, the pages of the clone site were still in the issue, and Chrome regularly reported to that the resource does not exist at this address. And again we wrote letters to the support of Yandex and Google. It was impossible to create a letter in Yandex webmaster, therefore, they simply wrote to tech support in free form.
The solution to the problem was found by chance and not at all where efforts were made: the Ping Admin utility stopped sending SMS when the site was hanging. We paid attention to this, went into the personal account of the service, checked the balance (it was positive), saw that Ping Admin defined our site as inaccessible for several days now. They wrote a letter to the support service, reported an error, and in response received a letter, where tech support reported that our primary domain gives the 301st redirect for search robots to the address of the Komov parasite site, respectively, the robots received the following when entering the main resource signal that the domain is a minor mirror of the new clone site.
Thanks to the 301 redirect, the “mirror man” of Yandex successfully glued our main domain to the new parasitic domain, which is why the site-clone began to be indexed and appeared in the results of search results instead of the main resource.
We found and fixed the vulnerability, having been glad that we got off easily, deleted the redirects that were written by others in our code. By the way, the parasitic code was found quite easily by calling eval (base64_encode ()).
Within 30 minutes, our primary domain returned to the Google index, and a little later, to Yandex. And that's another story.
Source: https://habr.com/ru/post/227267/
All Articles