Protection against content thieves
When you make a website with a lot of unique information and start earning traffic from search engines, it can be very disappointing when someone downloads a website entirely and takes away half of your visitors. Therefore, in the framework of the struggle with the above-mentioned problem, I developed a simple but effective protection.
The point of protection is that a summary of requests from suspicious IPs is sent to the admin, and he looks at the human eye and determines whether it is a robot or a human, and then blocks, or vice versa, allows requests from this IP.
')
The suspicion of an IP address is determined by the number of requests in a given period of time.
The summary is taken grep-ohm from the web server log, in the letter the admin is immediately given WHOIS-information, and links to ban / resolve the address.
After some time, the admin produces a trained eye, which immediately distinguishes the robot from a human, the processing of such a letter takes several seconds.
There are several obvious differences between a robot and a person, which are immediately noticeable in the log:
1. A person downloads all images, scripts and scripts from a page, robots are often only HTML.
2. A person spends different times on reading different pages, the robot most often quickly requests the pages one by one.
3. The passage of a person through the pages of the site is logical, the robot either iterates through all the links on a page in a row, or requests those for which there are no links on the current page at all.
So, we identified the thief and blocked his access, for example, we make a delay for a few seconds and issue a garbage packet instead of a page. But the thief is not a fool and he will immediately understand what the matter is and build in his rocking chair a simple check and, in the case of blocking, simply change his IP.
But the power is in truth, but the truth is with us, so we will apply a very insidious trick, which I will describe next time, and the thief will end up with bitter annoyance, resentment and disappointment.
The point of protection is that a summary of requests from suspicious IPs is sent to the admin, and he looks at the human eye and determines whether it is a robot or a human, and then blocks, or vice versa, allows requests from this IP.
')
The suspicion of an IP address is determined by the number of requests in a given period of time.
The summary is taken grep-ohm from the web server log, in the letter the admin is immediately given WHOIS-information, and links to ban / resolve the address.
After some time, the admin produces a trained eye, which immediately distinguishes the robot from a human, the processing of such a letter takes several seconds.
There are several obvious differences between a robot and a person, which are immediately noticeable in the log:
1. A person downloads all images, scripts and scripts from a page, robots are often only HTML.
2. A person spends different times on reading different pages, the robot most often quickly requests the pages one by one.
3. The passage of a person through the pages of the site is logical, the robot either iterates through all the links on a page in a row, or requests those for which there are no links on the current page at all.
So, we identified the thief and blocked his access, for example, we make a delay for a few seconds and issue a garbage packet instead of a page. But the thief is not a fool and he will immediately understand what the matter is and build in his rocking chair a simple check and, in the case of blocking, simply change his IP.
But the power is in truth, but the truth is with us, so we will apply a very insidious trick, which I will describe next time, and the thief will end up with bitter annoyance, resentment and disappointment.
Source: https://habr.com/ru/post/53649/
All Articles