Blocking bots and unwanted users at the nginx web server level

I, yes, and I think you, the web server logs are often clogged with requests like:

62.193.233.148 - - [28/May/2009:18:20:27 +0600] "GET /roundcube/ HTTP/1.0" 404 208 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
62.193.233.148 - - [28/May/2009:18:20:28 +0600] "GET /webmail/ HTTP/1.0" 404 206 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
212.150.123.234 - - [29/May/2009:20:51:12 +0600] "GET /admin/main.php HTTP/1.0" 404 212 "-" "-"
212.150.123.234 - - [29/May/2009:20:51:12 +0600] "GET /phpmyadmin/main.php HTTP/1.0" 404 217 "-" "-"
212.150.123.234 - - [29/May/2009:20:51:12 +0600] "GET /phpMyAdmin/main.php HTTP/1.0" 404 217 "-" "-"
212.150.123.234 - - [29/May/2009:20:51:13 +0600] "GET /db/main.php HTTP/1.0" 404 209 "-" "-"
212.150.123.234 - - [29/May/2009:20:51:13 +0600] "GET /PMA/main.php HTTP/1.0" 404 210 "-" "-"
212.150.123.234 - - [29/May/2009:20:51:14 +0600] "GET /admin/main.php HTTP/1.0" 404 212 "-" "-"
212.150.123.234 - - [29/May/2009:20:51:14 +0600] "GET /mysql/main.php HTTP/1.0" 404 212 "-" "-"
212.150.123.234 - - [29/May/2009:20:51:15 +0600] "GET /myadmin/main.php HTTP/1.0" 404 214 "-" "-"
212.150.123.234 - - [29/May/2009:20:51:15 +0600] "GET /phpadmin/main.php HTTP/1.0" 404 215 "-" "-"
212.150.123.234 - - [29/May/2009:20:51:16 +0600] "GET /webadmin/main.php HTTP/1.0" 404 215 "-" "-"


These are mainly bots, there are also users who scan the server for the presence of any folders, are looking for vulnerabilities.
So, I wanted to block these IP addresses immediately after an attempt to scan the server using nginx tools.
')

Comes to the aid of the geo module nzhinksa .

First, we write in the location section a new log_format of the form “IP address 1;”, which will be understood by the geo module:

log_format deny '$remote_addr 1;';


in the http section we write:

geo $deny {
default 0;
include /www/logs/deny;
}



This will allow us to read the file / www / logs / deny and take a list of IP addresses to block.

Now, in the location of the section we describe “bad” situations when the IP address needs to be blocked, for example:

set $ua $http_user_agent;

if ($ua ~* wget) {
access_log /www/logs/deny deny;
return 403;
}

if ($ua ~* curl) {
access_log /www/logs/deny deny;
return 403;
}

if ($request ~* "webadmin") {
access_log /www/logs/deny deny;
return 403;
}

if ($request ~* "\/admin\/main.php") {
access_log /www/logs/deny deny;
return 403;
}


As a result, bad queries and user agents get into the / www / logs / deny file as “IP address 1;”, and when re-reading the configuration, the IP address will be blocked.

It remains only to throw the nzhinksu command once every 1-5-10 minutes (when necessary) to re-read the config, and the list of blocked IP addresses will be denied access to the server.

According to Sysoev it looks like this:
kill –HUP `cat /var/log/nginx/nginx.pid`

Fill the list of rules with “bad” requests, and the devil himself is not a brother!