How we fought parsers

Key points:

* Implementing a script to check the PTR of visitors;

* Configuring nginx in IfIsEvil-style with branching map;

* Location names in map variables;

* Branch control via try_files / nonexist $ map_var.

Many high-load and popular sites suffer from the fact that in addition to live visitors they are visited by various parsers, bots and other automated scanners that do not have any beneficial effect, but only create spurious traffic and load on the already loaded system. In this case, I do not mean the search bots, which, although often load the project is not normalized, but are simply necessary for any project.

One of our clients regularly experienced the problem of an avalanche-like increase in load at certain times of the day. Periodically, once a day and more often, there was an influx of visits with a significant increase in LA on the servers. It was decided to build protection from parasitic traffic.

')

We found that sneak traffic has certain behavior patterns and properties, namely:

* By request source - subnet Amazon, Tor;

* By entry points - most requests to the section of goods;

* According to UserAgent - the main part of the bots was sent by the UA search engines Google, Yandex, Bing, but they were not objectively them;

* By referrer - basically the referrer was empty.

Implemented checks, restrictions and locks:

* Manually added pre-known IPs to the white list

* And previously compromised IP - blacklisted

* Limit limit_req_zone for all but whitelists

* Checking visitors from the UA search engines for PTR-record compliance with the real PTR-records of the search engines and put on the white list of the tested and always allow them.

* When LA increases the “Attack” threshold:

** We do not pass the UA check on PTR in the black list and block.

** Check the repeatability of referrers in the access-log and block visitors with the referrer exceeding the specified number of entries

** The rest is checked by captcha and resolved if successful.

The first three points are common solutions, so I’ll focus more on checking visitors by PTR, configuring nginx using the try_files / nonexist $ map_var construction and complex maps.

We have implemented a script for asynchronous checking of visitors from search engines UA to match PTR records with real PTR records of search engines.

It runs on cron once a minute. According to the unique IP list of visitors from the search engine UA, it performs a PTR check and checks the second-level domain name. If the domain matches, it adds IP to the white list, otherwise to the black list. When checking the list, the script does not check the PTR of previously verified IPs to speed up the verification process. This allows you to go through the IP list from the access-log every minute, even at a high speed of filling the access-log. Blacklisting entries are recorded with an indication of a fixed time to remove from the block list in order to avoid permanent blocking of common IPs in large NAT networks.

Thus, we create and maintain the ptr_blacklist.map and ptr_whitelist.map files that are included in the nginx config.

Runs every minute.

Listing of the UA and PTR compliance check script:

#!/bin/bash #   ,     , #        ,    . # Export inc file for nginx EXPORT_MAP=true # Domain list DOMAIN_LIST="domain" # Block time (in minutes) BLOCK_TIME=1440 # White list IP IP_WHITELIST="" # White list PTR BOTS="google|yandex|bing|Bing|msn" # false - not block IP if there is a PTR record BLOCK_WITH_PTR=true UNBLOCK_ENABLE=true LOGFILE=/var/log/ua-table.log LOGFILE2=/var/log/ua-table-history.log LOCK=/tmp/ua_check.lock D=$DOMAIN_LIST #   ptr_blacklist  ptr_whitelist       map- #      BL_FILE=/etc/nginx/vhosts.d/ptr_blacklist WL_FILE=/etc/nginx/vhosts.d/ptr_whitelist BL_FILE_MAP=$BL_FILE.map WL_FILE_MAP=$WL_FILE.map TMP_LOG=/tmp/$D-acc-temp.log TMP_LOG1=/tmp/$D-acc-temp1.log NGINX_LOG=/srv/www/$D/shared/log/$D-acc.log [ ! -f /usr/bin/host ] && echo "/usr/bin/host not found. Please yum install bind-utils" && exit [ -z "$DOMAIN_LIST" ] && echo "DOMAIN_LIST is empty" [ ! -f $LOGFILE ] && touch $LOGFILE [ ! -f $LOGFILE2 ] && touch $LOGFILE2 debug="0" function e { echo -e $(/bin/date "+%F %T") $1 } #     ,       [ -f $LOCK ] && e "Script $0 is already runing" && exit /bin/touch $LOCK DT=`/bin/date "+%F %T"` if [ ! -f $NGINX_LOG ];then echo "Log ($NGINX_LOG) not found." /bin/rm -rf $LOCK exit fi #    #    acc-,  ,         referer /bin/egrep "Yandex|Google|bingbot|Bing" $NGINX_LOG | /usr/bin/awk '{print $1}' | /bin/sort -n | /usr/bin/uniq > $TMP_LOG if [ "$EXPORT_MAP" == "true" ]; then [ ! -f $BL_FILE_MAP ] && /bin/touch $BL_FILE_MAP [ ! -f $WL_FILE_MAP ] && /bin/touch $WL_FILE_MAP [ ! -f $BL_FILE ] && /bin/touch $BL_FILE || /bin/cp -f $BL_FILE $BL_FILE.bak [ ! -f $WL_FILE ] && /bin/touch $WL_FILE || /bin/cp -f $WL_FILE $WL_FILE.bak fi #   UNBLOCK=0 while read line do if [[ "$line" == *=* ]]; then GET_TIME=`echo $line | /usr/bin/awk -F"=" '{print $2}'` NOW=`/bin/date '+%s'` #echo $NOW #echo $GET_TIME if [ "$NOW" -gt "$GET_TIME" ]; then IP=`echo $line | awk '{print $3}'` e "$IP unblocked." >> $LOGFILE2 /bin/sed -i '/'$IP'/d' $BL_FILE /bin/sed -i '/'$IP'/d' $LOGFILE UNBLOCK=1 #else #e "Nothing to unblock" >> $LOGFILE2 fi fi done < $LOGFILE #   while read line do IP=$line wl=0 bl=0 #    WL for I in $IP_WHITELIST do if [ "$I" = "$IP" ];then wl=1 fi done #       WL for I in $(/usr/bin/awk '{print $1}' < "$WL_FILE" ) do if [ "$I" = "$IP" ];then wl=1 fi done #       BL for I in $(/usr/bin/awk '{print $1}' < "$BL_FILE" ) do if [ "$I" = "$IP" ];then bl=1 fi done #  IP   ,    ,    PTR if [ "$wl" = "1" -o "$bl" = "1" ]; then [ "$debug" -gt "1" ] && e "$IP in white or black list" >> $LOGFILE2 else PTR="" SRCHBOT="" FINDPTR="`/usr/bin/host $IP | /bin/grep -v 'not found' | /bin/grep -v 'no PTR record' | /usr/bin/head -1 | /usr/bin/awk '{ print $5 }' | /bin/sed 's/\.$//'`" if [ -z "$FINDPTR" ];then PTR=" (PTR record not found)" else PTR=" ($FINDPTR)" fi SRCHBOT=`/usr/bin/host $IP | /usr/bin/awk '{ print $5 }' | /usr/bin/rev | /usr/bin/cut -d . -f 2-3 | /usr/bin/rev | /bin/egrep "$BOTS"` [ -n "$SRCHBOT" ] && BOT="YES" || BOT="NO" [ -z "$BLOCK_WITH_PTR" ] && BLOCK_WITH_PTR=true if [ "$EXPORT_MAP" == "true" ]; then if [ "$BOT" == "NO" ]; then e "$IP blocked $BLOCK_TIME minutes. ($D) Unblock = `/bin/date --date="$BLOCK_TIME minute" +%s`" >> $LOGFILE e "$IP$PTR blocked $BLOCK_TIME minutes. ($D)" >> $LOGFILE2 echo "$IP 0;" >> $BL_FILE else echo "$IP 1;" >> $WL_FILE fi fi fi done < $TMP_LOG #     map- if [ "$EXPORT_MAP" == "true" ]; then /bin/sort -u -o $BL_FILE $BL_FILE > /dev/null 2>&1 /bin/sort -u -o $WL_FILE $WL_FILE > /dev/null 2>&1 MAP_CHANGED=0 if ! diff $BL_FILE $BL_FILE.bak > /dev/null 2>&1; then /bin/cp -f $BL_FILE_MAP $BL_FILE_MAP.bak > /dev/null 2>&1 /bin/cp -f $BL_FILE $BL_FILE_MAP > /dev/null 2>&1 MAP_CHANGED=1 fi if ! diff $WL_FILE $WL_FILE.bak > /dev/null 2>&1; then /bin/cp -f $WL_FILE_MAP $WL_FILE_MAP.bak > /dev/null 2>&1 /bin/cp -f $WL_FILE $WL_FILE_MAP > /dev/null 2>&1 MAP_CHANGED=1 fi if [ "$MAP_CHANGED" -eq "1" -o "$UNBLOCK" -eq "1" ]; then RELOAD=`/usr/sbin/nginx -t 2>&1 | /bin/grep ok` if [ -n "$RELOAD" ];then /sbin/service nginx reload e "nginx is reloaded" >> $LOGFILE2 else ERROR_RELOAD=`/sbin/service nginx configtest 2>&1` /bin/cp -f $BL_FILE_MAP.bak $BL_FILE_MAP > /dev/null 2>&1 /bin/cp -f $WL_FILE_MAP.bak $WL_FILE_MAP > /dev/null 2>&1 e "nginx error config test failed" >> $LOGFILE2 fi fi fi /bin/rm -rf $LOCK

Script to check the frequency of referrers and the formation of the file referer-block.conf type

 ~domain.ru 0; ~… 1; ~… 1;

Runs every minute.

Listing of the referrer frequency checking script:

 #!/bin/bash # referer_protect v.1.0.6 #   ,     , #        ,    . RECORDS=500 DOMAIN_LIST=domain LA=15 # if Load Average > $LA = Referer is block BLOCK_TIME=360 #in minutes #REF_WHITELIST="" BLOCK_ENABLE=true # true/false - enable/disable add firewall rule. email="mail@mail.ru" LOGFILE=/var/log/referer-table.log LOGFILE2=/var/log/referer-table-history.log LOCK=/tmp/referer.lock MSG_ALERT=/tmp/msg-alert.tmp debug="0" LA_CURRENT="`cat /proc/loadavg | awk '{ print $1}' | awk 'BEGIN { FS="."; }{ print $1}'`" DT=`date "+%F %T"` [ ! -f $LOGFILE ] && touch $LOGFILE [ -f "$MSG_ALERT" ] && rm -f $MSG_ALERT function e { echo -e $(date "+%F %T") $1 } function msg { echo "Referer:$REFERER. Domain:$D" >> $MSG_ALERT } function send_mail { if [ "$BLOCK_ENABLE" = "true" -a "$LA_CURRENT" -gt "$LA" ];then cat $MSG_ALERT | mailx -s "Referers report. Warning" $email elif [ "$BLOCK_ENABLE" = "true" -a "$LA_CURRENT" -le "$LA" ];then cat $MSG_ALERT | mailx -s "Referers report. Notice " $email else cat $MSG_ALERT | mailx -s "Referers report. Notice (Test mode)" $email fi } [ -f $LOCK ] && e "Script $0 is already runing" && exit touch $LOCK NEED_NGINX_RELOAD=0 for D in $DOMAIN_LIST do TMP_LOG=/tmp/ddos-$D-acc-referer.log TMP_AWK=/tmp/tmp_$D-awk.tmp #NGINX_LOG=/srv/www/$D/logs/$D-acc NGINX_LOG=/srv/www/$D/shared/log/$D-acc.log REFCONF=/etc/nginx/referer-block-$D.conf [ ! -s "$REFCONF" ] && echo "~$D 0;" >> $REFCONF if [ ! -f $NGINX_LOG ];then echo "Log ($NGINX_LOG) not found." /bin/rm -rf $LOCK exit fi tail -10000 $NGINX_LOG | awk '($9 == "200") || ($9 == "404")' | awk '{print $11}' | sort | uniq -c | sort -n | awk -vx=$RECORDS ' $1 > x {print $2} ' > $TMP_LOG sed -i "s/\"//g" $TMP_LOG #   sed -i "/^-/d" $TMP_LOG #  referer "-" sed -i "/$D/d" $TMP_LOG #    sed -i "/^localhost/d" $TMP_LOG #  localhost awk -F/ '{print $3}' $TMP_LOG > $TMP_AWK #     url cat $TMP_AWK > $TMP_LOG #   referer while read line do if [[ "$line" == *=* ]]; then GET_TIME=`echo $line | awk -F"=" '{print $2}'` NOW=`date +%s` #echo $NOW #echo $GET_TIME if [ "$NOW" -gt "$GET_TIME" ]; then REFERER=`echo $line | awk '{print $4}'` e "Referer $REFERER unblocked." >> $LOGFILE2 /bin/sed -i '/'$REFERER'/d' $LOGFILE /bin/sed -i '/'$REFERER'/d' $REFCONF NEED_NGINX_RELOAD=1 fi fi done < $LOGFILE #  referer while read line do REFERER=$line DOUBLE=`cat $REFCONF | grep "$REFERER"` if [ -n "$DOUBLE" ]; then [ "$debug" != "0" ] && e "referer $REFERER exist in DROP rule" else if [ "$BLOCK_ENABLE" = "true" -a "$LA_CURRENT" -gt "$LA" ];then echo "~$REFERER 1;" >> $REFCONF e "Referer $REFERER blocked $BLOCK_TIME minutes ($D) Unblock = `date --date="$BLOCK_TIME minute" +%s`" >> $LOGFILE e "Referer $REFERER blocked $BLOCK_TIME minutes ($D)" >> $LOGFILE2 NEED_NGINX_RELOAD=1 if [ ! -s "$MSG_ALERT" ];then echo "Status: WARNING" > $MSG_ALERT echo "Date: $DT" >> $MSG_ALERT echo "Referer: $RECORDS matches from 10000" >> $MSG_ALERT echo "LA: $LA_CURRENT" >> $MSG_ALERT echo "Referer(s) is blocked on $BLOCK_TIME minutes:" >> $MSG_ALERT echo "" >> $MSG_ALERT fi msg elif [ "$BLOCK_ENABLE" = "true" -a "$LA_CURRENT" -le "$LA" ];then TESTDOUBLE=`cat $LOGFILE | grep "$REFERER"` if [ -z "$TESTDOUBLE" ]; then e "Referer $REFERER TEST blocked $BLOCK_TIME minutes ($D) Unblock = `date --date="$BLOCK_TIME minute" +%s`" >> $LOGFILE e "TEST. Referer $REFERER TEST blocked $BLOCK_TIME minutes ($D)" >> $LOGFILE2 if [ ! -s "$MSG_ALERT" ];then echo "Status: Notice" > $MSG_ALERT echo "Date: $DT" >> $MSG_ALERT echo "Referer: $RECORDS matches from 10000" >> $MSG_ALERT echo "LA: $LA_CURRENT" >> $MSG_ALERT echo "Referer(s) not blocking:" >> $MSG_ALERT echo "" >> $MSG_ALERT fi msg fi else TESTDOUBLE=`cat $LOGFILE | grep "$REFERER"` if [ -z "$TESTDOUBLE" ]; then e "Referer $REFERER TEST blocked $BLOCK_TIME minutes ($D) Unblock = `date --date="$BLOCK_TIME minute" +%s`" >> $LOGFILE e "TEST. Referer $REFERER TEST blocked $BLOCK_TIME minutes ($D)" >> $LOGFILE2 if [ ! -s "$MSG_ALERT" ];then echo "Date: $DT" > $MSG_ALERT echo "Current referer found over $RECORDS matches from 10000 records, but script working is TEST MODE " >> $MSG_ALERT echo "Current LA - $LA_CURRENT" >> $MSG_ALERT echo "Referer(s) not blocking:" >> $MSG_ALERT echo "" >> $MSG_ALERT fi msg fi fi fi done < $TMP_LOG [ -n "email" -a -s "$MSG_ALERT" ] && send_mail done # reload nginx if config change if [ $NEED_NGINX_RELOAD -eq 1 ]; then /sbin/service nginx reload >/dev/null 2>/dev/null fi /bin/rm -rf $LOCK

The nginx configuration file as a symlink points to one of two files operating in normal and high LA modes.

The mode is switched by the script executed every minute from Cron.

Script switching modes:

 #!/bin/bash ### check LA level MAX_LA=10 processid=`/sbin/pidof -x $(basename $0) -o %PPID` if [[ $processid ]];then exit fi CFG_DDOS='fpm.domain.ru.ddos' CFG_NODDOS='fpm.domain.ru.noddos' load_average=$(uptime | awk '{print $11}' | cut -d "." -f 1) echo "$(date '+%Y-%m-%d %H:%M') : LA $load_average" if [[ $load_average -ge $MAX_LA ]]; then if [ -f /tmp/la_flag ]; then date '+%s' > /tmp/la_flag exit 1 else # echo "$(date +%Y-%m-%d-%H-%M)" date '+%s' > /tmp/la_flag mv /etc/nginx/vhosts.d/new.domain.ru.conf /etc/nginx/vhosts.d/new.domain.ru.conf.bak > /dev/null 2>&1 ln -s /etc/nginx/vhosts.d/$CFG_DDOS /etc/nginx/vhosts.d/new.domain.ru.conf reload=`/usr/sbin/nginx -t 2>&1 | grep ok` if [ -n "$reload" ];then /sbin/service nginx reload rm -f /etc/nginx/vhosts.d/new.domain.ru.conf.bak > /dev/null 2>&1 echo "$(date '+%Y-%m-%d %H:%M') : DDOS config up $reload" exit 0 else /sbin/service nginx configtest 2>&1 mv /etc/nginx/vhosts.d/new.domain.ru.conf.bak /etc/nginx/vhosts.d/new.domain.ru.conf > /dev/null 2>&1 echo "nginx error config ddos test failed" echo "alarm nginx config ddos test failed" | mail -s alarm root exit 1 fi fi else if [ -f /tmp/la_flag ]; then TIMEA=`cat /tmp/la_flag` TIMEC=`date '+%s'` TIMED=$(( $TIMEC - $TIMEA )) if [ $TIMED -gt 600 ]; then echo "high LA ENDED $(date +%Y-%m-%d-%H-%M)" rm -f /tmp/la_flag > /dev/null 2>&1 mv /etc/nginx/vhosts.d/new.domain.ru.conf /etc/nginx/vhosts.d/new.domain.ru.conf.bak > /dev/null 2>&1 ln -s /etc/nginx/vhosts.d/$CFG_NODDOS /etc/nginx/vhosts.d/new.domain.ru.conf reload=`/usr/sbin/nginx -t 2>&1 | grep ok` echo "$(date '+%Y-%m-%d %H:%M') : NO DDOS config up $reload" if [ -n "$reload" ];then /sbin/service nginx reload rm -f /etc/nginx/vhosts.d/new.domain.ru.conf.bak > /dev/null 2>&1 echo "$(date '+%Y-%m-%d %H:%M') : NO ddos config up" exit 0 else /sbin/service nginx configtest 2>&1 mv /etc/nginx/vhosts.d/new.domain.ru.conf.bak /etc/nginx/vhosts.d/new.domain.ru.conf > /dev/null 2>&1 echo "nginx error config noddos test failed" echo "alarm nginx config noddos test failed" | mail -s alarm root exit 1 fi else exit 1 fi else exit 1 fi fi

A part of the configuration is placed in a separate file that is included in the main configuration files.

A file with general configuration parameters for both vhosts.d / map.domain.ru.inc modes:

 map_hash_bucket_size 128; geoip_country /usr/share/GeoIP/GeoIP.dat; limit_req_zone $newlimit_addres1 zone=newone:10m rate=50r/m; map $whitelist-$remote_addr:$remote_port $newlimit_addres1 { ~"^0" $binary_remote_addr; ~"^1-(?<match_rap>.*)" $match_rap; } geo $whitelist { default 0; 91.205.47.150 1; 194.87.91.154 1; 83.69.225.78 1; 77.88.18.82 1; 91.143.46.202 1; 213.180.192.0/19 1; 87.250.224.0/19 1; 77.88.0.0/18 1; 93.158.128.0/18 1; 95.108.128.0/17 1; 178.154.128.0/17 1; 199.36.240.0/22 1; 84.201.128.0/18 1; 141.8.128.0/18 1; 188.134.88.105 1; 89.163.3.25 1; 46.39.246.91 1; 84.21.76.123 1; 136.243.83.53 1; 77.50.238.152 1; 83.167.117.49 1; 109.188.82.40 1; 79.141.227.19 1; 176.192.62.78 1; 86.62.91.133 1; 144.76.88.101 1; } #     block_referer.sh map $http_referer $bad_referer { default "0"; include /etc/nginx/referer-block.conf; } map $http_referer:$request_method $bad_post_referer { default "0"; "~*domain.ru.*:POST$" "0"; "~*:POST$" "1"; include /etc/nginx/referer-block.conf; } #      map $query_string $bad_query { ... default 0; } #  ,     /checkcapcha.php map $http_cookie $allowed_cookie { "~somecookie" 1; default 0; }   GeoIP     map $geoip_country_code $allowed_country { RU 1; default 0; } #   Amazon include vhosts.d/deny-amazon.inc; #      map $remote_addr $valid_addr { include vhosts.d/main_blacklist.map; include vhosts.d/main_whitelist.map; default 2; } # UA - map $http_user_agent $user_agent_search_bot { "~Yandex" "1"; "~Google" "1"; "~*bing" "1"; "~*MSNBot" "1"; default ""; } map $remote_addr $ptr_wl_bl { include vhosts.d/ptr_blacklist.map; include vhosts.d/ptr_whitelist.map; default ""; } map "$user_agent_search_bot:$ptr_wl_bl" $searchbot { "1:1" "1"; "1:0" "0"; default "2"; }

Configuration File Listings

The main config for normal operation is vhosts.d/fpm.domain.ru.noddos:

 include vhosts.d/map.domain.ru.inc; map "$searchbot:$valid_addr:$bad_referer:$bad_query" $root_location_p1 { default @allow_limit; "~^1:" @allow; "~^2:1" @allow_limit; "~^0" @loc_403; "~^2:0" @loc_403; "2:2:1:0" @loc_403; "2:2:1:1" @loc_403; "2:2:0:1" @loc_403; } map "$searchbot:$valid_addr:$bad_post_referer:$bad_query" $root_only_location_p1 { default @allow_limit; "~^1:" @allow; "~^2:1" @allow_limit; "~^0" @loc_403; "~^2:0" @loc_403; "2:2:1:0" @loc_403; "2:2:1:1" @loc_403; "2:2:0:1" @loc_403; } ######################################################## server { listen 80; listen 443 ssl; fastcgi_read_timeout 300s; fastcgi_send_timeout 300s; fastcgi_connect_timeout 300s; server_name domain.ru www.domain.ru m.domain.ru www.m.domain.ru; ssl_certificate ssl/www.domain.ru.crt; ssl_certificate_key ssl/www.domain.ru.key; charset UTF-8; access_log /srv/www/domain/shared/log/domain-acc.log main; error_log /srv/www/domain/shared/log/domain-err.log; root /srv/www/domain/current/public/; error_page 500 502 /highla.html; #  capcha   POST  action="/checkcapcha.php" location = /highla.html { charset UTF-8; root /srv/www/domain/current/public/; allow all; } #       . location = /checkcapcha.php { charset UTF-8; root /srv/www/domain/current/public/; include fastcgi_params; fastcgi_buffers 8 16k; fastcgi_buffer_size 32k; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; fastcgi_param REQUEST_SCHEME $scheme; fastcgi_param HTTPS $https if_not_empty; fastcgi_pass 127.0.0.1:9000; allow all; } #  location       map location @loc_403 { access_log /srv/www/domain/shared/log/loc_403-acc main; return 403; } location @allow { access_log /srv/www/domain/shared/log/allow-acc main; add_header X-debug-message "Allow"; try_files $uri /index.php?$query_string; } location @allow_limit { limit_req zone=newone burst=15; access_log /srv/www/domain/shared/log/allow-acc main; add_header X-debug-message "Allow"; try_files $uri /index.php?$query_string; } location @deny { access_log /srv/www/domain/shared/log/deny-acc main; add_header X-debug-message "Deny"; return 403; } location @restrict { access_log /srv/www/domain/shared/log/resrtict-acc main; add_header X-debug-message "Restrict"; return 502; } location / { try_files /fake-nonexistens-location-forr273 $root_location_p1; } location = / { try_files /fake-nonexistens-location-forr273 $root_only_location_p1; } location ~* \.php { include fastcgi_params; fastcgi_buffers 8 16k; fastcgi_buffer_size 32k; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; fastcgi_param REQUEST_SCHEME $scheme; fastcgi_param HTTPS $https if_not_empty; fastcgi_pass 127.0.0.1:9000; } location ~* \.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|tar|mid|midi|wav|bmp|rtf|js|swf|flv|avi|djvu|mp3)$ { root /srv/www/domain/current/public; expires 7d; access_log off; log_not_found off; } location ~ /\.git { deny all; } location ~ /\.ht { deny all; } location ~ /\.svn { deny all; } }

Config for high LA mode vhosts.d/fpm.domain.ru.ddos:

 include vhosts.d/map.domain.ru.inc; map "$searchbot:$valid_addr:$bad_referer:$bad_query" $root_location { default @main; "~^1:" @allow; "~^2:1" @allow; "~^0" @loc_403; "~^2:0" @loc_403; "2:2:1:0" @loc_403; "2:2:1:1" @loc_403; "2:2:0:1" @loc_403; } map "$searchbot:$valid_addr:$bad_post_referer:$bad_query" $root_only_location { default @main; "~^1:" @allow; "~^2:1" @allow; "~^0" @loc_403; "~^2:0" @loc_403; "2:2:1:0" @loc_403; "2:2:1:1" @loc_403; "2:2:0:1" @loc_403; } map "$allowed_country:$allowed_cookie" $main_location { "1:0" @allow_limit; "1:1" @allow_limit; "0:1" @allow_limit; default @restrict; } ######################################################## server { listen 80; listen 443 ssl; fastcgi_read_timeout 300s; fastcgi_send_timeout 300s; fastcgi_connect_timeout 300s; server_name domain.ru www.domain.ru m.domain.ru www.m.domain.ru; ssl_certificate ssl/www.domain.ru.crt; ssl_certificate_key ssl/www.domain.ru.key; charset UTF-8; access_log /srv/www/domain/shared/log/domain-acc.log main; error_log /srv/www/domain/shared/log/domain-err.log; root /srv/www/domain/current/public/; #  capcha   POST  action="/checkcapcha.php" error_page 500 502 /highla.html; location = /highla.html { charset UTF-8; root /srv/www/domain/current/public/; allow all; } #       . location = /checkcapcha.php { charset UTF-8; root /srv/www/domain/current/public/; include fastcgi_params; fastcgi_buffers 8 16k; fastcgi_buffer_size 32k; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; fastcgi_param REQUEST_SCHEME $scheme; fastcgi_param HTTPS $https if_not_empty; fastcgi_pass 127.0.0.1:9000; allow all; } #  location       map location @loc_403 { access_log /srv/www/domain/shared/log/loc_403-acc main; return 403; } location @allow { access_log /srv/www/domain/shared/log/allow-acc main; add_header X-debug-message "Allow"; try_files $uri /index.php?$query_string; } location @allow_limit { limit_req zone=newone burst=55; access_log /srv/www/domain/shared/log/allow-limit-acc main; add_header X-debug-message "Allow"; try_files $uri /index.php?$query_string; } location @deny { access_log /srv/www/domain/shared/log/deny-acc main; add_header X-debug-message "Deny"; return 403; } location @restrict { access_log /srv/www/domain/shared/log/resrtict-acc main; add_header X-debug-message "Restrict"; return 502; } location @main { add_header X-debug-message "Main"; try_files /fake-nonexistens-location-forr273 $main_location; } location / { try_files /fake-nonexistens-location-forr273 $root_location; } location = / { try_files /fake-nonexistens-location-forr273 $root_only_location; } location ~* \.php { include fastcgi_params; fastcgi_buffers 8 16k; fastcgi_buffer_size 32k; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; fastcgi_param REQUEST_SCHEME $scheme; fastcgi_param HTTPS $https if_not_empty; fastcgi_pass 127.0.0.1:9000; } location ~* \.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|tar|mid|midi|wav|bmp|rtf|js|swf|flv|avi|djvu|mp3)$ { root /srv/www/domain/current/public; expires 7d; access_log off; log_not_found off; } location ~ /\.git { deny all; } location ~ /\.ht { deny all; } location ~ /\.svn { deny all; } }

Total

With this solution, we helped our client protect their project from spurious traffic and increase server stability.

Author: Leading System Administrator of the company Marat Rakhimov.

Source: https://habr.com/ru/post/278553/

All Articles