We are protected from HTTP DDoS and other Habraeffektov

A simple way to protect against HTTP DDoS is to enable syn-cookies and block scams. But what to do if it attacks 5k-10k hosts and even with dynamic IPs? This is where the frontend-backend architecture with intermediate caching comes to the rescue! Why with intermediate caching? And because in my case, from a barrage of requests from the frontend, the backend died taking the system with it.

So the algorithm of action:

• We change the port of our backend server to any other than 80 (let it be 2080)
• Install Varnish
• Install and configure lighttpd
• We limit the number of connections from one host to iptables

So, as I had several servers available, the configuration version with several servers will be considered, but no one bothers to push it all onto one server =)

I don’t need to tell how to change the port of your favorite web server, I suggest to go straight to setting up Varnish.

Actually install the package itself:
apt-get update && apt-get install varnish

Next, we provide a C-like configuration file (in Ubuntu, this is /etc/varnish/default.vcl ) to something like this:
backend default {
.host = "1.1.1.1"; #IP backend'
.port = "2080"; #
.first_byte_timeout = 300s; # varnish backend'
}

acl purge {
"localhost"; #
}

sub vcl_recv {
if (req.request == "GET" && req.url ~ "\.(jpg|jpeg|gif|ico)$") {
lookup;
}

if (req.request == "GET" && req.url ~ "\.(css|js)$") {
lookup;
}

if (req.request == "GET" && req.url ~ "\.(pdf|xls|vsd|doc|ppt|iso)$") {
lookup;
}

if (req.request == "POST") {
pipe;
}

if (req.request != "GET" && req.request != "HEAD") {

if (req.request == "PURGE") {
if (!client.ip ~ purge) {
error 405 "Not allowed.";
}
lookup;
}

pipe;
}

if (req.http.Expect) {
pipe;
}

if (req.http.If-None-Match) {
pass;
}

if (req.http.Authenticate || req.http.Authorization) {
pass;
}
lookup;

}

sub vcl_hit {
if (req.request == "PURGE") {
set obj.ttl = 0s;
error 200 "Purged.";
}
}

sub vcl_miss {
if (req.http.If-Modified-Since) {
pass;
}

if (req.request == "PURGE") {
error 404 "Not in cache.";
}
}

sub vcl_fetch {
if ( obj.http.x-accel-redirect ~ ".*" ) {
set req.url = obj.http.x-accel-redirect;
restart;
}
}

')
Restart varnish: service varnish restart

Now we can proceed to the installation and configuration of our frontend - lighttpd.
I prefer to take lighttpd from here , but no one bothers you to download it from the distribution repositories ( apt-get install lighttpd ).
And we govern the config until it takes the following form:
server.modules = (
"mod_cache",
"mod_proxy",
"mod_access",
"mod_evasive"
)

server.network-backend = "writev"
server.max-keep-alive-requests = 4
server.max-keep-alive-idle = 4
server.max-read-idle = 10
server.max-write-idle = 30
server.event-handler = "linux-sysepoll"
server.stat-cache-engine = "disable"
server.protocol-http11 = "enable"
server.max-worker = 2 # 1
server.max-fds = 10000
server.max-connections = 5000
server.port = 80
server.document-root = "/var/www"
server.errorlog = "/var/log/lighttpd/error.log"
server.pid-file = "/var/run/lighttpd.pid"
server.username = "www-data"
server.groupname = "www-data"
etag.use-inode = "enable"
etag.use-mtime = "enable"
etag.use-size = "enable"
server.dir-listing = "disable"
evasive.max-conns-per-ip = 3 #

cache.enable = "enable" #
cache.bases = ("/var/spool/cache") #
cache.max-memory-size = 40960 #40Gb
cache.lru-remove-count = 512
cache.support-queries = "enable"
cache.dynamic-mode = "enable"
cache.refresh-pattern = (
"\.(?i)(js|css|xml|po)$" => "240", # update js/css/xml every 4 hours and on refresh requests
"\.(?i)(htm|html|shtml)$" => "30 use-memory", # update html/htm/shtml every 30 minutes and on refresh requests
"\.(?i)(jpg|bmp|jpeg|gif|png)$" => "2880", # update graphics files every 2 days
"\.(?i)(rar|zip|wmv|iso|avi|mp3|ape|rm|mpeg|mpg|wma|asf|rmvb|flv|mkv|ogg|ogm|swf|flac)$" => "0 fetchall-for-range-request", # cache media file forever
".(?i)php$" => "5", # update php request every 5 minutes
"." => "30 use-memory" #
)

mimetype.use-xattr = "enable"
include_shell "/usr/share/lighttpd/create-mime.assign.pl"

#Bad users go to hell
$HTTP["useragent"] == "" {
url.access-deny = ( "" )
}

$HTTP["host"] =~ "(^|\.)habrahabr\.ru$" {
proxy.balance = "round-robin"
proxy.server = ( "/" =>
(
( "host" => "1.2.1.1", "port" => 6081 ), #
( "host" => "1.2.1.2", "port" => 6081 ), # varnish
( "host" => "1.2.1.3", "port" => 6081 ),
( "host" => "1.2.1.4", "port" => 6081 )
)
)
}
proxy.worked-with-mod-cache = "enable"


And finally, iptables and a small system tuning:

We allow 10 connections per second from one IP:
iptables -I INPUT 1 -p tcp -m hashlimit --hashlimit-upto 10/sec --hashlimit-burst 10 --hashlimit-mode srcip --hashlimit-name HTTPD_DOS -m tcp --dport 80 -m state --state NEW -j ACCEPT
Increase the number of open files:
ulimit -n 5000
Buns for sysctl.conf:
vm.swappiness=10
vm.vfs_cache_pressure=10000
vm.dirty_ratio = 1
vm.dirty_background_ratio = 1
vm.dirty_writeback_centisecs = 250
vm.dirty_expire_centisecs = 3000
kernel.panic = 10
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_sack = 1
net.core.rmem_max = 16777216
net.core.rmem_default = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
net.ipv4.netfilter.ip_conntrack_max = 1048576
net.nf_conntrack_max = 1048576
net.ipv4.icmp_echo_ignore_all = 1
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 15
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 15
net.ipv4.ip_local_port_range= 10000 65000

How does all this work? lighttpd receives a request from the client and if it meets certain criteria (in our case it is not an empty User-agent, the client requests the domain habrahabr.ru and this is not the 4th simultaneous request) sends the request to one of the varnish servers. Varnish checks its cache for the presence of the desired content by the user, either returns it from the cache or sends a request to the backend if the cache of this content is not available or is outdated.

UPD: Thanks for the karma, moved "Information Security"