Fail2ban [incremental]: Better, faster, more reliable

About fail2ban has already been written a lot, including on the habr. This article is a little about something else - how to make it even safer to protect and about fail2ban still unknown in a wide range of new functions. I will add right away - for the time being we will talk about the development branch, although it has been tested in combat for a long time.

Brief introduction

In the majority, fail2ban is installed from the distribution kit (as a rule, it is some kind of stable old version) and can be configured from the Internet in a few minutes. Then it works for years without the intervention of the administrator. Often, even logs that fail2ban seems to follow are not viewed.
So, I was forced to write this post by a case that happened to one server of my good friend. Classics of the genre - an abuse came, and the second followed it. Well, the attacker was still lazy - he did not lose the logs, and he was lucky that logrotate was set up to keep logs for months.

It turned out to be pretty trite, they picked up a password for his admin mail, which was also a password for ssh (of course, without a key). Not a root, but a vessel, with all that it implies. His first question was: how to pick up - I have the same fail2ban there. And here is just an ambush: not everyone imagines that it is not individual computers that are now engaged in the selection of passwords, but entire bot networks, by the way, they are wiser. So, according to the logs, we found out that there is just such a case: I went through the botnet, and in practice, I found out its settings in fail2ban (maxRetry = 5, findTime = 600 and banTime = 600). Those. To avoid a ban, the network made 4 attempts within 10 minutes from each IP. For a minute, the network has about 10 thousand unique IP = something more than 5 and a half million passwords per day.
In addition, his mailer did a great folly - namely, a pause of up to 10 seconds, with a login with the wrong name. Those. find out that some names, including the admin, really exist, this grid was not difficult. Then there was a targeted search for only passwords for existing names.
I will not dwell on “repair” in more detail - this is a long story, and in general a topic for a separate article. I can only say that everything was cleaned and everything was resolved with a little blood, and he got off with almost “slight” fright.

So, the idea to write an article arose after I (partially deserved) were expressed: “So you knew about this and did not say anything, did not warn you. Yes, and the solution is there and not shared. Well, you bastard. In short, therefore, the post - to be.
')

My fail2ban

I take security of “my own” servers extremely seriously. In addition to the same fail2ban, always very custom, I have monitoring there and a whole lot more. It just really pisses me off that because of the stupid gray mass, which allows you to take control of your botnets with your hardware, you have to kill a lot of time for protection (and constant maintenance and control in the future). By the way, to minimize this control, I actively participate in development and fail2ban, and other projects from security.

So, my latest enhanced version [sebres: ban-time-incr] allows you to display this annoying zoo once and for all (well, or until they adapt again). This feature was often discussed by all the members of the community, but somehow they did not reach. I lived in the form of separate scripts and some custom changes, until it was formed into a ready-made functional.

In short, the system, remembering bad IP addresses, allows each time to dynamically (exponentially) increase the blocking time (banTime) depending on the number of previous bans (banCount). In addition, each time reducing the number (maxRetry) of possible failures (failure) to the next ban. It can be clearly seen in the following example:
It is clearly seen here, as each following ban prolongs the blocking time from 15 minutes (0:15:00) for the first time, up to more than 5 days (5 days, 8:04:55) after the tenth blocking. I have an IP in the database that has a “deadline” already - from several months to a permanent ban.

Below you can see how the new functionality is reflected in the decision to actually ban IP XXX.XXX.XX.XXX. In the example, the maxRetry parameter is set to 5. So we see that while the IP is not considered bad, it was first banned after 5 attempts, the second time, already as bad - after 3 (each attempt was counted as 2), the third etc. - after 2 (the attempt goes for 3) and is banned for the fourth time right after the first attempt (counted immediately after 5 minutes):

Without this logic of counting failure, smart botnets have learned how to fine-tune their work so as not to get banned. When I finished this logic and rolled out into production, in a matter of days I got rid of almost all the evil that I used to see in my logs for years. For example, now the average normal daily increase of my auth.log is somewhere around 20-50 lines, earlier on some servers it was hundreds and thousands of times more.

So far this is a development branch, there is a pull request, the release is planned for now in version 0.9.2.
Who cares, read more about the implementation and solution history here - Ban time incr by sebres · Pull Request # 716 · fail2ban / fail2ban .

However, while this version will be updated to your server, a lot of time will pass - until the release is released on the mainline, while it is taken to the distributions ... The story is long, for example, the same debian still uses 0.8.x - that's why the article itself. So shake hands, set ... profit.

You can get this version here fail2ban-ban-time-incr.zip (sebres master branch).
Port for debian-s: ban-time-incr-debian.zip (merged master debian branch, and although not the main line - I will try to keep it up-to-date if possible).

Installing it is quite simple. If you already had fail2ban installed from the distribution, save the old “fail2ban.local” and “jail.local” from “/ etc / fail2ban /” (Well, the old ones are better than the “fail2ban.conf” and “jail.conf” ). I would just in case (due to possible personal changes in filter and action) save the entire directory "/ etc / fail2ban /" somewhere.

Next, we demolish the old distributive fail2ban, for example:

 sudo service fail2ban stop sudo apt-get remove fail2ban 

Actually installation:

 cd /tmp unzip ~/downloads/fail2ban-ban-time-incr.zip cd fail2ban-ban-time-incr/ sudo python setup.py install 

[UPD] Sometimes on some distributions, with a manual installation, for some reason the service is not installed (for example, there is no /etc/init.d/fail2ban file) - and accordingly the service does not (auto) start, only through fail2ban-client start .[/ UPD]

Now, in order for the new functionality to work, you need to add an option in your jail.local in [default] (or for each specific jail): bantime.increment = true . An example and description can still be found in " jail.conf ".
Some here are brief:

• bantime.rndtime is the maximum time used to add random time to banTime to prevent smart botnets from calculating the exact time when the IP is unlocked again. Example bantime.rndtime = 10m
• bantime.factor - coefficient for calculating the growth exponent for the formula bantime.formula or multipliers bantime.multipliers , the default value of the coefficient is 1, which corresponds to an increase in the ban time for 1, 2, 4, 8, 16 ... By increasing this parameter for some jail, increase blocking time more aggressively.
• bantime.formula - used by default to calculate the next value of the ban time, the default value: bantime.formula = ban.Time * (1<<(ban.Count if ban.Count<20 else 20)) * banFactor
  The same increase in the ban time will be achieved using the factors bantime.multipliers equal to 1, 2, 4, 8, 16, 32 ...
  An example of a more aggressive formula for factor "1" and has the same growth values ​​only for a factor equal to "2.0 / 2.885385":
  bantime.formula = ban.Time * math.exp(float(ban.Count+1)*banFactor)/math.exp(1*banFactor)
  
• bantime.multipliers - the parameter can be used instead of the formula discretely to calculate the next value of the time of the ban. The value of the multiplier is "-1" (can only be at the end of the list) and puts the address in a permanent ban (before manual unlocking).
  Example 1: bantime.multipliers = 1 2 4 8 16 32 64 - increases the ban time by 1, 2, 4, ... and if the last ban count was more than the last multiplier index, the last multiplier will always be used (64 in the example), which with a factor equal to "1" and the original time of the ban (10 minutes) - corresponds to 10.6 hours.
  Example 2: bantime.multipliers = 1 5 30 60 300 720 1440 2880 - can be used for a small initial ban time (bantime = 60) - because the increase becomes more aggressive, we have bantime equal: 1 min, 5 min, 30 min, 1 hour, 5 hours, 12 hours, 1 day, 2 days, respectively.

If during the trial or in production any (good) IP accidentally repeatedly flew to the ban (and became correspondingly bad), it will be forgotten (it will become “white” again) itself after the three times last ban and dbpurgeage (located in fail2ban.local) , or if with his hands to remove the ban, using:

 fail2ban-client set $JAIL unbanip $IP 

I use fail2ban-regex for testing regulars, and for the performance test something like this:

 logger -t 'test:auth' -i -p auth.info "pam_unix(test:auth): authentication failure; logname= uid=0 euid=0 tty=test ruser=admin rhost=1.2.3.4" 

Do not forget about the start:

 sudo service fail2ban start 

That's all, now I hope your server has become a little more secure. Well, you do not be lazy and look all the same in the logs (trust, but verify).

PS Standard postscript: WITHOUT ANY WARRANTY. In short - use on health, but at your own peril and risk ...
And may your servers be safe.

PPS Yes, I almost forgot, I have something planned here to finish something, something is already ready, you just need to arrange it normally and post it, and so - the poll “What do you have to do (before) in the first place”.