Effective anti-spam technology.
Hardly anyone else knows what spam is and has no desire to get rid of it. Those who at the same time are the "owners" of the corporate mail server finally have a less effective remedy for spammers.
This is a technology gray listing (gray lists). To put it simply, the essence of the technology lies in checking the correspondence of the host servers of the opposite server, as well as in controlling the speed of transmission of letters. The sender's mail server should try to send an email later when receiving 4xx errors. Spammers, on the other hand, usually ignore this error and try to send the next letter from another sender to another address in the mail domain. As a result, all their attempts that do not meet this condition are discarded.
Those who wish to understand all the details of the technology I send to search engines, which will provide links to comprehensive information, since a complete and detailed description will take an unreasonably large amount of space.
')
Policyd.
Each technology has many implementations from the developers. No exception and gray lists technology. I want to talk about one of the most successful implementations - policyd. This program is most easily installed along with the popular postfix mail service and, as will be seen below, it connects in just a couple of seconds.
The essence of work policyd is to maintain lists of connections from the outside world with the mail server. During the initial attempt to deliver any letter to our system, the connection is terminated at the very beginning, and the sending system is notified that the mail system is temporarily unavailable and is offered to try to deliver the message later. At the same time, the information about the sender enters the MySQL database. The address of the sender, recipient and ip of the sending system is recorded in the database. In the Policyd system, such entries are called “triplets”.
If, after a specified period of time (or later), the sender tries to deliver the message again, then policyd, checking and making sure that the first connection is available, will allow Postfix to accept the message and take the necessary actions to deliver the letter to the user's mailbox.
All subsequent similar messages from this sender to the user of our system will be delivered without delay, and the counter of successful deliveries will increase incrementally. When the number of successful deliveries (those same triplets) from the sender system to ours passes the line of 500 connections from one domain to another (this value can also be adjusted in the config), the sending system’s ip will be whitelisted and will not be filtered later.
If the repeated message reprint does not take place, the following scenario will take effect. The number of failed connections with the ip system will also be counted when triplets with a zero value become more than 500 - the system ip will also be included in the lists. But this time in black. After that, all connections from this address will be terminated immediately, giving an error of 550, which will certainly save resources and traffic of the postal system.
I think you already understand the essence of technology. As you know, most of the spam software does not handle the send error and does not re-send. This is both a strong and a weak side of the gray lists. The fact is that spam messages sent from full-fledged systems that establish full-fledged smtp connections and process them according to all the rules will be delivered in the same way as legitimate messages. The reasons for this are completely transparent - Policyd does not analyze message content. It controls only the connection and either allows it or breaks it.
However, no one bothers to put a spamassassin or dspam on the second front of defense (which will be better). That is, any system that works on message signatures (the so-called bayes).
Install Policyd.
As mentioned above, it is not difficult to establish a policyd even for a not very advanced mail system administrator. At the address of the project [1] you can always download the source of the most recent version of the system (at the moment, this is version 1.80). The size of the archive is very small and is only 63 kilobytes. Users of the gentoo system can also find ebuilds at [2]. I guess in a short time the policyd package will fall into the main portage tree.
The build does not require any prior configuration, and the output of the make command contains the necessary explanations.
Initially run
make build
and
make install
for proper installation. By default, the program will be installed in / usr / local / policyd. This directory will contain 4 files. Where:
cleanup - a program for cleaning outdated information from the database;
policyd is the Policyd service itself;
policyd.conf - configuration file;
stats (in gentoo, this command is called policyd_stats) is a program to display policyd operation statistics.
Since Policyd works with the MySQL database, it requires installation and issuance of the necessary rights. The easiest way to install a database is from the file scheme included in the archive. This is done by a simple command:
root: /usr/src/policyd-v1.80 / # mysql -p <DATABASE.mysql
After that, it remains only to log into mysql and issue permissions to the user for Policyd:
Now we can add a policyd call to the Postfix configuration file. This is done in main.cf as follows. In line:
smtpd_recipient_restrictions =
we add an appeal to our gray list service
check_policy_service inet: 127.0.0.1: 10031
Of course, this is the default string for accessing the Policyd service. If you want to assign a different address or port, then here you must specify your parameters.
In order for cleanup triplets and lists to work cleanup, you need to add the following line to crontab:
Alternatively, place a script similar in functionality to bash in the /etc/cron.hourly directory
In the case of the gentoo package, the installation of the necessary script in /etc/cron.hourly occurs automatically.
After this installation Policyd can be considered complete.
Configure Policyd.
All policyd configuration is done in the /usr/local/policyd/policyd.conf file. In this article, I pay attention primarily to the work with gray lists, but the program itself has several more settings and features. If information about them will be in demand, perhaps I will write another additional article on how to use them most effectively. In the meantime, pay attention to the most important parameters of the configuration file. In general, it should be noted that the configuration file is provided with fairly understandable comments and figure out what will be quite simple if you possess minimal knowledge of English technical language.
First of all, these are the settings for access to the MySQL database. The variables MYSQLHOST, MYSQLDBASE, MYSQLUSER and MYSQLPASS have such “speaking” names that I do not think it is worth explaining their meaning.
FAILSAFE allows (or does not allow) policyd to skip emails when there is no access to its database. For me, this parameter is not relevant, because if MySQL does not work, then Postfix will not know where to put the letters.
The following options are interesting in the DAEMON CONFIG section:
DEBUG = 3 indicates that, for a start, we would like to see very detailed information on how Policyd works.
DAEMON = 0 says that policyd will run as an application, not as a service. Subsequently, we will change this parameter to 1.
BINDHOST and BINDPORT indicate which address and port will be accessing Policyd.
SYSLOG_FACILITY = “LOG_MAIL | LOG_INFO ”says that information about the operation of the service will be reflected via syslog in the same file (in any case, on most systems), where the work of the postal system itself is.
CONN_ACL indicates which network addresses are allowed to access the service.
Next come the two most important sections of the configuration file for us. This is WHITELISTING and BLACKLISTING. The variables in them are identical except, of course, the names. WHITE in one case and BLACK in the other.
WHITELISTING = 1 and BLACKLISTING = 1 respectively mean the inclusion of black and white lists.
AUTO_WHITELIST_NUMBER = 500 and AUTO_BLACKLIST_NUMBER = 500 means that after accumulating 500 successful (unsuccessful) triplets, the host will be listed in the white (black) list.
AUTO_WHITELIST_EXPIRE = 7d and AUTO_BLACKLIST_EXPIRE = 7d indicate that the aging time of the information in the lists is 7 days old. Once every seven days, hosts will be removed from both lists. This variable is questionable by some. Not everyone thinks the sending system can become normal. But for the full automation of the work of Policyd, I would recommend to leave these parameters on.
The BLACKLIST_REJECTION option is only in the blacklist description section and contains a message that will be issued to systems on the list when trying to deliver a message from them.
Now let's pay attention to the most important section, which contains the settings of the gray list system itself (GREYLISTING).
GREYLISTING = 1 says that we want to use gray lists.
GREYLIST_REJECTION, by analogy with BLACKLIST_REJECTION, contains a string that will be sent to the sending system when the message is initially sent.
GREYLIST_X_HEADER = 1 indicates whether to put x-header in every message that successfully passed through the gray lists.
TRAINING_MODE = 0 is required at the testing stage. I missed this option and did not use it. If the variable is set to 1 (that is, enabled), then Policyd works in test mode when it displays the results of its work on the screen (or in the log file), but does not prevent Postfix from receiving messages (that is, it does not actually participate in the mail operation) .
TRIPLET_TIME indicates in how many minutes when the message is retried (from the time of the first attempt) the message will be received and delivered.
TRIPLET_AUTH_TIMEOUT = 30d means that information about successful triplets (delivered) will be stored for 30 days. This parameter is necessary so that all successful triplets are not stored forever. For example, if there were only two messages from someone delivered and the correspondence with that person ended. In this way, we make a modest contribution to minimizing useless garbage in the database.
TRIPLET_UNAUTH_TIMEOUT = 2d by analogy means that information about unsuccessful triplets (undelivered) will be stored for 2 days.
At this parameters, which I would like to draw attention to, end. Now you can safely run Policyd and Postfix and see what happens when you try to deliver messages to our system.
After you make sure that everything works correctly and fall in love with the detailed logs - do not forget to set the DAEMON parameter to 1, and the logging level to less detailed.
Conclusion
Honestly, I don’t know about the problems that could arise when installing Policyd, because there’s no place to be wrong. From myself I want to note that after installing the gray lists on the corporate server of the company, the load decreased by several orders of magnitude. There were significantly fewer calls to spamassassin (and this software is known for its “speed” achieved through the use of perl), and the pressing question of upgrading the server unexpectedly faded into the background. In addition, due to the fact that viruses are sent the same way as spam - the system has been protected from a huge number of viruses, which often do not have time to recognize the mail antivirus. (Especially, given the quality of the work of the Kaspersky anti-virus laboratory, whose product we have. But this is not a separate story for this article.)
In order to make it clearer how effective Policyd works, I can say that if earlier during the time from Friday night to Monday morning I received more than 500-600 spam letters, now the number has decreased to quite acceptable 15-20.
Here is the output of the stats command from my system, after a few days of work:
Greylisting: Triplet information
->
Triplets: -> 42523
Verified: -> 1811
Unverified: -> 40712
Comments, I believe, are superfluous.
By the way, stats runs as:
Moreover, you should pay attention to the fact that if Policyd is working in daemon mode and logging via syslog, then the output of the stats command will be in the log file.
Good luck with spamming.
[1] - policyd.sourceforge.net
[2] - bugs.gentoo.org/show_bug.cgi?id=112261
(c) akeeper Korshunov Alexey.
The first published in the electronic supplement to the journal "System Administrator" called OSA .
This is a technology gray listing (gray lists). To put it simply, the essence of the technology lies in checking the correspondence of the host servers of the opposite server, as well as in controlling the speed of transmission of letters. The sender's mail server should try to send an email later when receiving 4xx errors. Spammers, on the other hand, usually ignore this error and try to send the next letter from another sender to another address in the mail domain. As a result, all their attempts that do not meet this condition are discarded.
Those who wish to understand all the details of the technology I send to search engines, which will provide links to comprehensive information, since a complete and detailed description will take an unreasonably large amount of space.
')
Policyd.
Each technology has many implementations from the developers. No exception and gray lists technology. I want to talk about one of the most successful implementations - policyd. This program is most easily installed along with the popular postfix mail service and, as will be seen below, it connects in just a couple of seconds.
The essence of work policyd is to maintain lists of connections from the outside world with the mail server. During the initial attempt to deliver any letter to our system, the connection is terminated at the very beginning, and the sending system is notified that the mail system is temporarily unavailable and is offered to try to deliver the message later. At the same time, the information about the sender enters the MySQL database. The address of the sender, recipient and ip of the sending system is recorded in the database. In the Policyd system, such entries are called “triplets”.
If, after a specified period of time (or later), the sender tries to deliver the message again, then policyd, checking and making sure that the first connection is available, will allow Postfix to accept the message and take the necessary actions to deliver the letter to the user's mailbox.
All subsequent similar messages from this sender to the user of our system will be delivered without delay, and the counter of successful deliveries will increase incrementally. When the number of successful deliveries (those same triplets) from the sender system to ours passes the line of 500 connections from one domain to another (this value can also be adjusted in the config), the sending system’s ip will be whitelisted and will not be filtered later.
If the repeated message reprint does not take place, the following scenario will take effect. The number of failed connections with the ip system will also be counted when triplets with a zero value become more than 500 - the system ip will also be included in the lists. But this time in black. After that, all connections from this address will be terminated immediately, giving an error of 550, which will certainly save resources and traffic of the postal system.
I think you already understand the essence of technology. As you know, most of the spam software does not handle the send error and does not re-send. This is both a strong and a weak side of the gray lists. The fact is that spam messages sent from full-fledged systems that establish full-fledged smtp connections and process them according to all the rules will be delivered in the same way as legitimate messages. The reasons for this are completely transparent - Policyd does not analyze message content. It controls only the connection and either allows it or breaks it.
However, no one bothers to put a spamassassin or dspam on the second front of defense (which will be better). That is, any system that works on message signatures (the so-called bayes).
Install Policyd.
As mentioned above, it is not difficult to establish a policyd even for a not very advanced mail system administrator. At the address of the project [1] you can always download the source of the most recent version of the system (at the moment, this is version 1.80). The size of the archive is very small and is only 63 kilobytes. Users of the gentoo system can also find ebuilds at [2]. I guess in a short time the policyd package will fall into the main portage tree.
The build does not require any prior configuration, and the output of the make command contains the necessary explanations.
root: /usr/src/policyd-v1.80 / # make
Possible options are:
make build
make install | install strip
make upgrade
make clean
Initially run
make build
and
make install
for proper installation. By default, the program will be installed in / usr / local / policyd. This directory will contain 4 files. Where:
cleanup - a program for cleaning outdated information from the database;
policyd is the Policyd service itself;
policyd.conf - configuration file;
stats (in gentoo, this command is called policyd_stats) is a program to display policyd operation statistics.
Since Policyd works with the MySQL database, it requires installation and issuance of the necessary rights. The easiest way to install a database is from the file scheme included in the archive. This is done by a simple command:
root: /usr/src/policyd-v1.80 / # mysql -p <DATABASE.mysql
After that, it remains only to log into mysql and issue permissions to the user for Policyd:
mysql grant all on policyd. * to postfix @ localhost identified by 'somepassword';
Now we can add a policyd call to the Postfix configuration file. This is done in main.cf as follows. In line:
smtpd_recipient_restrictions =
we add an appeal to our gray list service
check_policy_service inet: 127.0.0.1: 10031
Of course, this is the default string for accessing the Policyd service. If you want to assign a different address or port, then here you must specify your parameters.
In order for cleanup triplets and lists to work cleanup, you need to add the following line to crontab:
0 * * * * / usr / local / policyd / cleanup -c /usr/local/policyd/policyd.conf
Alternatively, place a script similar in functionality to bash in the /etc/cron.hourly directory
In the case of the gentoo package, the installation of the necessary script in /etc/cron.hourly occurs automatically.
After this installation Policyd can be considered complete.
Configure Policyd.
All policyd configuration is done in the /usr/local/policyd/policyd.conf file. In this article, I pay attention primarily to the work with gray lists, but the program itself has several more settings and features. If information about them will be in demand, perhaps I will write another additional article on how to use them most effectively. In the meantime, pay attention to the most important parameters of the configuration file. In general, it should be noted that the configuration file is provided with fairly understandable comments and figure out what will be quite simple if you possess minimal knowledge of English technical language.
First of all, these are the settings for access to the MySQL database. The variables MYSQLHOST, MYSQLDBASE, MYSQLUSER and MYSQLPASS have such “speaking” names that I do not think it is worth explaining their meaning.
FAILSAFE allows (or does not allow) policyd to skip emails when there is no access to its database. For me, this parameter is not relevant, because if MySQL does not work, then Postfix will not know where to put the letters.
The following options are interesting in the DAEMON CONFIG section:
DEBUG = 3 indicates that, for a start, we would like to see very detailed information on how Policyd works.
DAEMON = 0 says that policyd will run as an application, not as a service. Subsequently, we will change this parameter to 1.
BINDHOST and BINDPORT indicate which address and port will be accessing Policyd.
SYSLOG_FACILITY = “LOG_MAIL | LOG_INFO ”says that information about the operation of the service will be reflected via syslog in the same file (in any case, on most systems), where the work of the postal system itself is.
CONN_ACL indicates which network addresses are allowed to access the service.
Next come the two most important sections of the configuration file for us. This is WHITELISTING and BLACKLISTING. The variables in them are identical except, of course, the names. WHITE in one case and BLACK in the other.
WHITELISTING = 1 and BLACKLISTING = 1 respectively mean the inclusion of black and white lists.
AUTO_WHITELIST_NUMBER = 500 and AUTO_BLACKLIST_NUMBER = 500 means that after accumulating 500 successful (unsuccessful) triplets, the host will be listed in the white (black) list.
AUTO_WHITELIST_EXPIRE = 7d and AUTO_BLACKLIST_EXPIRE = 7d indicate that the aging time of the information in the lists is 7 days old. Once every seven days, hosts will be removed from both lists. This variable is questionable by some. Not everyone thinks the sending system can become normal. But for the full automation of the work of Policyd, I would recommend to leave these parameters on.
The BLACKLIST_REJECTION option is only in the blacklist description section and contains a message that will be issued to systems on the list when trying to deliver a message from them.
Now let's pay attention to the most important section, which contains the settings of the gray list system itself (GREYLISTING).
GREYLISTING = 1 says that we want to use gray lists.
GREYLIST_REJECTION, by analogy with BLACKLIST_REJECTION, contains a string that will be sent to the sending system when the message is initially sent.
GREYLIST_X_HEADER = 1 indicates whether to put x-header in every message that successfully passed through the gray lists.
TRAINING_MODE = 0 is required at the testing stage. I missed this option and did not use it. If the variable is set to 1 (that is, enabled), then Policyd works in test mode when it displays the results of its work on the screen (or in the log file), but does not prevent Postfix from receiving messages (that is, it does not actually participate in the mail operation) .
TRIPLET_TIME indicates in how many minutes when the message is retried (from the time of the first attempt) the message will be received and delivered.
TRIPLET_AUTH_TIMEOUT = 30d means that information about successful triplets (delivered) will be stored for 30 days. This parameter is necessary so that all successful triplets are not stored forever. For example, if there were only two messages from someone delivered and the correspondence with that person ended. In this way, we make a modest contribution to minimizing useless garbage in the database.
TRIPLET_UNAUTH_TIMEOUT = 2d by analogy means that information about unsuccessful triplets (undelivered) will be stored for 2 days.
At this parameters, which I would like to draw attention to, end. Now you can safely run Policyd and Postfix and see what happens when you try to deliver messages to our system.
After you make sure that everything works correctly and fall in love with the detailed logs - do not forget to set the DAEMON parameter to 1, and the logging level to less detailed.
Conclusion
Honestly, I don’t know about the problems that could arise when installing Policyd, because there’s no place to be wrong. From myself I want to note that after installing the gray lists on the corporate server of the company, the load decreased by several orders of magnitude. There were significantly fewer calls to spamassassin (and this software is known for its “speed” achieved through the use of perl), and the pressing question of upgrading the server unexpectedly faded into the background. In addition, due to the fact that viruses are sent the same way as spam - the system has been protected from a huge number of viruses, which often do not have time to recognize the mail antivirus. (Especially, given the quality of the work of the Kaspersky anti-virus laboratory, whose product we have. But this is not a separate story for this article.)
In order to make it clearer how effective Policyd works, I can say that if earlier during the time from Friday night to Monday morning I received more than 500-600 spam letters, now the number has decreased to quite acceptable 15-20.
Here is the output of the stats command from my system, after a few days of work:
Greylisting: Triplet information
->
Triplets: -> 42523
Verified: -> 1811
Unverified: -> 40712
Comments, I believe, are superfluous.
By the way, stats runs as:
cd / usr / local / policyd
./stats –c policyd.conf
Moreover, you should pay attention to the fact that if Policyd is working in daemon mode and logging via syslog, then the output of the stats command will be in the log file.
Good luck with spamming.
[1] - policyd.sourceforge.net
[2] - bugs.gentoo.org/show_bug.cgi?id=112261
(c) akeeper Korshunov Alexey.
The first published in the electronic supplement to the journal "System Administrator" called OSA .
Source: https://habr.com/ru/post/5399/
All Articles