The mini-detective is NOT a system administrator

Hello!

I'm Ilya. I’m more a programmer and an eternal-in-the-try entrepreneur than a system administrator. But, as you know, in startups you are a mower, a reaper, and a igrets on a dude. So today I had to play all the roles. The article will be able to help the same administrators as beginners, who by the will of fate / service have to deal with the server.

I am writing from the heat, from the heat, some of the teams wrote from memory. I would welcome any adjustments.

So, this morning at 11:35 Minsk time, one of the managers wrote that the websites stopped working. Specifically, dump MySQL errors
')
Since The initial setting up of the server was carried out by the hosting company specialists, I immediately wrote a request to them. But I got a turn from the gate:



Well, maybe this is not customer-oriented, but right. Not every client can help with MySQL. I did not want to use the paid service. We will be.

Indeed, when entering the site appeared unpleasant:

SQLSTATE[HY000]: General error: 1 Can't create/write to file '/tmp/#sql_b80_0.MYI' (Errcode: 28 - No space left on device), 

At first glance, everything seemed very simple - there is not enough space on the disk where MySQL is located. But on the other hand, it is surprising - only yesterday there was 40 GB of free space on that disk.

Connect via SSH, look:

 bakharevich@server:/# df -h Filesystem Size Used Avail Use% Mounted on rootfs 60G 17G 40G 30% / /dev/xvda1 60G 17G 40G 30% / /dev/xvda3 79G 57G 19G 76% /home/site2.by /dev/xvda5 69G 36G 30G 55% /home/site3.by 

Strange. On the disk with the database, 40GB is really free, but MySQL surely does not want to run - there is no space.

A quick search on the Internet gave the result - in addition to disk space, there are also inodes, of which we have just 0.

 bakharevich@server:/# df -i Filesystem Inodes IUsed IFree IUse% Mounted on rootfs 3932160 3932160 0 100% / /dev/xvda1 3932160 3932160 0 100% / /dev/xvda3 5242880 1122900 4119980 22% /home/site2.by /dev/xvda5 4587520 801087 3786433 18% /home/site3.by 

Those. each file, directory and even a symlink is occupied by an inode. Somewhere in the subconscious, I certainly understood that this is the place to be. But the lack of experience in the affairs of the admin / zhelezyachnyh did not let me immediately think about it. Well, we will know! :-) In the meantime, you need to release part of the inode in order to launch sites as soon as possible.

The first thing I decided to remove is sessions from PHP. Yes, people will log in authorization, but it's not so bad, because sites do not work. Our sessions are stored on disk. Therefore:

 rm -rf /var/lib/php5/* 

And also reconfigure their saving to another disk in php.ini :

 session.save_path = "/path/to/new/dir/" 

We look, how many released inode:

 bakharevich@server:/home# df -i Filesystem Inodes IUsed IFree IUse% Mounted on rootfs 3932160 32928660 3500 100% / /dev/xvda1 3932160 3932160 0 100% / /dev/xvda3 5242880 1122900 4119980 22% /home/site2.by /dev/xvda5 4587520 801087 3786433 18% /home/site3.by 

Only 3500 :-( But this may be enough to start MySQL, and therefore temporarily restore the work sites.

Try restarting MySQL:

 bakharevich@server:~$ sudo service mysql start [ ok ] Starting MySQL (Percona Server) database server: mysqld . . .. 

Great, the sites have earned! But the number of inods continues to decrease. After about 10 minutes they will become 0 again.

We continue to delete files. Instinctively, I started with / var / log / i / var / cache / :

 du --max-depth=1 /var/log/ | sort -n -r du --max-depth=1 /var/cache/ | sort -n -r 

Removed excess. It became about another 3.000 inode, but of course this is still very little. And in general, who could create such a number of files to use about 1.5 million inode?



With further search, I saw the folder / var / spool / exim4 / input . Exim4 is the mail server, which, as I understand it, comes with Debian. I tried to count the number of files in the directory, but the result was not shown for a long time. So there really are a lot of files there:

 ls -l /var/spool/exim4/input | wc -l 

A quick search on the Internet showed that this directory is for letters in the queue. Um ... This is interesting because none of our sites use a local mail server. Everything is translated into Yandex.Mail for the Domain and Google. Okay, back to this later. For now you need to free the inodes!

Just do not want to delete files in the directory. We are looking for the correct removal and find something like this:

 exim -bp | exiqgrep -i -f $user | xargs exim -Mrm 

I run. I wait for some time and understand that the inodes are not exempt. I quickly think that this is most likely due to the first Exim -bp command , which first lists the letters in the queue. Once there are more than 1 million letters, then it will take a long time.

Moreover, by this time the free inodes had already ended. Sites stopped again. Managers begin to write, they come one by one SMS about the inoperability of sites, the phone rings ... The situation is heating up. Okay! We'll have to act tough. Delete directly:

 rm -rf /var/spook/exim4/input/* 

But it was not there. Anodes are not released. Whether files are deleted too I cannot look - the volume more than one million does not allow them to be counted quickly. Most likely, rm * also first reads the entire directory, and then starts deleting.

All right, we try more rigid option:

 rm -r /var/spool/exim4/input/ 

... and finally the Ainods are freed by the thousands. I feel that I won a small battle, but the enemy went into the forest for regrouping and another attack in about a month :-) We must step on.

Already more calm and prudent, I started watching exim logs. First, nothing interesting ... as if something like this:

 2015-07-31 13:58:41 1ZIwuU-0007a9-Rm ** joan_galloway@site.by R=dnslookup T=remote_smtp: retry time not reached for any host after a long failure period 2015-07-31 13:58:41 1ZIwuU-0007a9-Rm joan_galloway@site.by: error ignored 2015-07-31 13:58:41 1ZIwuU-0007a9-Rm Completed 

Interesting! As I already wrote, we do not use the local mail server. Moreover, site.by is not ours. We just temporarily host it on our server. What he does no one knows.

I look at the structure of the site:



Something familiar. Let's look at index.php :



Yeah, Joomla! And from 2013. Probably took advantage of the vulnerability of the old versions.

With the great hope that the site is not very popular, I open webserver logs to find something suspicious. Immediately I do a search by email "joan_galloway" , which was found in the Exim logs. But there is nothing ... Looking further. Requests for pictures, stylesheets ... And suddenly:

 50.62.177.108 - - [31/Jul/2015:14:37:41 +0300] "POST /components/com_weblinks/views/categories/system.php HTTP/1.1" 404 2056 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)" 

Very interesting! Some file system.php . There was a POST request, so the request parameters are not visible in the address. The name is very suspicious. Once in my childhood I also had such a directory on my home computer :-)

See the file itself:



Clear! Enemy found. We change the name of the file and silence comes again in the Exim logs. Finally, you can touch the food, which the future wife put about 10 minutes ago :-)

Next will be the interrogation of the prisoner:

- how did you get to the server?
- Are there any accomplices on other sites / directories?

And also you need to understand on our territory:

- check permissions to folders
- except for automatic check of free space, set up a free inode check.
- configure Exim (for example, block SMTP).

Probably something else is needed. I will be glad to comments / corrections / stories from your practice.

I hope someone will find this article useful.

Thanks for attention!