Unusual hard disk overflow or how to delete millions of files from one folder

Foreword

Most likely, the article will not be very interesting for experienced system administrators. First of all, it focuses on newbies, as well as on people who have encountered a similar problem - the need to delete a huge number of files from one folder on Linux (Debian in my case), as well as ended disk space when df -h issues which is almost 30% free.

Start

Nothing foreshadowed trouble.
The server has been working without any problems for more than a year (uptime almost 500 days), there were no problems, and I calmly went on vacation with a pure soul.

On the first day of vacation, they call me with a complaint - the site is unavailable. MySQL crashes with Error 28 “No space left on device”.

It would seem that the problem is banal - the disk space is over. True, df -h shows that the disk has a sufficient amount of free space, well, I'm on vacation, I’m too lazy to sort it out - I advised them to look for unnecessary files on the disk (old backups, etc.) and delete them. Removed, it seems everything worked.
')
A couple of hours passed and the problem returned. Strange - the free space on the hard disk during this time almost did not decrease. After a quick googling, a topic appeared on the serverfault , which states that the problem may also arise due to the fact that it was not disk space, but inodes !

I enter df -i into the console - and it turns out really that I have ended my inodes.

Problem

I started looking for where I have so many files on my hard drive that they have devoured all the inodes (and I have over 30 million inins on a 500-gigabyte hard drive).

And I found it - it turned out that the problem was in the folder with the php sessions .

Apparently, for some reason, the auto-cleaning mechanism of this folder broke, which led to the accumulation of a huge number of files in it. How huge is difficult to say, because no standard Linux commands, such as ls, find, rm, etc. - do not work with this folder. Just hang, at the same time hanging the entire server. I can only say that the directory file itself weighs about a gigabyte, and also that there are exactly more than half a million files there, because I have already deleted so much from there.

Decision

The solution is obvious - you need to delete all these session files. In this case, it is desirable that the server continued to work in normal mode. For starters, I renamed the sessions folder, which contains a bunch of files, and instead I created an empty one - so that I can delete all files from the old (renamed) file and that would not prevent the creation of new session files.

Also in CZK added automatic deletion of session files older than one hour, so that the problem does not recur.

And went to the main problem - cleaning the hard disk.

I tried the solution "in the forehead":

 rm -rf ./* 

The server hung, nothing was removed.

I tried a known method to delete a large number of files.

 find . -type f -exec rm -v {} \; 

Nothing, the server hangs, the files are not deleted.

And now that the most interesting thing is that the file manager mc has successfully coped with the task of deleting these files! That is, when you start deleting a folder - the files are deleted, mc does not hang. Deletion goes at a speed of about 5,000 files per minute, although it creates a huge load on the hard disk, which leads to server inoperability.

And I would like these files to be gradually deleted in the background, and do not interfere with the normal operation of the site.

Actually, the solution was again found in Google - Olark shares the way it displayed a list of 8 million files in 1 folder, using the getdents system call

Here is the documentation for the getdents function, as well as sample code that uses it.

However, this example did not quite fit me - even if I put a large buffer size, as Olark advises in my blog, the server still hangs when trying to read the entire folder at once.

Experimentally, I selected a buffer size of 30 kilobytes, which allows you to read about 550 file names from a directory, while not suspending the server and not creating unnecessary load on the disk. He also rewrote the sample code a bit so that instead of displaying the file name, he deleted it.

As a result, I got this code:

 #define _GNU_SOURCE #include <dirent.h> /* Defines DT_* constants */ #include <fcntl.h> #include <stdio.h> #include <unistd.h> #include <stdlib.h> #include <sys/stat.h> #include <sys/syscall.h> #define handle_error(msg) \ do { perror(msg); exit(EXIT_FAILURE); } while (0) struct linux_dirent { long d_ino; off_t d_off; unsigned short d_reclen; char d_name[]; }; #define BUF_SIZE 1024*30 int main(int argc, char *argv[]) { int fd, nread; char buf[BUF_SIZE]; struct linux_dirent *d; int bpos; int deleted; char d_type; char temp[100]; fd = open(argc > 1 ? argv[1] : ".", O_RDONLY | O_DIRECTORY); if (fd == -1) handle_error("open"); deleted = 0; nread = syscall(SYS_getdents, fd, buf, BUF_SIZE); if (nread == -1) handle_error("getdents"); if (nread != 0) { for (bpos = 0; bpos < nread;) { d = (struct linux_dirent *) (buf + bpos); d_type = *(buf + bpos + d->d_reclen - 1); if(d->d_ino && d->d_ino != 22332748 && d->d_ino != 22332761) { //    inode      ,       "."  ".." -  ,     sprintf(temp,"%s/%s", argv[1], (char *) d->d_name); remove(temp); deleted += 1; } bpos += d->d_reclen; } } printf("deleted %d\n", deleted); exit(EXIT_SUCCESS); } 

The code is compiled by the usual gcc

 gcc listdir.c -o listdir 

And just run from the command line:

 ./listdir mod-tmp2 

I put the resulting file in crowns and now I have 547 files per minute deleted, while the server load is within the normal range - and I hope that within a week or two all the files will be deleted.

findings

1. If df -h indicates that there is still room on the hard disk - it may not be there. We must also look df -i
2. You shouldn’t hope for auto-cleaning mechanisms for such things as session files - at some point they may not work and you’ll end up with a whole mountain of files that you’ll remove is not a trivial task.
3. Standard Linux commands such as ls, rm, find , etc. can pass in front of non-standard situations like millions of files in one folder. In this case, use low-level system calls.