The best way to store a large number of files on the server

As you know, you should not store a large number of files in one folder because very quickly, a crash can occur in the system or simply the files will be read very slowly.

To solve this problem, many programmers take the md5 file name f789f789abc898d6892df98d09a8f8, and then split the name like this:

/f7/89/f789abc898d6892df98d09a8f8.jpg

The math here is very simple - one character is 16 options.
Thus, 2 characters is already 16 * 16 = 256 variants.
In our case, we have 2 nestings of 2 characters each, so the maximum number of folders will be 256 * 256 = 65536 folders.
If we need to save 1,000,000 files, the number of files in each folder will not exceed 1,000,000 / 65536 = 15 files.
')
Yes, the option is simple, but what if we need not only to save files well, but also to quickly find them?

For example, we have a social network and we want for each user to create a separate folder with its id number and store files in it, which in turn also have their own id.
And for us it is important not only to save the file but also to quickly find where it lies on its id.

To solve this problem, I wrote a class that allows you to save a large number of files or folders on the server in a tree-like folder structure.

Here is the structure the class creates:


To calculate the maximum number of files that fit in this structure, you need to raise the maximum number of files in a folder to the power of the number of branches plus one.

In the image we see 2 branches and 3 files in each folder.
Thus 3 needs to be raised to the power of 2 + 1 = 3 * 3 * 3 = 27 files.

To save no more than 1,000,000 files in such a structure, we need 2 branches of 100 files in each folder (100 * 100 * 100).

You need to pass an array of parameters to the class - the path to the folder where the tree will be built, the maximum number of files in the folder, the number of branches, or you can apply a pattern (parameter) to the maximum number of files that have already been calculated in advance - bigint, int, mediumint, smallint:

array ('upload_dir' => Q_PATH. '/ uploads /', 'max_file_count' => 1000, 'branches' => 2, 'pattern' => '')

The class itself:

<?php //file index define("Q_PATH",dirname(__FILE__)); // class Functions { public static function arr_union(array $def_arr,array $new_arr) { foreach($new_arr as $key => $value) { if(array_key_exists($key, $def_arr) && is_array($value)) { $def_arr[$key]=self::arr_union($def_arr[$key], $new_arr[$key]); } else { $def_arr[$key]=$value; } } return $def_arr; } } /** *    */ class Upload { public $id; private $upload_dir; private $max_file_count; private $branches; public function __construct(array $param=array()) { $def_param=array('upload_dir'=>Q_PATH.'/uploads/','max_file_count'=>1000,'branches'=>2,'pattern'=>''); $upload_param=Functions::arr_union($def_param,$param); $this->upload_dir=$upload_param['upload_dir']; $this->max_file_count=$upload_param['max_file_count']; $this->branches=$upload_param['branches']; // ,     df -i  tune2fs -l /dev/hda1  df -Ti switch($upload_param['pattern']) { case 'bigint': $this->max_file_count=512; $this->branches=6; break; case 'int': $this->max_file_count=216; $this->branches=3; break; case 'mediumint': $this->max_file_count=204; $this->branches=2; break; case 'smallint': $this->max_file_count=182; $this->branches=1; break; } $this->del_id(); } public function set_id($id) { $this->id=$id; } public function del_id() { $this->id=0; } public function find_upload($url) { if(is_file($url)) { return true; } else { return false; } } public function get_upload($id,$fl) { $this->set_id($id); for($i=$this->branches;$i>=1;$i--) { $dir=ceil($this->id/pow($this->max_file_count,$i))%$this->max_file_count; $dir_file_arr[]=$dir>0?$dir:$this->max_file_count; } $dir_file_str=implode("/", $dir_file_arr); return $this->upload_dir.$dir_file_str.'/'.$this->id.$fl; } public function put_upload($id,$fl,$data) { $this->set_id($id); for($i=$this->branches;$i>=1;$i--) { $dir=ceil($this->id/pow($this->max_file_count,$i))%$this->max_file_count; $dir_file_arr[]=$dir>0?$dir:$this->max_file_count; $dir_file_str=implode("/", $dir_file_arr); if(!is_dir($this->upload_dir.$dir_file_str)) { mkdir($this->upload_dir.$dir_file_str, 0777); //chmod($this->upload_dir.$dir_file_str, 0777); } } file_put_contents($this->upload_dir.$dir_file_str.'/'.$this->id.$fl, $data); return $this->upload_dir.$dir_file_str.'/'.$this->id.$fl; } public function set_upload($id,$fl) { $this->set_id($id); for($i=$this->branches;$i>=1;$i--) { $dir=ceil($this->id/pow($this->max_file_count,$i))%$this->max_file_count; $dir_file_arr[]=$dir>0?$dir:$this->max_file_count; $dir_file_str=implode("/", $dir_file_arr); if(!is_dir($this->upload_dir.$dir_file_str)) { mkdir($this->upload_dir.$dir_file_str, 0777); //chmod($this->upload_dir.$dir_file_str, 0777); } } return $this->upload_dir.$dir_file_str.'/'.$this->id.$fl; } public function get_upload_dir($id) { $this->set_id($id); for($i=$this->branches;$i>=1;$i--) { $dir=ceil($this->id/pow($this->max_file_count,$i))%$this->max_file_count; $dir_file_arr[]=$dir>0?$dir:$this->max_file_count; } $dir_file_str=implode("/", $dir_file_arr); return $this->upload_dir.$dir_file_str.'/'.$this->id; } public function set_upload_dir($id) { $this->set_id($id); for($i=$this->branches;$i>=1;$i--) { $dir=ceil($this->id/pow($this->max_file_count,$i))%$this->max_file_count; $dir_file_arr[]=$dir>0?$dir:$this->max_file_count; $dir_file_str=implode("/", $dir_file_arr); if(!is_dir($this->upload_dir.$dir_file_str)) { mkdir($this->upload_dir.$dir_file_str, 0777); //chmod($this->upload_dir.$dir_file_str, 0777); } } if(!is_dir($this->upload_dir.$dir_file_str.'/'.$this->id)) { mkdir($this->upload_dir.$dir_file_str.'/'.$this->id, 0777); //chmod($this->upload_dir.$dir_file_str.'/'.$this->id, 0777); } return $this->upload_dir.$dir_file_str.'/'.$this->id; } } 

Download in the archive

For the social network option described above, you need to use the class 2 times: first to build the folder tree, then to build the tree for the files in each folder.

I also draw your attention to the fact that in this post I omitted (but did not know) the topic "The maximum allowable number of files on the hard disk."