Tracks Flow + Open Source = Simple File Storage

We are pleased to announce that the Tracks Flow project is starting to support the Open Source community. Today we are laying out a simple development in open access - a file storage server in PHP with a client library in C #.

We ask you not to judge strictly - this project was written a long time ago and has since been used without special changes at fidel.ru, and then at tracksflow.com . Before exposing to open access, we brushed it a little.

This code is intended for those who want to implement in their storage system large (and not so) files with the least effort.

Virtues

• compact code - easy to check and fix for yourself
• good scalability of stored content
• no database
• ability to work with files of arbitrary size *

* The only function that may not work correctly on large files is to check the md5 file amounts. The function may cause a TimeOutException.
')

How does it work

A file is loaded in chunks of data — one POST request per one chunk. Loading occurs sequentially. When processing the next request, the server appends the received part to the resulting file. All operations are time invariant to the length of the target file.
All files are stored on the file system in a directory structure:

 /<_>//<___id>/<___id>/<-__id>/<filename.ext> 
         ,   . 
       Nginx  X-ACCEL_REDIRECT    .    Nginx. 
 
     
  Nginx + php-fpm.  nginx         : 
 server { listen <your_LAN_ip>; server_name <you_file_storage_hostname>; location /<location_name> { internal; root <storage_root_path_should_be_the_same_in_lib.php>; } location ~ \.php { root /<path_to_php_code>; fastcgi_pass 127.0.0.1:9000; #fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME /<path_to_php_code>/$fastcgi_script_name; include fastcgi_params; } } 
 /<_>//<___id>/<___id>/<-__id>/<filename.ext> 
         ,   . 
       Nginx  X-ACCEL_REDIRECT    .    Nginx. 
 
     
  Nginx + php-fpm.  nginx         : 
 server { listen <your_LAN_ip>; server_name <you_file_storage_hostname>; location /<location_name> { internal; root <storage_root_path_should_be_the_same_in_lib.php>; } location ~ \.php { root /<path_to_php_code>; fastcgi_pass 127.0.0.1:9000; #fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME /<path_to_php_code>/$fastcgi_script_name; include fastcgi_params; } } 
 /<_>//<___id>/<___id>/<-__id>/<filename.ext> 
         ,   . 
       Nginx  X-ACCEL_REDIRECT    .    Nginx. 
 
     
  Nginx + php-fpm.  nginx         : 
 server { listen <your_LAN_ip>; server_name <you_file_storage_hostname>; location /<location_name> { internal; root <storage_root_path_should_be_the_same_in_lib.php>; } location ~ \.php { root /<path_to_php_code>; fastcgi_pass 127.0.0.1:9000; #fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME /<path_to_php_code>/$fastcgi_script_name; include fastcgi_params; } } 
 /<_>//<___id>/<___id>/<-__id>/<filename.ext> 
         ,   . 
       Nginx  X-ACCEL_REDIRECT    .    Nginx. 
 
     
  Nginx + php-fpm.  nginx         : 
 server { listen <your_LAN_ip>; server_name <you_file_storage_hostname>; location /<location_name> { internal; root <storage_root_path_should_be_the_same_in_lib.php>; } location ~ \.php { root /<path_to_php_code>; fastcgi_pass 127.0.0.1:9000; #fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME /<path_to_php_code>/$fastcgi_script_name; include fastcgi_params; } } 
 /<_>//<___id>/<___id>/<-__id>/<filename.ext> 
         ,   . 
       Nginx  X-ACCEL_REDIRECT    .    Nginx. 
 
     
  Nginx + php-fpm.  nginx         : 
 server { listen <your_LAN_ip>; server_name <you_file_storage_hostname>; location /<location_name> { internal; root <storage_root_path_should_be_the_same_in_lib.php>; } location ~ \.php { root /<path_to_php_code>; fastcgi_pass 127.0.0.1:9000; #fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME /<path_to_php_code>/$fastcgi_script_name; include fastcgi_params; } }  /<_>//<___id>/<___id>/<-__id>/<filename.ext> 
         ,   . 
       Nginx  X-ACCEL_REDIRECT    .    Nginx. 
 
     
  Nginx + php-fpm.  nginx         : 
 server { listen <your_LAN_ip>; server_name <you_file_storage_hostname>; location /<location_name> { internal; root <storage_root_path_should_be_the_same_in_lib.php>; } location ~ \.php { root /<path_to_php_code>; fastcgi_pass 127.0.0.1:9000; #fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME /<path_to_php_code>/$fastcgi_script_name; include fastcgi_params; } } 
 /<_>//<___id>/<___id>/<-__id>/<filename.ext> 
         ,   . 
       Nginx  X-ACCEL_REDIRECT    .    Nginx. 
 
     
  Nginx + php-fpm.  nginx         : 
 server { listen <your_LAN_ip>; server_name <you_file_storage_hostname>; location /<location_name> { internal; root <storage_root_path_should_be_the_same_in_lib.php>; } location ~ \.php { root /<path_to_php_code>; fastcgi_pass 127.0.0.1:9000; #fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME /<path_to_php_code>/$fastcgi_script_name; include fastcgi_params; } } 

Copy the server files in the directory <path_to_php_code>

Checking lib.php - the getfsroot () function should return the path specified in the config in <storage_root_path_should_be_the_same_in_lib.php>; the getlocroot () function must contain the same as specified in the config in <location_name>.

We set the rights to <storage_root_path_should_be_the_same_in_lib.php> the same as those of Nginx / php-fpm, we give write access.

If we want to use chunks larger than 512KB, then we need to register in php-fpm, nginx and php.ini the appropriate values ​​for restrictions on upload.

How to scale

When scaling by the size of the stored files, it becomes necessary to use several servers. In this case, it is convenient to put on the required number of Lufs servers, mount the file system to a web server and work with it as with a local directory.

Use Case

With this solution, we now store user avatars and playlist covers. Accordingly, we use two spaces (scope) for each type of entity. The identifier of each map is equal to the identifier of the entity in the database. The file name consists of a size system (for example, ImageLarge) that defines the file type for the entity and the jpg extension. This approach allows knowing only the entity identifier and having a directory of file types corresponding to the entity to form a link to the file.

Links

Solution Git-repository: https://github.com/tracksflow/FileStorage

And, of course, we invite you to TracksFlow.com . The service is actively developing, so you will definitely find something interesting if you haven’t been around for a long time. Well, for those who have not been yet - today we distribute invites to everyone who sends a request to the PM to any employee of the company (do not forget to include an email address in the message).