Making your webfile

For some reason I always wanted to make my own service for downloading files. All sorts of slil / zalil did not suit their speed. ifolder - an abundance of advertising. Used not very popular (because of this, he didn’t get any worse) with the service up.spbland.ru . But this is somehow not correct. And then I decided to write my service. I will not go into details and routine, just a concept.

Requirements before the resource:
- uploading files with the POST method of at least 100 MB, although why not 1000?
- mandatory process visualization, i.e. upload progressbar
- the ability to resume files by users (during download)
- exclude the possibility of installing direct links to files on third-party forums, sites. Only through site.ru/ID

First I will tell about upload progressbar:
During the reception of a large number of multipart / form-data information, apache adds everything to / tmp, and after the download is finished (that is, when the entire request is submitted to the web server), it transfers control to php, which in turn makes move_uploaded_file . By its nature, php is designed in such a way that process A cannot know at what stage of the download process B. The process of uploading progress was invented for this very experimentally. By passing input type = "file", add input type = "hidden" name = "UPLOAD_IDENTIFIER" value = "some number" to the form, later on this identifier you can use the uploadprogress_info_info () function to track the download status: count -by received bytes, total bytes, speed and other parameters. In my case, I used AJAX to access the script once a second and receive up-to-date information.
')
Tasks can be performed:
- badly
- fine
- cool on a global scale

Bad - it means to receive files and give files through php. A little bit to improve this path could be enlisted with support for HTTP 206 Partial Content . However, this path is a path for patching holes that does not deserve attention.

Cool: files should be dealt with by Apache, and better nginx. Firstly, these programs fully support the HTTP / 1.1 specification, secondly Apache will always work faster than interpreted php. Nginx implements a special X-Accel-Redirect mechanism for uploading files. The bottom line is that nginx is installed as a transparent proxy server. Nginx processes, coming from the user, a request to the file site.ru/somefiles/123.mp3 and transmits it to Apache & php. Further, either by means of nginx, or through mod_rewrite

RewriteRule ^/somefiles/(.*)$ /get.php?path=$1 [L]

request is reduced to /get.php?path=123.mp3
In php analyzing the ip address of the client, session, etc. a decision is made to issue a file, or to refuse. Further php transfers in a special way the formed heading:

header("X-Accel-Redirect: /files/123.mp3")

nginx takes control over itself and already begins to give the file by its own means, making it a million times more efficient than apache. However, this method has its own cant: when a file is uploaded to the server, nginx completely receives the entire request, and only then sends it to apache. Until nginx fully receives the entire file, apache will not even know that the download is in progress, which makes it impossible for the upload progress bar to work.

Conclusion: the files should be uploaded directly through apache, and the return through nginx. In my case, it would be possible to hang nginx (or apache) on some other port. But to force users stuffed with firewalls to download from the port is different from 80 - not comme il faut. Therefore, it would be better to hang up nginx to a separate address, for example, download-0.site.ru (with this concept, you could add more servers later). But since I did not have the opportunity to use 2 addresses, the planetary version of file hosting had to be abandoned.

However, I still wanted to keep all the tasks set earlier.

Okay And here I came to the rescue: Rewritemap

By constructing a structure like this:

Rewritemap ipmap txt:/home/anton/iplist.txt
RewriteCond ${ipmap:%{REMOTE_ADDR}} =allow
RewriteRule ^([0-9]+)/download/ /xfiles/$1 [L]
RewriteRule ^([0-9]+)/download/ /$1/ [L,R=302]
RewriteRule ^([0-9]+)/? /view.php?id=$1 [L]

I got the opportunity to redirect links like site.ru/100200300/download/22.mp3 to the file / xfiles / 100200300 (where the file is actually located) if the client’s ip address is in the iplist.txt file, or redirected to the site.ru/ 100200300 with captcha, which, in the case of the correct indication of the code in the picture, will transfer the client's ip to the iplist.txt file and redirect it again via the Location to
site.ru/100200300/download/22.mp3 , but this time there will be a real return of the file.

The iplist.txt file has the following format:

77.1.1.1 allow # 1207100906

After the grid I have a timestamp, which informs a special service script when to delete this line. The service script runs once an hour via crontab.

As a result, I got a fairly correct and reliable system. Perhaps its disadvantage is the inefficient use of the RAM, because Each apache process (with the prefork mpm module) can simultaneously serve only one client, while the process itself consumes approximately 10 MB of memory. If we assume that 1000 modem users will start simultaneously downloading from the server in 10 threads each, a collapse will occur and there should not be enough memory. But the directive from Httpd.conf MaxClients will save me from this. With nginx, such a problem could not be in principle. In addition, nginx has good functionality for restricting users, for example, by the number of simultaneous connections from one address.

Ways to improve to a global hosting:
- transition to nginx as a frontend
- creation of a distributed system with a file reception server in the center
- refuse to store all files in one folder / xfiles /, create another 100 subfolders inside xfiles based on the first 2 characters.
- switch to SAS disks, because on the Internet scale this is not the exchange of single heavy files, but the constant download of hundreds of small files (music, photos)

The working version of the described: up.giga.su

PS My first habrapost