File Download Service on Golang
During the development of the server part of the file download service on Golang, a separate application was born - pavo . The application's tasks include downloading entire files, one or more at a time, piecewise file download (chunked upload), image converter. Implemented data loading via
To install the application, you need a Golang compiler. Instructions for installing it can be found on the official website . You also need to configure the
For an example of how this can be done in MacOS:
Repositories with code in the community are not centralized. Various version control systems are used:
Installation example in MacOS:
ImageMagick is used to convert images to the server:
When first installed, run the command in the console:
When updating the application and dependent libraries:
In order to see how the application works with a basic example, run the command in the console:
Thus, we launched the application with the root directory in the specified directory through the
Typical server response when loading an image:
The most common way to upload files to a server is to use forms. In this case, the request is as follows:
In the
The modern approach allows you to send binary data using a XHR type query on the client side. The request that the server sees is as follows:
In this way, you can send only one file for one request, the name of which will be available in the
To download large files on the client side a packet of requests is formed with parts of the source file. Request example:
The name of the source file and the cookie value by the pavo key are used to identify the intermediate file with the loaded parts of the original. The
The application is written in the Golang language. Gin is used as a web framework. The code is divided into two packages ( upload and attachment ) and the main application (executable file). The upload package is responsible for downloading the source files or a piece of the file. The attachment package is responsible for creating the final directory for storing the file and its versions, converting images, generating data. The main application starts the web server and implements the controller role.
Source code with small examples in tests is available on github .
The application has the startup options
The application accepts all download requests to
For all files, the default version is original . For images, a thumbnail is added with a value of
To work in the production environment, it is desirable to use the nginx web server. The tasks of the web server will include receiving requests from clients, writing the body to a temporary file, authorizing the request on the main application, sending the headers of the original request to the pavo application.
Recommended nginx configuration:
From the client to the web server comes a request. Nginx waits to receive the entire request (this feature of it does not allow implementing a full-fledged progress bar with proxying to the application server). After receiving the request, the body will be written to a temporary file in the directory specified by the
Before sending a request to the server application pavo will be made authorization. This is done using the ngx_http_auth_request_module module. A subquery will be made on location
Next, a new pair is added to the headers of the original request, the
The service was conceived as a standalone application in the web project infrastructure, which assumes the role of downloading and sharing files, converting images, video and audio. With interface through HTTP Json API.
UPDATE: Fixed settings for proxied request headers in the nginx configuration.
multipart/form-data and loading the file in binary form in the request body. To work in the production environment, nginx is used to authorize and process slow connections. As a client library, you can use the jQuery File Uploader .Installation
Compiler installation
To install the application, you need a Golang compiler. Instructions for installing it can be found on the official website . You also need to configure the
$GOPATH environment variable.For an example of how this can be done in MacOS:
- Install the compiler;
$ brew install go $ mkdir $HOME/go - Set up the environment variable by editing the user profile.
')# Add this line in your .zshrc or .bash_profile export GOPATH=$HOME/go export PATH=$PATH:$GOPATH/bin
Installing version control systems
Repositories with code in the community are not centralized. Various version control systems are used:
Installation example in MacOS:
$ brew install git mercurial svn bazaar Install ImageMagick
ImageMagick is used to convert images to the server:
$ brew install imagemagick Application installation
When first installed, run the command in the console:
$ go get github.com/kavkaz/pavo When updating the application and dependent libraries:
$ go get -u github.com/kavkaz/pavo/... Fast start
In order to see how the application works with a basic example, run the command in the console:
$ pavo --storage=$GOPATH/src/github.com/kavkaz/pavo/dummy/root_storage Thus, we launched the application with the root directory in the specified directory through the
--storage option. A service with a basic example will be available at localhost:9073/example/jfu-basic.html localhost:9073/example/jfu-basic.html . To specify a different host and port, use the console option --host .Protocol details
Typical server response when loading an image:
{ "files": [ { "dir": "/image/2014/6s/1c5cnx", "name": "original_user_filename.jpg", "type": "image", "versions": { "original": { "filename": "original-1qeh.jpg", "height": 420, "size": 28057, "url": "/image/2014/6s/1c5cnx/original-1qeh.jpg", "width": 300 }, "thumbnail": { "filename": "thumbnail-1qef.jpg", "height": 90, "size": 3566, "url": "/image/2014/6s/1c5cnx/thumbnail-1qef.jpg", "width": 120 } } } ], "status": "ok" } The most common way to upload files to a server is to use forms. In this case, the request is as follows:
POST /files HTTP/1.1 Content-Length: 21929 Content-Type: multipart/form-data; boundary=----5XhQf4IXV9Q26uHM ------5XhQf4IXV9Q26uHM Content-Disposition: form-data; name="files[]"; filename="pic.jpg" Content-Type: image/jpeg ...bytes... In the
Content-Type header, the value of the boundary passed, which serves to separate the values in the request body. Thus, you can transfer several files in one request. jQuery File Upload has the appropriate option for sending multiple files.The modern approach allows you to send binary data using a XHR type query on the client side. The request that the server sees is as follows:
POST /files HTTP/1.1 Content-Length: 21744 Content-Disposition: attachment; filename="pic.jpg" ...bytes... In this way, you can send only one file for one request, the name of which will be available in the
Content-Disposition header.To download large files on the client side a packet of requests is formed with parts of the source file. Request example:
POST /files HTTP/1.1 Content-Length: 10240 Content-Range: bytes 0-10239/36431 Content-Disposition: attachment; filename="pic.jpg" Cookie:pavo=377cb76c-2538-40d3-a3d0-13d86d206ba7 ...bytes... The name of the source file and the cookie value by the pavo key are used to identify the intermediate file with the loaded parts of the original. The
Content-Range header contains information about what part of the file the client is betraying and what is the size of the original file. If the last piece is loaded, the server completes the download procedure and generates a response with the data about the file received and its versions.Application code
The application is written in the Golang language. Gin is used as a web framework. The code is divided into two packages ( upload and attachment ) and the main application (executable file). The upload package is responsible for downloading the source files or a piece of the file. The attachment package is responsible for creating the final directory for storing the file and its versions, converting images, generating data. The main application starts the web server and implements the controller role.
Source code with small examples in tests is available on github .
Application options
The application has the startup options
--host and --storage . Specify host:port to start the web server and the root directory of the repository, respectively.The application accepts all download requests to
/files . In the query_string converts parameter, converts can pass the conversion parameters for images. For example: POST /files?converts={"pic":"400x300"} For all files, the default version is original . For images, a thumbnail is added with a value of
120x90 .Production environment
To work in the production environment, it is desirable to use the nginx web server. The tasks of the web server will include receiving requests from clients, writing the body to a temporary file, authorizing the request on the main application, sending the headers of the original request to the pavo application.
Recommended nginx configuration:
server { listen 80; server_name pavo.local; access_log /usr/local/var/log/nginx/pavo/access.log; error_log /usr/local/var/log/nginx/pavo/error.log notice; location /auth { internal; proxy_method GET; proxy_set_header Content-Length ""; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass_request_body off; proxy_pass http://localhost:3000/auth/url/in/your/app; client_max_body_size 0; } location /files { auth_request /auth; client_body_temp_path /tmp; client_body_in_file_only on; client_body_buffer_size 521K; client_max_body_size 10G; proxy_pass_request_headers on; proxy_set_header X-FILE $request_body_file; proxy_pass_request_body off; proxy_set_header Content-Length 0; proxy_pass http://127.0.0.1:9073; } location / { root /Path/To/Root/Of/Storage; } } From the client to the web server comes a request. Nginx waits to receive the entire request (this feature of it does not allow implementing a full-fledged progress bar with proxying to the application server). After receiving the request, the body will be written to a temporary file in the directory specified by the
client_body_temp_path option.Before sending a request to the server application pavo will be made authorization. This is done using the ngx_http_auth_request_module module. A subquery will be made on location
/auth , which in turn proxies the headers of the original request to the server of the main application. In case of successful authorization, the server should return an empty body with the response status code 200 .Next, a new pair is added to the headers of the original request, the
X-File key, and the value is the path to the temporary file with the request body. And only after that the resulting request (headers and empty body) is sent to the application pavo. It processes the request, saving the files, and returns a response with the data about the uploaded files in JSON format.Conclusion
The service was conceived as a standalone application in the web project infrastructure, which assumes the role of downloading and sharing files, converting images, video and audio. With interface through HTTP Json API.
UPDATE: Fixed settings for proxied request headers in the nginx configuration.
Source: https://habr.com/ru/post/234693/
All Articles