Large Files and Sinatra

I recently encountered an interesting problem when trying to give a large file through Sinatra::Helpers.send_file resulted in the release of all RAM (typical file size is 14Gb).

The study showed that Sinatra itself reads and sends the file in pieces of 512 bytes, but the web server thin (as well as WEBrick) buffers the output in RAM at its own level, which leads to such sad consequences.

To solve the problem, it was enough to go to the Rainbows web server (a web server based on unicorn code, but designed to work without proxying, for slow clients and / or services). But with the return of large files, the process ate about 30% of the CPU on a single core.
')
Rainbows allows you to optimize file delivery using, for example, the sendfile gem, which provides the appropriate operating system APIs. But for this, it is necessary that the file is returned via the Rack :: File API.

In the current Sinatra master branch, the send_file method was rewritten using the Rack :: File API, so we can simply backport the corresponding functionality into existing versions of the Sinatra gem:

 if Sinatra::VERSION < '1.3.0' && Rack.release >= '1.3' # Monkey patch old Sinatra to use Rack::File to serve files. Sinatra::Helpers.class_eval do # Got from Sinatra 1.3.0 sources def send_file(path, opts={}) if opts[:type] or not response['Content-Type'] content_type opts[:type] || File.extname(path), :default => 'application/octet-stream' end if opts[:disposition] == 'attachment' || opts[:filename] attachment opts[:filename] || path elsif opts[:disposition] == 'inline' response['Content-Disposition'] = 'inline' end last_modified opts[:last_modified] if opts[:last_modified] file = Rack::File.new nil file.path = path result = file.serving env result[1].each { |k,v| headers[k] ||= v } halt result[0], result[2] rescue Errno::ENOENT not_found end end end 

The rainbows configuration file will look something like this:

 # try to use sendfile when available begin require 'sendfile' rescue LoadError end Rainbows! do use :ThreadSpawn end 

Now we use an effective file upload technique if the system has a rack gem version 1.3 or higher installed and the sendfile gem is installed. By the way, when using ruby ​​1.9 hem, sendfile is most likely not required.

PS: If your service is located behind a proxy server, then it is more optimal to use the capabilities provided by proxy servers, for example, the API X-Accel-Redirect (nginx) or X-Sendfile (Lighttpd, Apache).