Home file hosting based on Sinatra and DataMapper. Part 3 - Very Advanced features
The previous two articles ( one and two ) turned out to be much more popular than I could have expected. And now it's time for the third and final article about the file sharing service based on Sinatra and DataMapper.
This time we will look at:
')
The last time Kane noticed an important error in the application: the key to download the file is a digest on its behalf - but what happens if we load two files with the same name? Unfortunately, since their digests are the same, in the current version we cannot even tell which of the files will be given to the user - the first or the second. But, fortunately, this error is easy to correct: we simply add the id of the file we want to download to the download link. So two identical files will have different links of the form:
But users will still not be able to identify links for downloading other files (the only thing that can be done is to iterate over the ID for the same digest hoping to download the same file). For this fix, we will have to change quite a bit: the code in init.rb and the template list.haml.
init.rb :
list.haml :
Now we are not afraid of the namesake files!
Imagine a situation: your friend asks you to send him a photo made by you. You drop it into your file sharing service and send it a link to the file sharing service, it starts downloading it. And now let's complicate the task: the file size is 20 megabytes, and a friend sits on GPRS. Naturally, if he knew the file size in advance, he would not download it to save expensive traffic. Solution: create a page that will be displayed before downloading and place information on the file name and size on it.
Let's start with init.rb:
So: if in the link parameters it is transferred “nowait = true”, then the download starts instantly, otherwise we just show the download.haml template.
And here he is:
download.haml :
At the beginning there is a simple JavaScript that waits 10 seconds and redirects us to the same link, but with the parameter “nowait = true”, and then the text itself, indicating the file name and its size.
It remains only to expand the file list template so that it contains two links - for immediate download (we will use it ourselves) and for downloading with a delay (we will send this link to ICQ). Looks like that:
list.haml :
Put a tick and proceed to the next item.
SASS is the part of the Haml package that is responsible for creating CSS files. In terms of syntax, SASS is located between CSS and Haml: it uses a scheme with selectors and attributes (CSS), but it uses indents (Haml) as a delimiter, rather than braces.
SASS file consists of a set of rules:
Where SELECTOR (S) is one or more ordinary CSS selectors (class, id, tag name), and PROPRETY_X / VALUE_X are the names and values ​​of CSS properties. Very similar to CSS, but there are some differences:
But back to our sheep: SASS file can be used in two ways - you can get a CSS file from it and connect it to the application, or you can use the SASS template engine built into Sinatra to generate CSS on the fly. We will use the second method in spite of its senselessness :)
init.rb :
layout.haml :
Well, and my file style.sass you can look at this link .
Now our application has some kind of design.
It's time to attach a normal authentication mechanism to our application. We want everyone to be able to download files via direct links, but uploading and deleting files, as well as viewing the general list should be available only after entering the password (for simplicity, we will install one hard-coded password).
I solved the problem as follows: I took the HTTP authentication module (code by reference ), put it in the lib folder and made the following changes to init.rb:
In short, the “helpers do ... end” block is executed in the context of all our blocks - URL handlers, that is, we make the Sinatra: Authorization module accessible within the application. In the same block, you can define methods that can be used in templates and the main application (the so-called helpers are auxiliary methods that allow you to avoid repetitions of the same code in the templates).
So, our application has reached industrial heights and is ready for deployment on a production server. Let me remind you that now we run it with the command “ruby init.rb” and it works while the console is open with ruby ​​- naturally, this is not serious - the web application should be launched by the web server. I choose thin as a web server - a compact and extremely fast server for Ruby applications. Installation is simple:
Now it's time to create several folders in the directory of our application.
In the config folder, move the config.rb file from the lib folder (at the same time adjusting the path to it in init.rb). To configure thin, we need a file that we call thin.yml - create it in the config folder and write the following:
We tell thin that we need to work in the production environment, make chdir in the root directory of the application, place the PID file in the tmp folder, take the Rackup file (see below) in the config folder, keep the log in log / thin.log, support up to 1024 simultaneous connections with a timeout of 30 seconds, hold up to 512 persistent connections and work as a daemon (that is, regardless of the presence of a logged in user in the system).
Now about the rackup file: in fact, this is the configuration file for the Rack - the interface between Ruby and the web server (in our case, thin). This file contains only two lines:
The first line connects init.rb (that is, our application), the second tells Rack that you need to run thin on port 3000 and give it a Sinatra application.
It is done! Now the application is launched by this command.
We simply transfer the thin configuration file.
Stop is a team
I specifically left this section in the end because I understand that very few people will test applications for Sinatra. I will not go into details and tell you what RSpec is , just show you what the specs look like.
No tricks - the same describe / it / should, as in Rails, for example. The main thing is not to forget to connect sinatra / test / rspec.
First is the benchmark of the main page (file list).
ab -n 1000 -c 1 -A admin: secret http://127.0.0.1.1000000/
ab -n 1000 -c 10 -A admin: secret http://127.0.0.1.1000000/
ab -n 1000 -c 100 -A admin: secret http://127.0.0.1.1000000/
Upload files to server (file size 1.5 Kb)
ab -n 1000 -c 1 -A admin: secret -T 'application / x-www-form-urlencoded' -p post.data http://127.0.0.1.73000/
ab -n 1000 -c 10 -A admin: secret -T 'application / x-www-form-urlencoded' -p post.data http://127.0.0.1.73000/
ab -n 1000 -c 100 -A admin: secret -T 'application / x-www-form-urlencoded' -p post.data http://127.0.0.1天000/
Note that the performance practically does not change with an increase in the number of simultaneous requests 100 (one hundred!) Times. Testing was done on Mac Book Core 2 Duo 2.4 Ghz, 2 GB ram with several applications running in the background.
It is time to complete my randomly begun epic. I hope you were interested and I was able to encourage at least some to study non-mainstream technologies (Sinatra, DataMapper, thin, haml, sass). The application in its latest version is posted on github . Thanks to everyone who took the time to read these non-brief articles.
This time we will look at:
- Problem with the same file names
- Waiting page before loading
- Creating CSS with SASS
- Authentication
- Run from under thin
- Testing with RSpec
- Benchmarks
')
Same file names
The last time Kane noticed an important error in the application: the key to download the file is a digest on its behalf - but what happens if we load two files with the same name? Unfortunately, since their digests are the same, in the current version we cannot even tell which of the files will be given to the user - the first or the second. But, fortunately, this error is easy to correct: we simply add the id of the file we want to download to the download link. So two identical files will have different links of the form:
/ DIGEST / ID1 / DIGEST / ID2
But users will still not be able to identify links for downloading other files (the only thing that can be done is to iterate over the ID for the same digest hoping to download the same file). For this fix, we will have to change quite a bit: the code in init.rb and the template list.haml.
init.rb :
get '/: sha /: id' do @file = StoredFile.first: sha => params [: sha],: id => params [: id] Next, no change get '/: sha /: id / delete' do @file = StoredFile.first: sha => params [: sha],: id => params [: id] Next, no change
list.haml :
% a {: href => "/#{file.sha}/##file.id}",: title => file.filename} = file.filename Now we are not afraid of the namesake files!
Page before loading
Imagine a situation: your friend asks you to send him a photo made by you. You drop it into your file sharing service and send it a link to the file sharing service, it starts downloading it. And now let's complicate the task: the file size is 20 megabytes, and a friend sits on GPRS. Naturally, if he knew the file size in advance, he would not download it to save expensive traffic. Solution: create a page that will be displayed before downloading and place information on the file name and size on it.
Let's start with init.rb:
get '/: sha /: id' do
@file = StoredFile.first: sha => params [: sha],: id => params [: id]
unless params [: nowait] == ​​'true'
haml: download
else
@ file.downloads + = 1
@ file.save
send_file "./files/#{@file.id}.upload",: filename => @ file.filename,: type => 'Application / octet-stream'
end
end
So: if in the link parameters it is transferred “nowait = true”, then the download starts instantly, otherwise we just show the download.haml template.
And here he is:
download.haml :
% script {: type => "text / javascript"}
nowait = '? nowait = true';
var timeout = true;
setTimeout ('if (timeout) {window.location = window.location + nowait;}', 10,000);
% h1 file download
.info
You are about to download the file.
% span.filename> = "#{@file.filename}"
by size
% span.filesize
= @ file.filesize / 1024
kilobyte
The download will start for
% span # seconds 10
seconds Press on
% a {: href => "/ # {params [: sha]}? nowait = true",: onclick => 'timeout = false;' } this link
if you don't want to wait
At the beginning there is a simple JavaScript that waits 10 seconds and redirects us to the same link, but with the parameter “nowait = true”, and then the text itself, indicating the file name and its size.
It remains only to expand the file list template so that it contains two links - for immediate download (we will use it ourselves) and for downloading with a delay (we will send this link to ICQ). Looks like that:
list.haml :
% td.filename
% a {: href => "/#{file.sha}/#{file.id}?nowait=true",: title => file.filename} = file.filename
= "(# {file.filesize / 1024} Kb)"
% a {: href => "/#{file.sha}/##file.id}"} For shipping
Put a tick and proceed to the next item.
SASS
SASS is the part of the Haml package that is responsible for creating CSS files. In terms of syntax, SASS is located between CSS and Haml: it uses a scheme with selectors and attributes (CSS), but it uses indents (Haml) as a delimiter, rather than braces.
SASS file consists of a set of rules:
SELECTOR (S) : PROPERTY1 VALUE1 : PROPERTY2 VALUE2 ... : PROPERTY_N VALUE_N
Where SELECTOR (S) is one or more ordinary CSS selectors (class, id, tag name), and PROPRETY_X / VALUE_X are the names and values ​​of CSS properties. Very similar to CSS, but there are some differences:
- Instead of curly braces, an indent of 2 spaces is used.
- You can use nested rules (damn convenient in the case of a complex set of CSS rules):
#main p : color # 00ff00 : width 97% .redbox : background-color # ff0000 : color # 000000compiled into#main p { color: # 00ff00; width: 97%; } #main p .redbox { background-color: # ff0000; color: # 000000; } - Nested "namespaces":
.funky font : family fantasy : size 30em : weight boldCompiles to.funky { font-family: fantasy; font-size: 30em; font-weight: bold; } - Using constants and arithmetic operations:
! main_width = 10 ! unit1 = em ! unit2 = px ! bg_color = # a5f39e #main : background-color =! bg_color p : background-color =! bg_color + # 202020 : width =! main_width +! unit1 img.thumb : width = (! main_width + 15) +! unit2 - SASS comments - they are present in the SASS template, but they are not in the final CSS
- Several options for formatting the final CSS (starting from the most readable expanded and ending with minimal compressed)
But back to our sheep: SASS file can be used in two ways - you can get a CSS file from it and connect it to the application, or you can use the SASS template engine built into Sinatra to generate CSS on the fly. We will use the second method in spite of its senselessness :)
init.rb :
get '/style.css' do response ['Content-Type'] = 'text / css; charset = utf-8 '# Set the response header sass: style end
layout.haml :
% link {: href => "/style.css",: media => "screen",: rel => "stylesheet",: type => "text / css"} Well, and my file style.sass you can look at this link .
Now our application has some kind of design.
Authentication
It's time to attach a normal authentication mechanism to our application. We want everyone to be able to download files via direct links, but uploading and deleting files, as well as viewing the general list should be available only after entering the password (for simplicity, we will install one hard-coded password).
I solved the problem as follows: I took the HTTP authentication module (code by reference ), put it in the lib folder and made the following changes to init.rb:
require 'lib / authorization' helpers do include Sinatra :: Authorization end get '/' do require_administrative_privileges # Get no further end post '/' do require_administrative_privileges # Further post without changes end get '/: sha /: id / delete' do require_administrative_privileges # Delete without changes end
In short, the “helpers do ... end” block is executed in the context of all our blocks - URL handlers, that is, we make the Sinatra: Authorization module accessible within the application. In the same block, you can define methods that can be used in templates and the main application (the so-called helpers are auxiliary methods that allow you to avoid repetitions of the same code in the templates).
Run from under thin
So, our application has reached industrial heights and is ready for deployment on a production server. Let me remind you that now we run it with the command “ruby init.rb” and it works while the console is open with ruby ​​- naturally, this is not serious - the web application should be launched by the web server. I choose thin as a web server - a compact and extremely fast server for Ruby applications. Installation is simple:
sudo gem install thin
Now it's time to create several folders in the directory of our application.
mkdir config mkdir tmp mkdir log
In the config folder, move the config.rb file from the lib folder (at the same time adjusting the path to it in init.rb). To configure thin, we need a file that we call thin.yml - create it in the config folder and write the following:
---
environment: production
chdir: APPLICATION CATALOG
pid: APPLICATION CATALOG / tmp / thin.pid
rackup: APPLICATION CATALOG / config / config.ru
log: APPENDIX / log / thin.log
max_conns: 1024
timeout: 30
max_persistent_conns: 512
daemonize: true
We tell thin that we need to work in the production environment, make chdir in the root directory of the application, place the PID file in the tmp folder, take the Rackup file (see below) in the config folder, keep the log in log / thin.log, support up to 1024 simultaneous connections with a timeout of 30 seconds, hold up to 512 persistent connections and work as a daemon (that is, regardless of the presence of a logged in user in the system).
Now about the rackup file: in fact, this is the configuration file for the Rack - the interface between Ruby and the web server (in our case, thin). This file contains only two lines:
require 'init' Rack :: Handler :: Thin.run Sinatra :: Application,: Port => 3000,: Host => "0.0.0.0"
The first line connects init.rb (that is, our application), the second tells Rack that you need to run thin on port 3000 and give it a Sinatra application.
It is done! Now the application is launched by this command.
thin start -C config / thin.yml
We simply transfer the thin configuration file.
Stop is a team
thin stop -C config / thin.yml
Testing with RSpec
I specifically left this section in the end because I understand that very few people will test applications for Sinatra. I will not go into details and tell you what RSpec is , just show you what the specs look like.
require 'sinatra'
require 'sinatra / test / rspec'
require 'init'
describe 'TrashFiles app' do
it 'should render template with delay' do
@file = StoredFile.first
get "/#{@file.sha}/#{@file.id}"
@response ['Content-Type']. should == "text / html"
end
it 'should give file if? nowait = true is supplied' do
@file = StoredFile.first
get "/#{@file.sha}/#{@file.id}?nowait=true"
@response ['Content-Type']. should == "Application / octet-stream"
@response ['Content-Disposition']. should == "attachment; filename = \" # {@ file.filename} \ ""
end
end
No tricks - the same describe / it / should, as in Rails, for example. The main thing is not to forget to connect sinatra / test / rspec.
Benchmarks
In one of the comments I was asked to measure the performance of the resulting application - no problem.First is the benchmark of the main page (file list).
ab -n 1000 -c 1 -A admin: secret http://127.0.0.1.1000000/
Concurrency Level: 1 Time taken for tests: 24.109 seconds Total transferred: 3739000 bytes HTML transferred: 3604000 bytes Requests per second: 41.48 [# / sec] (mean) Time per request: 24.109 [ms] (mean) Time per request: 24.109 [ms] (mean, across all concurrent requests) Transfer rate: 151.45 [Kbytes / sec] received
ab -n 1000 -c 10 -A admin: secret http://127.0.0.1.1000000/
Concurrency Level: 10 Time taken for tests: 24.381 seconds Total transferred: 3739000 bytes HTML transferred: 3604000 bytes Requests per second: 41.02 [# / sec] (mean) Time per request: 243.811 [ms] (mean) Time per request: 24.381 [ms] (mean, across all concurrent requests) Transfer rate: 149.76 [Kbytes / sec] received
ab -n 1000 -c 100 -A admin: secret http://127.0.0.1.1000000/
Concurrency Level: 100 Time taken for tests: 23.798 seconds Total transferred: 3739000 bytes HTML transferred: 3604000 bytes Requests per second: 42.02 [# / sec] (mean) Time per request: 2379.816 [ms] (mean) Time per request: 23.798 [ms] (mean, across all concurrent requests) Transfer rate: 153.43 [Kbytes / sec] received
Upload files to server (file size 1.5 Kb)
ab -n 1000 -c 1 -A admin: secret -T 'application / x-www-form-urlencoded' -p post.data http://127.0.0.1.73000/
Concurrency Level: 1
Time taken for tests: 16.305 seconds
Total transferred: 160000 bytes
Total POSTed: 242000
Requests per second: 61.33 [# / sec] (mean)
Time per request: 16.305 [ms] (mean)
Time per request: 16.305 [ms] (mean, across all concurrent requests)
Transfer rate: 9.58 [Kbytes / sec] received
14.49 kb / s sent
24.08 kb / s total
ab -n 1000 -c 10 -A admin: secret -T 'application / x-www-form-urlencoded' -p post.data http://127.0.0.1.73000/
Concurrency Level: 10
Time taken for tests: 18.463 seconds
Total transferred: 161280 bytes
Total POSTed: 243936
HTML transferred: 0 bytes
Requests per second: 54.16 [# / sec] (mean)
Time per request: 184.631 [ms] (mean)
Time per request: 18.463 [ms] (mean, across all concurrent requests)
Transfer rate: 8.53 [Kbytes / sec] received
12.90 kb / s sent
21.43 kb / s total
ab -n 1000 -c 100 -A admin: secret -T 'application / x-www-form-urlencoded' -p post.data http://127.0.0.1天000/
Concurrency Level: 100
Time taken for tests: 16.029 seconds
Total transferred: 160160 bytes
Total POSTed: 242242
HTML transferred: 0 bytes
Requests per second: 62.39 [# / sec] (mean)
Time per request: 1602.899 [ms] (mean)
Time per request: 16.029 [ms] (mean, across all concurrent requests)
Transfer rate: 9.76 [Kbytes / sec] received
14.76 kb / s sent
24.52 kb / s total Note that the performance practically does not change with an increase in the number of simultaneous requests 100 (one hundred!) Times. Testing was done on Mac Book Core 2 Duo 2.4 Ghz, 2 GB ram with several applications running in the background.
The end
It is time to complete my randomly begun epic. I hope you were interested and I was able to encourage at least some to study non-mainstream technologies (Sinatra, DataMapper, thin, haml, sass). The application in its latest version is posted on github . Thanks to everyone who took the time to read these non-brief articles.
Source: https://habr.com/ru/post/50348/
All Articles