Home file hosting based on Sinatra and DataMapper. Part 2 - Advanced features.
In the first article, I talked about how to write a simple web application using Sinatra and DataMapper. This time we will add many new features and improve the code as a whole.
In this section, we will prepare the ground for all future research.
So, last time we got an application with the following file structure:
')
In this case, all the code is in the same file myapp.rb. In my opinion, this is very inconvenient and ugly, so this time we will start by creating folders and files, some of which will be useful to us in the course of the story. I have the following structure:
In the files folder we have files that are accessible from the outside. In the lib folder there are libraries. The tasks folder will contain rake-tasks. In the manual folder, we will put the files, bypassing the web interface. Public - storage of files sent to the browser directly. In the test, we put, oddly enough, tests. Finally, views are already familiar to us - there are templates for generating HTML pages (as long as we have only one).
Now let's install Sinatra locally (directly to the folder with the application) - this will allow us not to worry about installing Sinatra gem. To do this, download the Sinatra code from the git repository. To do this, you must have git installed:
We will place the main code of our application in the init.rb file, having previously changed its beginning to this:
This code connects Sinatra and the configuration file config.rb from the lib folder - we’ll now create and write the following code in it (not doing anything yet):
Then Rakefile comes in handy - a special file for the rake utility, which tells it where to get the tasks. A file named Rakefile (without extension) located in the root directory should contain the following:
In it, we simply load all the rake files from the lib / tasks directory and its subdirectories (the regular expression /**/*.rake is used for this).
The application code that lies in one file is not only inconvenient, but also wrong :) Therefore, now we will start by putting our StoredFile class into a separate one, sorry for the pun, file. In the lib folder we should have the file stored_file.rb with the following contents:
And at the end of the countless require in init.rb we add the line
The functionality of our application has not changed a bit, but now it looks a little more decent.
In the first version of the application, we used the language Erb (Embedded RuBy) for generating HTML pages, which is HTML-code with included fragments of Ruby-code. This approach is traditional in Rails, but we are engaged in marginal solutions :) In this regard, I decided to use Haml as a language for templates.
Haml is a markup language that preaches beauty, as the main deity of the pattern. Indeed, the Haml code, in my opinion, looks nicer (and they say that it works faster). Another important feature of Haml is that it will not allow you not to close the tag (because you will not use explicit HTML tags at all)
Installation is simple:
Let's create the list.haml file in our views directory and fill it with content. Haml is similar to Python in that it uses indentation in the code as block delimiters (in HTML, closing tags are used for this, and in Erb, the end keyword is used). So the html code
On Haml looks like this:
That, you see, is much shorter.
A little about the syntax, which is useful to us:
Creates a <TAG /> or <TAG> </ TAG> tag - depending on the tag (for hr the first option will be chosen, for a - the second one)
Creates a <TAG> CONTENT </ TAG> tag.
Turns into <tag> CONTENT <tag>. In this case, CONTENT may also contain the operator% and any other operators.
The syntax for defining attributes is the same as for Ruby hashes:
It will become <span class = "header" id = "news_135" style = "float: left"> Elk ate 3 kilograms of dog food and barked! </ Span>
In fact, to define a class and id, there is a simpler syntax (very similar to CSS):
And to create divs with the right class, it's still easier:
Now interject here Ruby. To execute a line of code (equivalent to <%%> in Erb), put a "-" in front of it:
To turn the result into a string, use the operator "=":
Displays the current time.
Of course, you can use = bundled with tag operators:
And the last thing we need to know for writing our Haml template is the "!!!" operator. By default, it creates a DOCTYPE Transitional. And we don’t need more of it.
I got the following template to display a list of files and a new file upload form:
I can not fail to note that the resulting pattern is very elegant (especially if you have a text editor with the backlight of the Haml syntax).
Let's imagine that we may also need other views for our application (and we will need them) - we will not write a common header with "!!!", "% html" and so on in everyone. To solve this problem in Sinatra, as in Rails, there is a layouts mechanism. A layout can be viewed as a top-level template — the contents of the template are inserted into the layout and together they go to the user.
To use the layout, we just need to place the layout.haml file in the views folder and it will automatically start being used by our application. By the way, we could also create layout layout.erb, which would be used for Erb templates.
The following code will be transferred from the template to the layout:
New here is only the line "= yield" at the end. It, as again in the case of Erb layouts, causes the Haml template to be processed in the place of its call (that is, inside a div with the main class). The code that we placed in layout.haml, of course, must be removed from the list.haml.
Everything! Now we are solid people and use Haml as a language for templates. The only thing left to do is to tell Sinatra that it is necessary to output not list.erb, but list.haml. To do this, go to our init.rb, and replace the line “erb: list” with “haml: list”. Now that's all - we use Haml and enjoy life.
Although, I’m probably ready to put some money on the fact that you didn’t succeed and Sinatra gave you an error message related to the data in the template or layout file. The thing is this: Haml works only with templates and layouts, in which 2 spaces are used as indentation - just spaces, not tabs, for example. And exactly 2. Correct it and, most likely, the error will disappear. In particular, if you copied the code directly from the page, then you definitely have tabs instead of spaces.
I have already explained why users should not be given the opportunity to know the id of the file they are downloading, so I will not dwell on this. Now it's time to solve this problem. As download addresses, we will use the forty - character SHA-1 digest (digest) on behalf of the file as its identifier. To do this, we do the following:
Add to stored_file.rb a new field to store the digest:
At the end of the file we have written “DataMapper.auto_upgrade!”, Which means that the database will be automatically updated by DataMapper.
Open the file init.rb and change all the blocks except the first one (get '/'):
It remains to change our template:
Done! Now our links look like this:
This is a very simple point. We will store the number of downloads and file size in the database, which means we need to expand our description of the class StoredFile:
Upload and download file now look like this:
We will display the new data as follows:
This section is not directly related to the study of basic technologies, but it does allow a little more to get used to the new “value system”. Agree that deleting the file without any confirmations is not the best solution even for home use: you can click randomly. Therefore, we add confirmation when trying to delete a file.
In the public folder, create a script.js file with the following elementary content:
Then we include this javascript file layout:
And in our template we make a small edit:
Thus, when you click the "Delete" button, we will be asked if we are sure, and this is an extra chance to come to its senses.
Let's return to the original formulation of the problem: a file sharing service with a convenient interface, running on a home server. While at home, it’s not so convenient to go to a “website” and upload a file — it’s much easier to drop it into a shared folder so that it will be added to our list.
Luckily, this is easy if we use Rake - the equivalent of Make for Ruby. This utility allows you to create so-called tasks (tasks) that are executed with the rake command TASK NAME. When launched, rake searches for the Rakefile file in the current directory and executes the commands specified in it, then searches for the task with the specified name and executes it.
What is rake task? This is nothing more than a specially designed Ruby code. In our case, the rake task will take all the files from the manual directory and simulate their loading in the usual way (that is, copy them into the files folder and create the necessary records in the database).
In case you suddenly have no Rake (which is unlikely), the installation is done in the traditional way for Ruby:
So, our Rakefile file that we created at the beginning tells Rake that it needs to load all the files with the .rake extension from the lib / tasks directory. So it is there that we create our file, which we call manual_monitor.rake. We will write the following to it:
First we connect ftools (file utilities) and our stored_file.rb (describing the StoredFile class - we will need to work with the database. Next is the description of the check_files task. As you might guess, in general, the task is described as:
The optional command desc in front of the task allows you to set an arbitrary text description for the task (you can view it with the command “rake --tasks”)
Inside the task itself, we do almost the same thing as in the code that loads the file through the browser, but with a couple of differences:
You can check how our task works by placing a file in the manual folder and invoking the command
Rake should report that the file was copied successfully. Through the web interface, we can make sure that it is.
Now we need to learn how to call our team automatically. I considered two options: through crontab and through a Ruby daemon. You and I, of course, are perverts and outcasts, but in this case I didn’t reinvent the wheel (and just use up system resources) and just used a crontab. However, if someone is interested in the launch option through the Ruby daemon, I can tell you about it.
My crontab looks like this
That is, the command is executed every minute.
Initially, I planned to meet one article, but practice has shown that even two are few. In the third part, I plan to talk about:
Right now I have written a list and I already doubt that it will be enough for me to have three parts ... But if you want to learn something else - write in the comments, we will understand :)
Once again, thank you for your attention!
Preliminary preparation
In this section, we will prepare the ground for all future research.
So, last time we got an application with the following file structure:
/
myapp.rb
views /
list.erb
files /
')
In this case, all the code is in the same file myapp.rb. In my opinion, this is very inconvenient and ugly, so this time we will start by creating folders and files, some of which will be useful to us in the course of the story. I have the following structure:
/
files /
lib /
tasks /
manual /
public /
test /
views /
In the files folder we have files that are accessible from the outside. In the lib folder there are libraries. The tasks folder will contain rake-tasks. In the manual folder, we will put the files, bypassing the web interface. Public - storage of files sent to the browser directly. In the test, we put, oddly enough, tests. Finally, views are already familiar to us - there are templates for generating HTML pages (as long as we have only one).
Now let's install Sinatra locally (directly to the folder with the application) - this will allow us not to worry about installing Sinatra gem. To do this, download the Sinatra code from the git repository. To do this, you must have git installed:
cd lib git clone git: //github.com/bmizerany/sinatra.git rm -rf sinatra / .git rm sinatra / .gitignore cd ..
We will place the main code of our application in the init.rb file, having previously changed its beginning to this:
require 'rubygems' require File.expand_path (File.dirname (__ FILE__) + '/ lib / sinatra / lib / sinatra') require File.expand_path (File.dirname (__ FILE__) + '/ lib / config') require 'ftools' require 'dm-core' require 'dm-validations' require 'dm-timestamps'
This code connects Sinatra and the configuration file config.rb from the lib folder - we’ll now create and write the following code in it (not doing anything yet):
configure do # Here will be the configuration settings for Sinatra end
Then Rakefile comes in handy - a special file for the rake utility, which tells it where to get the tasks. A file named Rakefile (without extension) located in the root directory should contain the following:
require 'rake'
require 'rake / testtask'
require 'rake / rdoctask'
Dir ["# {File.dirname (__ FILE __)} / lib / tasks / ** / *. Rake"]. Sort.each {| ext | load ext} In it, we simply load all the rake files from the lib / tasks directory and its subdirectories (the regular expression /**/*.rake is used for this).
Separate the code
The application code that lies in one file is not only inconvenient, but also wrong :) Therefore, now we will start by putting our StoredFile class into a separate one, sorry for the pun, file. In the lib folder we should have the file stored_file.rb with the following contents:
require 'dm-core'
require 'dm-validations'
require 'dm-timestamps'
DataMapper.setup (: default, "sqlite3: // # {Dir.pwd} /files.sqlite3")
class StoredFile
include DataMapper :: Resource
property: id, Integer,: serial => true # primary serial key
property: filename, String,: nullable => false # cannot be null
property: created_at, DateTime
default_scope (: default) .update (: order => [: created_at.desc])
end
DataMapper.auto_upgrade!
And at the end of the countless require in init.rb we add the line
require 'lib / stored_file'
The functionality of our application has not changed a bit, but now it looks a little more decent.
Viva la Haml!
In the first version of the application, we used the language Erb (Embedded RuBy) for generating HTML pages, which is HTML-code with included fragments of Ruby-code. This approach is traditional in Rails, but we are engaged in marginal solutions :) In this regard, I decided to use Haml as a language for templates.
Haml is a markup language that preaches beauty, as the main deity of the pattern. Indeed, the Haml code, in my opinion, looks nicer (and they say that it works faster). Another important feature of Haml is that it will not allow you not to close the tag (because you will not use explicit HTML tags at all)
Installation is simple:
sudo gem install haml
Let's create the list.haml file in our views directory and fill it with content. Haml is similar to Python in that it uses indentation in the code as block delimiters (in HTML, closing tags are used for this, and in Erb, the end keyword is used). So the html code
<div> <span> Some text </ span> </ div> <ul> <li> Item 1 </ li> <li> Item 2 </ li> </ ul>
On Haml looks like this:
% div % span some text % ul % li Item 1 % li Item 2
That, you see, is much shorter.
A little about the syntax, which is useful to us:
% TAG
Creates a <TAG /> or <TAG> </ TAG> tag - depending on the tag (for hr the first option will be chosen, for a - the second one)
% TAG CONTENT
Creates a <TAG> CONTENT </ TAG> tag.
% TAG CONTENT
Turns into <tag> CONTENT <tag>. In this case, CONTENT may also contain the operator% and any other operators.
The syntax for defining attributes is the same as for Ruby hashes:
% span {: class => "header",: id => "news_135",: style => "float: left"} The elk ate 3 kilograms of dog food and barked! It will become <span class = "header" id = "news_135" style = "float: left"> Elk ate 3 kilograms of dog food and barked! </ Span>
In fact, to define a class and id, there is a simpler syntax (very similar to CSS):
% span.header # news_135 Another Yellow Title
And to create divs with the right class, it's still easier:
.content =% div.content =% div {: class => "content"} = <div class = "content"> Now interject here Ruby. To execute a line of code (equivalent to <%%> in Erb), put a "-" in front of it:
- str = @ my_object.get_some_string
To turn the result into a string, use the operator "=":
- now = Time.now = now
Displays the current time.
Of course, you can use = bundled with tag operators:
% span # current_time = Time.now
And the last thing we need to know for writing our Haml template is the "!!!" operator. By default, it creates a DOCTYPE Transitional. And we don’t need more of it.
We write a template
I got the following template to display a list of files and a new file upload form:
!!!
% html {: xmlns => "http://www.w3.org/1999/xhtml"}
% head
% title faylopomoyka named after me
% meta {: "http-equiv" => "content-type",: content => "text / html; charset = UTF-8"}
% body
.main
- if @ files.empty?
% h1 No files.
- else
.list
% h1 File List
% table {: cellspacing => 0}
% tr
% th file
% th Uploaded
% th Remove
- @ files.each do | file |
% tr
% td.filename
% a {: href => "/#{file.id}",: title => file.filename} = file.filename
% td.created_at = file.created_at.strftime ("% d% b")
% td.delete
% a {: href => "/#{file.id}/delete",: title => "Delete"} Delete
.upload
% h1 add
% form {: name => "new_file",: enctype => "multipart / form-data",: method => "post",: action => "/"}
% input {: name => "file",: type => "file"}
% br
% input {: type => "submit",: value => "Download"}
I can not fail to note that the resulting pattern is very elegant (especially if you have a text editor with the backlight of the Haml syntax).
Separate the layout from the template
Let's imagine that we may also need other views for our application (and we will need them) - we will not write a common header with "!!!", "% html" and so on in everyone. To solve this problem in Sinatra, as in Rails, there is a layouts mechanism. A layout can be viewed as a top-level template — the contents of the template are inserted into the layout and together they go to the user.
To use the layout, we just need to place the layout.haml file in the views folder and it will automatically start being used by our application. By the way, we could also create layout layout.erb, which would be used for Erb templates.
The following code will be transferred from the template to the layout:
!!!
% html {: xmlns => "http://www.w3.org/1999/xhtml"}
% head
% title faylopomoyka named after me
% meta {: "http-equiv" => "content-type",: content => "text / html; charset = UTF-8"}
% body
.main
= yield New here is only the line "= yield" at the end. It, as again in the case of Erb layouts, causes the Haml template to be processed in the place of its call (that is, inside a div with the main class). The code that we placed in layout.haml, of course, must be removed from the list.haml.
Everything! Now we are solid people and use Haml as a language for templates. The only thing left to do is to tell Sinatra that it is necessary to output not list.erb, but list.haml. To do this, go to our init.rb, and replace the line “erb: list” with “haml: list”. Now that's all - we use Haml and enjoy life.
Although, I’m probably ready to put some money on the fact that you didn’t succeed and Sinatra gave you an error message related to the data in the template or layout file. The thing is this: Haml works only with templates and layouts, in which 2 spaces are used as indentation - just spaces, not tabs, for example. And exactly 2. Correct it and, most likely, the error will disappear. In particular, if you copied the code directly from the page, then you definitely have tabs instead of spaces.
Hiding links from prying eyes
I have already explained why users should not be given the opportunity to know the id of the file they are downloading, so I will not dwell on this. Now it's time to solve this problem. As download addresses, we will use the forty - character SHA-1 digest (digest) on behalf of the file as its identifier. To do this, we do the following:
Add to stored_file.rb a new field to store the digest:
property: sha, String
At the end of the file we have written “DataMapper.auto_upgrade!”, Which means that the database will be automatically updated by DataMapper.
Open the file init.rb and change all the blocks except the first one (get '/'):
post '/' do
tempfile = params ['file'] [: tempfile]
filename = params ['file'] [: filename] digest = Digest :: SHA1.hexdigest (filename) # Calculate digest
@file = StoredFile.create: filename => filename,: sha => digest # Write the file name and search in the database
File.copy (tempfile.path, "./files/#{@file.id}.upload")
redirect '/'
end
# download file
get '/: sha' do # now look for files by sha value, not by id
@file = StoredFile.first: sha => params [: sha]
send_file "./files/#{@file.id}.upload",: filename => @ file.filename,: type => 'Application / octet-stream'
end
# delete file
get '/: sha / delete' do # and delete also by sha, not by id
@file = StoredFile.first: sha => params [: sha]
File.delete ("./ files/#{@file.id}.upload")
@ file.destroy
redirect '/'
end It remains to change our template:
% td.filename
% a {: href => "/ # {<strong> file.sha </ strong>}",: title => file.filename} = file.filename
% td.created_at = file.created_at.strftime ("% d% b")
% td.delete
% a {: href => "/#{<strong>file.sha</strong>}/delete",: title => "Delete"} Delete
Done! Now our links look like this:
localhost: 4567 / 1c62e8aa8072c8a3cd5df1ba49bb4513bc1d8a88
Add download count and file size
This is a very simple point. We will store the number of downloads and file size in the database, which means we need to expand our description of the class StoredFile:
property: downloads, Integer,: default => 0 property: filesize, Integer,: nullable => false
Upload and download file now look like this:
post '/' do
tempfile = params ['file'] [: tempfile]
filename = params ['file'] [: filename]
digest = Digest :: SHA1.hexdigest (filename)
@file = StoredFile.create: filename => filename,: sha => digest,: filesize => File.size (tempfile.path)
File.copy (tempfile.path, "./files/#{@file.id}.upload")
redirect '/'
end
get '/: sha' do
@file = StoredFile.first: sha => params [: sha]
@ file.downloads + = 1
@ file.save
send_file "./files/#{@file.id}.upload",: filename => @ file.filename,: type => 'Application / octet-stream'
end
We will display the new data as follows:
% tr
% th file
% th Uploaded
% th Downloaded
% th Remove
- @ files.each do | file |
% tr
% td.filename
% a {: href => "/#{file.sha}",: title => file.filename} = file.filename
= "(# {file.filesize / 1024} Kb)"
% td.created_at = file.created_at.strftime ("% d% b")
% td.downloads = file.downloads
% td.delete
% a {: href => "/#{file.sha ı/delete",: title => "Delete"} Delete Some javascript
This section is not directly related to the study of basic technologies, but it does allow a little more to get used to the new “value system”. Agree that deleting the file without any confirmations is not the best solution even for home use: you can click randomly. Therefore, we add confirmation when trying to delete a file.
In the public folder, create a script.js file with the following elementary content:
function ORLY () {
return confirm ('Are you sure');
}
Then we include this javascript file layout:
% meta {: "http-equiv" => "content-type",: content => "text / html; charset = UTF-8"}
% script {: type => "text / javascript",: src => "/script.js"}
% body And in our template we make a small edit:
% td.delete
% a {: href => "/#{file.sha-/delete",: title => "Delete",: onclick => "return ORLY ();"} Delete
Thus, when you click the "Delete" button, we will be asked if we are sure, and this is an extra chance to come to its senses.
Automatically search and add new files.
Let's return to the original formulation of the problem: a file sharing service with a convenient interface, running on a home server. While at home, it’s not so convenient to go to a “website” and upload a file — it’s much easier to drop it into a shared folder so that it will be added to our list.
Luckily, this is easy if we use Rake - the equivalent of Make for Ruby. This utility allows you to create so-called tasks (tasks) that are executed with the rake command TASK NAME. When launched, rake searches for the Rakefile file in the current directory and executes the commands specified in it, then searches for the task with the specified name and executes it.
What is rake task? This is nothing more than a specially designed Ruby code. In our case, the rake task will take all the files from the manual directory and simulate their loading in the usual way (that is, copy them into the files folder and create the necessary records in the database).
In case you suddenly have no Rake (which is unlikely), the installation is done in the traditional way for Ruby:
sudo gem install rake
So, our Rakefile file that we created at the beginning tells Rake that it needs to load all the files with the .rake extension from the lib / tasks directory. So it is there that we create our file, which we call manual_monitor.rake. We will write the following to it:
require 'ftools'
require 'lib / datamapper / stored_file'
desc "Checks directory 'manual' and uploads all files from it"
task: manual_monitor do
Dir ["manual / *"]. Each do | path |
file = File.new (path)
filename = File.basename (file.path)
digest = Digest :: SHA1.hexdigest (filename)
stored_file = StoredFile.create: filename => filename,: sha => digest,: filesize => File.size (file.path)
File.move (path, "./files/#{stored_file.id}.upload")
puts "File # {path} succesfully uploaded. ID = # {stored_file.id}"
end
end
First we connect ftools (file utilities) and our stored_file.rb (describing the StoredFile class - we will need to work with the database. Next is the description of the check_files task. As you might guess, in general, the task is described as:
task: TASK_NAME do CODE end
The optional command desc in front of the task allows you to set an arbitrary text description for the task (you can view it with the command “rake --tasks”)
Inside the task itself, we do almost the same thing as in the code that loads the file through the browser, but with a couple of differences:
- We work with all files from the manual directory.
- We move the file from manual to files (so as not to process it again)
You can check how our task works by placing a file in the manual folder and invoking the command
rake manual_monitor
Rake should report that the file was copied successfully. Through the web interface, we can make sure that it is.
Now we need to learn how to call our team automatically. I considered two options: through crontab and through a Ruby daemon. You and I, of course, are perverts and outcasts, but in this case I didn’t reinvent the wheel (and just use up system resources) and just used a crontab. However, if someone is interested in the launch option through the Ruby daemon, I can tell you about it.
My crontab looks like this
* / 1 * * * * cd / path / to / my / app /; rake manual_monitor
That is, the command is executed every minute.
Whew!
Initially, I planned to meet one article, but practice has shown that even two are few. In the third part, I plan to talk about:
- Normal authorization (to allow everyone to download files, but forbid anyone to download and delete them)
- Auxiliary methods (helpers)
- How to add a delay when downloading (so that you can make sure that you have come to the right link and do not download a hundred megabyte file via GPRS)
- Configuration
- Creating CSS with SASS
- Code testing
- Performance scores
Right now I have written a list and I already doubt that it will be enough for me to have three parts ... But if you want to learn something else - write in the comments, we will understand :)
Once again, thank you for your attention!
Source: https://habr.com/ru/post/50084/
All Articles