Background video processing in Ruby on Rails

Suppose you have a website where users upload videos, and this video needs to be converted to flv format.
It is not good to do this directly after loading, in the current workflow, since The number of streams is limited and the site will be inaccessible under heavy load.
It will be much better if the daemon takes care of processing these video files.

The logic of the daemon can be as follows - the daemon selects the raw object from the database (for example, using the state flag, and sorting by date), then converts the corresponding video file and updates the record in the database, setting the status to "processed" (or "processing error") .
To write your own demon, you can use the daemons library (oddly enough). The daemon will consist of 2 parts: a script to control the background process, and a script with the demon's logic. In our case, these will be the scripts / video_converter_control and lib / video_converter files.

scripts / video_converter_control:

require 'rubygems'
require 'daemons'

options = {:app_name => "video_converter",
:ARGV => ARGV,
:dir_mode => :script,
:dir => '../tmp/pids',
:log_output => false,
:multiple => false,
:ontop => false,
:mode => :load,
:backtrace => false,
:monitor => false
# daemons ( monitor), monit
}

Daemons.run(File.join(File.dirname(__FILE__), '../lib/video_converter.rb'), options)


lib / video_converter:

require File.dirname(__FILE__) + '/../config/boot'
require RAILS_ROOT + "/config/environment"

SLEEP_TIME = 1

logger = RAILS_DEFAULT_LOGGER

loop do
video = Video.find(:first, :conditions => {:state => Video::PENDING_STATE}, :order => "id ASC")
success = video.process
if success.empty?
video.update_attribute(:state, Video::CONVERTED_STATE)
else
logger.info "VideoConverter: video #{video.id} converting failure"
video.update_attribute(:state, Video::FAILURE_STATE)
end
sleep(SLEEP_TIME)
end


')
You can work with the daemon as follows:

estarter@ny $ ./script/video_converter_control status
video_converter: no instances running
estarter@ny $ ./script/video_converter_control start
estarter@ny $ ./script/video_converter_control status
video_converter: running [pid 5957]
estarter@ny $ ./script/video_converter_control stop
estarter@ny $ ./script/video_converter_control status
video_converter: no instances running


Now let's increase the “power” of the converter, since It is possible and necessary to process several video files simultaneously. The easiest option is to run several demons, however, this causes a synchronization problem, since it cannot be assumed that several processes would simultaneously process one file. This can be solved, for example, by “blocking” an entry in the database by setting the corresponding flag (Video :: PROCESSING_STATE).
But if processing occurs only on one server, you can get smarter - do the thread in the main daemon.
For these purposes, the spawn plugin is suitable, which allows you to simply create and work with streams. Now the daemon will select several objects at once, and process them in separate threads.

lib / video_converter:

require File.dirname(__FILE__) + '/../config/boot'
require RAILS_ROOT + "/config/environment"
include Spawn

THREAD_COUNT = 5 # ,
SLEEP_TIME = 1

logger = RAILS_DEFAULT_LOGGER
spawn_ids = []

loop do
videos = Video.find(:all, :conditions => {:state => Video::PENDING_STATE}, :order => "id ASC", :limit => THREAD_COUNT)
videos.each_with_index do |video, idx|
logger.info "VideoConverter: convert #{videos.size} videos"
spawn_ids[idx] = spawn do
success = video.process
if success.empty?
video.update_attribute(:state, Video::CONVERTED_STATE)
else
logger.info "VideoConverter: video #{video.id} converting failure"
video.update_attribute(:state, Video::FAILURE_STATE)
end
end
end
wait(spawn_ids)
sleep(SLEEP_TIME)
end


This approach can be used for a number of resource-intensive tasks.

Original - in my blog.