Zendframework + ffmpeg + gearman + amazon = Video encoder service
I want to share with the community about the experience of creating a web service for video conversion and saving to the cloud storage. Immediately make a reservation, the service is written for internal use by a single European company, and has been operating for more than 6 months. One of the company's trends is the WebTv product. It was very problematic to deploy a structure for converting video on each new site, and often these processes very slow down the server.
It was decided to create a service that would satisfy the following requirements:
The most used scheme is video loading by POST. On the client site, a flash downloader is built into the form of adding a new video, which sends the file directly to the video service along with the authorization data. The diagram below shows all the steps in detail:
')
Authentication
Because login and password for authorization are available in html code, and this is not good, I had to think of something. The solution was the use of one-time passwords with a limited lifespan and a two-step authentication system. The client service generates a password and saves it to itself in the database (as well as the generation time and the marker is used / not used). After receiving the request, the video service knocks on the client server, according to a predetermined URL, and receives confirmation of the current password.
Video check
Naturally, we cannot blindly believe that the downloaded file is really a video file.
With the help of the ffmpeg_movie class, we are trying to find out information about the video file, if we failed to return an error to the client server, indicating that the file cannot be used.
Actually there was a task after saving the request to the database to start converting the video. Naturally, the good old cron here is poorly suited, run once a minute, consistently executed. Before that, I already used Gearman and my choice fell on this tool. In fact, I had to tinker with him, for some reason at first, I didn’t want to work as I should have:
Workers constantly returned -1 and died. I remembered that in previous projects, the workers did a little differently. And inherited from GearmanWorker, made the base layer for all types of worker:
and then already collected the necessary workers:
Next, another not pleasant thing came out, the connection with the DB was closed after a certain period of time, so I decided that I would close the connection before the conversion process began:
Statistics show that it is enough to drive 3-4 workers to properly serve up to 10 requests per hour. Read more about Gearman here Gearman.org
I have a lot of sleepless nights with this “fruit”. At first, everything was trivial, I had already debugged the standard queries, like doing resize, bitrates and all that - everything is trivial. The most interesting thing started when it was necessary to resize and rotate in a single pass, and add a mask to broadcast and paddings, in general, I didn’t want to work instead. Vfilters came to help , which were added relatively recently. The beauty is that you can consistently apply filters, i.e. For example, first turn, remove, add padding, hang water signs. Here is the typical command for ffmpeg in my service:
Because each request requires the creation of two flv + mp4 videos, then at the end of the work, the data is synchronized and the worker whose work is finished later puts in turn the task of downloading the finished data to amazon. If there were errors during conversion, we notify the server about it.
Another type of Gearman worker is loading the finished data on Amazon S3 using the standard Zend library for working with Amazon api.
In general, I want to say that over the past year, Amazon has been frustrating several times, for example, a manager from Europe writes to me that they say his video does not play in the front, and I write to him, they say how I’m sitting right here watching this video that you have does not work. Then it came to, we appeal to different data centers Amazon. And there are also problems when downloading large videos, about once a month, some video and not loading, and the Amazon server will return some vague error code.
About Amazon S3, you can read more here Amazon Simple Storage .
The on-demand service is able to hang watermarks on the video, can rotate the video, correctly resize according to the specified parameters, filling the empty space with black. To do this, use the vfilters that come with ffmpeg in the latest versions.
Able to collect video from ftp, imap, amazon s3 storage and also convert it, further notifying the server (But these methods are used extremely rarely).
Actually a curious thing about IMAP, it is about collecting videos sent via MMS. I did not go into the details of the implementation, my business was just to get into the mail using the IMAP protocol to find new messages, collect all the video files from them and process them.
And now the company is thinking about another goodness. Screw Asteriks , and for example, the client makes a video call from his phone that supports 3G, leaves a video message, through Asteriks, this thing gets into a special daddy. Every 5 minutes Video service via FTP comes to it, if it has found something new, processes it and returns it to the client’s server, ready for video use.
The last 3 months the server has been running steadily, and it never fell. For half a year, processed more than 4,000 requests.
It is interesting to hear opinions, comments, constructive criticism.
It was decided to create a service that would satisfy the following requirements:
- Easy integration with any site.
- Implement the process of converting more than one video file at the same time.
- Implement the process of collecting video from ftp, IMAP and just directly receive the video file via HTTP POST.
- Optionally with the help of additional parameters, to be able to rotate the video, broadcast watermarks, etc.
- Implement a secure authorization system in the front, where users can see what data goes to the video service in addition to the video file.
- Send ready results to Amazon S3.
- Properly handle errors, and notify the client service.
The simplest scheme of integration with the service.
The most used scheme is video loading by POST. On the client site, a flash downloader is built into the form of adding a new video, which sends the file directly to the video service along with the authorization data. The diagram below shows all the steps in detail:
Check data and video on the service side.
')
Authentication
Because login and password for authorization are available in html code, and this is not good, I had to think of something. The solution was the use of one-time passwords with a limited lifespan and a two-step authentication system. The client service generates a password and saves it to itself in the database (as well as the generation time and the marker is used / not used). After receiving the request, the video service knocks on the client server, according to a predetermined URL, and receives confirmation of the current password.
Video check
Naturally, we cannot blindly believe that the downloaded file is really a video file.
With the help of the ffmpeg_movie class, we are trying to find out information about the video file, if we failed to return an error to the client server, indicating that the file cannot be used.
Gearman
Actually there was a task after saving the request to the database to start converting the video. Naturally, the good old cron here is poorly suited, run once a minute, consistently executed. Before that, I already used Gearman and my choice fell on this tool. In fact, I had to tinker with him, for some reason at first, I didn’t want to work as I should have:
function converter() { // } $worker = new GearmanWorker(); $worker->addFunction("convert", "converter"); // .. Workers constantly returned -1 and died. I remembered that in previous projects, the workers did a little differently. And inherited from GearmanWorker, made the base layer for all types of worker:
class App_Model_Worker_Abstract extends GearmanWorker { public function __construct($gearman) { parent::__construct(); if (!$this->addServer($gearman['host'], $gearman['port'])) { throw new Exception("Can't connect to Gearman"); } } public function work() { while (parent::work()); return true; } } and then already collected the necessary workers:
// class App_Model_Worker_Converter extends App_Model_Worker_Abstract { public function __construct($gearman) { parent::__construct($gearman); // } public function action($job) { $fileId = $job->workload(); $file = $this->_modelFileResponse->getPrepareToConvertFile($fileId); // . } } //, class App_Model_Worker_SendResponse extends App_Model_Worker_Abstract { public function __construct($gearman) { parent::__construct($gearman); } public function action($job) { $responseId = $job->workload(); // . } } Next, another not pleasant thing came out, the connection with the DB was closed after a certain period of time, so I decided that I would close the connection before the conversion process began:
$model->getAdapter()->closeConnection(); Statistics show that it is enough to drive 3-4 workers to properly serve up to 10 requests per hour. Read more about Gearman here Gearman.org
Ffmpeg
I have a lot of sleepless nights with this “fruit”. At first, everything was trivial, I had already debugged the standard queries, like doing resize, bitrates and all that - everything is trivial. The most interesting thing started when it was necessary to resize and rotate in a single pass, and add a mask to broadcast and paddings, in general, I didn’t want to work instead. Vfilters came to help , which were added relatively recently. The beauty is that you can consistently apply filters, i.e. For example, first turn, remove, add padding, hang water signs. Here is the typical command for ffmpeg in my service:
/usr/local/bin/ffmpeg -i "/home/encoderws/public_html/data/files/requests/example-test-5c2a1c8a2e9e540690ba05670a526872_d917a8705cade8efdd87008df20ef19c" -acodec libfaac -vcodec mpeg4 -b 400k -ar 44100 -f mp4 -ab 96k -ac 2 -vf "[in] rotate=90 [rt0], [rt0] scale=203:360 [sc0], [sc0] pad=640:360:219 [pd0], [pd0] scale=640:360 [sc1], movie=0:png:/home/encoderws/public_html/data/files/watermarks/watermark.png [wm], [sc1] [wm] overlay=0:0:1 [out]" -y '/home/encoderws/public_html/data/files/responses/example-test-5c2a1c8a2e9e540690ba05670a526872_d917a8705cade8efdd87008df20ef19c.mp4
Because each request requires the creation of two flv + mp4 videos, then at the end of the work, the data is synchronized and the worker whose work is finished later puts in turn the task of downloading the finished data to amazon. If there were errors during conversion, we notify the server about it.
Amazon
Another type of Gearman worker is loading the finished data on Amazon S3 using the standard Zend library for working with Amazon api.
In general, I want to say that over the past year, Amazon has been frustrating several times, for example, a manager from Europe writes to me that they say his video does not play in the front, and I write to him, they say how I’m sitting right here watching this video that you have does not work. Then it came to, we appeal to different data centers Amazon. And there are also problems when downloading large videos, about once a month, some video and not loading, and the Amazon server will return some vague error code.
About Amazon S3, you can read more here Amazon Simple Storage .
Other goodies
The on-demand service is able to hang watermarks on the video, can rotate the video, correctly resize according to the specified parameters, filling the empty space with black. To do this, use the vfilters that come with ffmpeg in the latest versions.
Able to collect video from ftp, imap, amazon s3 storage and also convert it, further notifying the server (But these methods are used extremely rarely).
Actually a curious thing about IMAP, it is about collecting videos sent via MMS. I did not go into the details of the implementation, my business was just to get into the mail using the IMAP protocol to find new messages, collect all the video files from them and process them.
And now the company is thinking about another goodness. Screw Asteriks , and for example, the client makes a video call from his phone that supports 3G, leaves a video message, through Asteriks, this thing gets into a special daddy. Every 5 minutes Video service via FTP comes to it, if it has found something new, processes it and returns it to the client’s server, ready for video use.
The last 3 months the server has been running steadily, and it never fell. For half a year, processed more than 4,000 requests.
It is interesting to hear opinions, comments, constructive criticism.
Source: https://habr.com/ru/post/132433/
All Articles