Creating a full-fledged video hosting with their own hands (nginx + php5-fpm + ffmpeg + cumulusclips)
Good afternoon, habrovchane!
Recently, our company has a need to create its own video resource, closed, but at the same time a bit public. And finally, it is finished and I am ready to share knowledge and applications.
The task was as follows:
Create a video resource capable of conducting multi-stream one-way broadcasts from a web camera, as well as from any file (for example, to protect against direct download), a video game with the ability to view video in different formats and bitrates.
The basis was laid free server! Not very powerful, but pretty suitable.
')
Intel® Xeon® CPU L5520 @ 2.27GHz
number of cores 16
16372 MB RAM
A little run ahead, when decoding a video processor load reaches 500% (approximately 6 cores);
Let's start from the very beginning, I chose Ubuntu Server 13.04 x64 from the OS, because I spend more time with it and I actually understand it better than other Linux families.
As a WEB server, I chose the nginx + php5-fpm bundle, because nginx quite successfully handles loads, as well as video output.
nginx is installed by default without a streaming module, so we set of sorts
Required dependencies for building packages:
Download the source:
Download the necessary modules for streaming:
For convenience, create an installation script:
with content
Perhaps during the compilation process errors may occur. Therefore, we do this:
In the auto / cc / gcc file, we comment the line:
Run:
After installation, create the necessary symlinks and directories (if not created):
This completes the installation of nginx and php5-fpm.
Go back to the setting later.
Next in line is ffmpeg. Installing apt-get is not desirable, the project is already deprecatet and much refuses to work. In search of adequate and more recent instructions, I spent almost 2 nights. I will not hide, I found a pretty good package for installation.
Oddly enough, the project is called www.ffmpeginstaller.com, and even with all the installers in the public domain, it offers its services for 50 bucks.
And everything is done quite simply.
Download the package:
After the first installation, I realized ... there is not enough codec. Let off <<< Put the codec before installation:
Open ffmpeg.sh and add the "--enable-libvpx" codec to the installation:
This completes the preparation. You can install: sh start.sh
Installation takes about 15-20 minutes, depending on the capabilities of the iron. You can go to drink tea (or coffee).
After installation we perform:
Clip view in PS
Everything. Congratulations, we did it!
Now we need a CMS to manage these tools.
There were few options, or rather just 2 (and only one — cumulusclips — was suitable for my needs).
The source code is clear, without much hassle figured out with the component. That's just had to rewrite the project from mysql to mysqli. All the CMS code is structured and the templates are separate and flexible. For the basis of the chosen template pseudo YouTube.
I had to completely twist the player, because jwplayer was unable to switch video streams. A little polaziv on github found a simple player under the simple name jQplayer.
This player is able to switch streams easily. True, there is one drawback. Playing a file starts from the beginning. And this was not a problem - video files are easily cut nginx out of the box.
Now we need to set up web hosting for our project.
I attach a small script to automate this process. Included with CMS was .htaccess - and nginx refuses to understand it, so I rewrote it to the needs of this web server.
That's all. This configuration can encode video in different formats, as well as stream the stream. If this article is of interest to anyone, I will gladly give live examples of streaming.
Unfortunately, I can’t show the full stuffing, but here’s what I got at stream.etagi.com
Thanks for attention.
PS1 Found a more relaxed build path ffmpeg. Corrected below.
1. echo “deb www.deb-multimedia.org main non-free squeeze” /etc/apt/sources.list
2. apt-get update
3. apt-get install deb-multimedia-keyring
4. apt-get update
5. apt-get remove ffmpeg
6. apt-get purge ffmpeg
7. apt-get autoremove
8. apt-get install ffmpeg x264 (the installation will tighten all necessary dependencies)
9. ffmpeg -version
(in the output should not be DEPRECATED, if not from 5 to 9 point again)
PS2 Main design feature
Only works on PHP 5.3 from dotdeb
Recently, our company has a need to create its own video resource, closed, but at the same time a bit public. And finally, it is finished and I am ready to share knowledge and applications.
The task was as follows:
Create a video resource capable of conducting multi-stream one-way broadcasts from a web camera, as well as from any file (for example, to protect against direct download), a video game with the ability to view video in different formats and bitrates.
The basis was laid free server! Not very powerful, but pretty suitable.
')
Intel® Xeon® CPU L5520 @ 2.27GHz
number of cores 16
16372 MB RAM
A little run ahead, when decoding a video processor load reaches 500% (approximately 6 cores);
Let's start from the very beginning, I chose Ubuntu Server 13.04 x64 from the OS, because I spend more time with it and I actually understand it better than other Linux families.
As a WEB server, I chose the nginx + php5-fpm bundle, because nginx quite successfully handles loads, as well as video output.
nginx is installed by default without a streaming module, so we set of sorts
Required dependencies for building packages:
apt-get install build-essential checkinstall subversion unzip yamdi imagemagick php5-curl libssl-dev zlib1g-dev libpcre3-dev rpl php5-fpm git Download the source:
cd /tmp wget http://nginx.org/download/nginx-1.5.2.zip unzip nginx-1.5.2.zip -d nginx/ rm -f nginx-1.5.2.zip cd nginx Download the necessary modules for streaming:
mkdir modules git clone https://github.com/masterzen/nginx-upload-progress-module.git modules/nginx-upload-progress-module wget http://www.kernel-video-sharing.com/files/nginx_mod_h264_streaming-2.3.2.zip unzip nginx_mod_h264_streaming-2.3.2.zip -d modules/ rm -f nginx_mod_h264_streaming-2.3.2.zip git clone https://github.com/arut/nginx-rtmp-module.git modules/nginx-rtmp-module For convenience, create an installation script:
touch nginx.sh nano nginx.sh with content
./configure \ --conf-path=/etc/nginx/nginx.conf \ --error-log-path=/var/log/nginx/error.log \ --pid-path=/var/run/nginx.pid \ --lock-path=/var/lock/nginx.lock \ --http-log-path=/var/log/nginx/access.log \ --http-client-body-temp-path=/var/lib/nginx/body \ --http-proxy-temp-path=/var/lib/nginx/proxy \ --http-fastcgi-temp-path=/var/lib/nginx/fastcgi \ --with-debug \ --with-http_stub_status_module \ --with-http_secure_link_module \ --with-http_gzip_static_module \ --with-http_realip_module \ --with-http_mp4_module \ --with-http_flv_module \ --with-http_ssl_module \ --with-http_dav_module \ --with-md5=/usr/lib \ --add-module=modules/nginx-upload-progress-module \ --add-module=modules/nginx-rtmp-module \ --add-module=modules/nginx_mod_h264_streaming-2.3.2 make -j16 (16 - . . "grep -c processor /proc/cpuinfo") checkinstall Perhaps during the compilation process errors may occur. Therefore, we do this:
In the auto / cc / gcc file, we comment the line:
#CFLAGS="$CFLAGS -Werror" Run:
sh nginx.sh After installation, create the necessary symlinks and directories (if not created):
ln -s /usr/local/nginx/sbin/nginx /usr/sbin/nginx mkdir -p /var/lib/nginx/body mkdir /var/lib/nginx/proxy mkdir /var/lib/nginx/fastcgi chown -R root /var/lib/nginx/ wget http://nginx-init-ubuntu.googlecode.com/files/nginx-init-ubuntu_v2.0.0-RC2.tar.bz2 tar -jxvf nginx-init-ubuntu_v2.0.0-RC2.tar.bz2 -C /etc/init.d/ chmod 715 /etc/init.d/nginx /usr/sbin/update-rc.d -f nginx defaults rm -f nginx-init-ubuntu_v2.0.0-RC2.tar.bz2 rpl 'DAEMON=/usr/local/sbin/nginx' 'DAEMON=/usr/local/nginx/sbin/nginx' /etc/init.d/nginx rpl 'NGINX_CONF_FILE="/usr/local/nginx/conf/nginx.conf"' 'NGINX_CONF_FILE="/etc/nginx/nginx.conf"' /etc/init.d/nginx This completes the installation of nginx and php5-fpm.
Go back to the setting later.
Next in line is ffmpeg. Installing apt-get is not desirable, the project is already deprecatet and much refuses to work. In search of adequate and more recent instructions, I spent almost 2 nights. I will not hide, I found a pretty good package for installation.
cd /tmp wget http://mirror.ffmpeginstaller.com/old/scripts/ffmpeg7/ffmpeginstaller.7.4.tar.gz tar -xzf ffmpeginstaller.7.4.tar.gz ffmpeg cd ffmpegapt-get install libvpx
nano ffmpeg.sh ./configure --prefix=$INSTALL_DDIR --enable-shared --enable-nonfree \ --enable-gpl --enable-pthreads --enable-libopencore-amrnb --enable-decoder=liba52 \ --enable-libopencore-amrwb --enable-libfaac --enable-libmp3lame \ --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid --enable-libvpx \ --extra-cflags=-I/usr/local/cpffmpeg/include/ --extra-ldflags=-L/usr/local/cpffmpeg/lib \ --enable-version3 --extra-version=syslint hash x264 ffmpeg ffplayClip view in PS
Everything. Congratulations, we did it!
Now we need a CMS to manage these tools.
There were few options, or rather just 2 (and only one — cumulusclips — was suitable for my needs).
The source code is clear, without much hassle figured out with the component. That's just had to rewrite the project from mysql to mysqli. All the CMS code is structured and the templates are separate and flexible. For the basis of the chosen template pseudo YouTube.
I had to completely twist the player, because jwplayer was unable to switch video streams. A little polaziv on github found a simple player under the simple name jQplayer.
This player is able to switch streams easily. True, there is one drawback. Playing a file starts from the beginning. And this was not a problem - video files are easily cut nginx out of the box.
Now we need to set up web hosting for our project.
I attach a small script to automate this process. Included with CMS was .htaccess - and nginx refuses to understand it, so I rewrote it to the needs of this web server.
An example of my config
#!/bin/bash echo -n " : " read host echo -n " nginx: " read users sap=/etc/nginx/sites-available/$host.conf mkdir -p /var/hosting/ touch $sap chmod 777 $sap directives="upstream backend-${host} {server unix:/var/run/php5-${host}.sock;} server { listen 80; server_name ${host} www.${host}; root /var/hosting/${host}/www; access_log /var/log/nginx/${host}-access.log; error_log /var/log/nginx/${host}-error.log; index index.php; rewrite_log on; if ($host = '${host}' ) { rewrite ^/(.*)$ http://www.${host}/$1 permanent; } location /im { rewrite ^/im/(.*)$ /cc-core/controllers/thumbs.php?$1 last; } location /videos { rewrite ^/videos/([0-9]+)/(.*)$ /cc-core/controllers/play.php?vid=$1 last; rewrite ^/videos/page/([0-9]+)/$ /cc-core/controllers/videos.php?page=$1 last; rewrite ^/videos/(most-recent|most-viewed|most-discussed|most-rated)/$ /cc-core/controllers/videos.php?load=$1 last; rewrite ^/videos/(most-recent|most-viewed|most-discussed|most-rated)/page/([0-9]+)/$ /cc-core/controllers/videos.php?load=$1&page=$2 last; rewrite ^/videos/([a-zA-Z0-9\-]+)/$ /cc-core/controllers/videos.php?category=$1 last; rewrite ^/videos/([a-zA-Z-]+)/page/([0-9]+)/$ /cc-core/controllers/videos.php?category=$1&page=$2 last; rewrite ^/videos/([0-9]+)/comments/$ /cc-core/controllers/comments.php?vid=$1 last; rewrite ^/videos/([0-9]+)/comments/page/([0-9]+)/$ /cc-core/controllers/comments.php?vid=$1&page=$2 last; rewrite ^/videos/$ /cc-core/controllers/videos.php last; } location /private { rewrite ^/private/get/$ /cc-core/controllers/play.php?get_private=true last; rewrite ^/private/videos/([a-zA-Z0-9]+)/$ /cc-core/controllers/play.php?private=$1 last; rewrite ^/private/comments/([a-zA-Z0-9]+)/$ /cc-core/controllers/comments.php?private=$1 last; rewrite ^/private/comments/([a-zA-Z0-9]+)/page/([a-z0-9]+)/$ /cc-core/controllers/comments.php?private=$1&page=$2 last; } location /members { rewrite ^/members/page/([0-9]+)/$ /cc-core/controllers/members.php?page=$1 last; rewrite ^/members/([a-zA-Z0-9]+)/$ /cc-core/controllers/profile.php?username=$1 last; rewrite ^/members/([a-zA-Z0-9]+)/videos/$ /cc-core/controllers/member_videos.php?username=$1 last; rewrite ^/members/([a-zA-Z0-9]+)/videos/page/([0-9]+)/$ /cc-core/controllers/member_videos.php?username=$1&page=$2 last; rewrite ^/members/$ /cc-core/controllers/members.php last; } location /search { rewrite ^/search(/page/([0-9]+))?/$ /cc-core/controllers/search.php?page=$2 last; } location /login { rewrite ^(.*)$ /cc-core/controllers/login.php last; } location /login/forgot { rewrite ^(.*)$ /cc-core/controllers/login.php?action=forgot last; } location /logout { rewrite ^(.*)$ /cc-core/system/logout.php last; } location /register { rewrite ^(.*)$ /cc-core/controllers/register.php last; } location /activate { rewrite ^(.*)$ /cc-core/controllers/activate.php last; } location /opt { rewrite ^/opt-out/$ /cc-core/controllers/opt_out.php last; } location /contact { rewrite ^(.*)$ /cc-core/controllers/contact.php last; } location /embed { rewrite ^/embed/([0-9]+)/$ /cc-core/system/embed.php?vid=$1 last; } location /page { rewrite ^(.*)$ /cc-core/system/page.php last; } location /translation { rewrite ^(.*)$ /cc-core/system/translation.php last; } location /notify { rewrite ^(.*)$ /cc-core/system/notify.php last; } location /language/get { rewrite ^(.*)$ /cc-core/system/language.php?get last; } location /language { rewrite ^/language/set/(.*)/$ /cc-core/system/language.php?set&language=$1 last; } location /feed { rewrite ^/feed(/([a-zA-Z0-9]+))?/$ /cc-core/system/feed.php?username=$2 last; } location /video { rewrite ^/video-sitemap(-([0-9]+))?\.xml$ /cc-core/system/video_sitemap.php?page=$2 last; } location /myaccount/upload/avatar { rewrite ^(.*)$ /cc-core/system/avatar.ajax.php last; } location /myaccount/upload/validate { rewrite ^(.*)$ /cc-core/system/upload.ajax.php last; } location /myaccount/grab/validate { rewrite ^(.*)$ /cc-core/system/grab.ajax.php last; } location /actions/username { rewrite ^(.*)$ /cc-core/system/username.ajax.php last; } location /actions/flag { rewrite ^(.*)$ /cc-core/system/flag.ajax.php last; } location /actions/favorite { rewrite ^(.*)$ /cc-core/system/favorite.ajax.php last; } location /actions/subscribe { rewrite ^(.*)$ /cc-core/system/subscribe.ajax.php last; } location /actions/rate { rewrite ^(.*)$ /cc-core/system/rate.ajax.php last; } location /actions/comment { rewrite ^(.*)$ /cc-core/system/comment.ajax.php last; } location /actions/post { rewrite ^(.*)$ /cc-core/system/post.ajax.php last; } location /actions/stream { rewrite ^(.*)$ /cc-core/system/stream.ajax.php last; } location /actions { rewrite ^/actions/mobile-(videos|search)/$ /cc-core/system/mobile_$1.ajax.php?mobile last; } location /myaccount { rewrite ^/myaccount/upload/complete/$ /cc-core/controllers/myaccount/upload_complete.php last; rewrite ^/myaccount/upload/video/$ /cc-core/controllers/myaccount/upload_video.php last; rewrite ^/myaccount/upload/$ /cc-core/controllers/myaccount/upload.php last; rewrite ^/myaccount/myvideos(/page/([0-9]+))?/$ /cc-core/controllers/myaccount/myvideos.php?page=$2 last; rewrite ^/myaccount/myvideos/([0-9]+)/$ /cc-core/controllers/myaccount/myvideos.php?vid=$1 last; rewrite ^/myaccount/editvideo/([0-9]+)/$ /cc-core/controllers/myaccount/edit_video.php?vid=$1 last; rewrite ^/myaccount/myfavorites(/page/([0-9]+))?/$ /cc-core/controllers/myaccount/myfavorites.php?page=$2 last; rewrite ^/myaccount/myfavorites/([0-9]+)/$ /cc-core/controllers/myaccount/myfavorites.php?vid=$1 last; rewrite ^/myaccount/privacy-settings/$ /cc-core/controllers/myaccount/privacy_settings.php last; rewrite ^/myaccount/change-password/$ /cc-core/controllers/myaccount/change_password.php last; rewrite ^/myaccount/subscriptions(/([0-9]+))?/$ /cc-core/controllers/myaccount/subscriptions.php?id=$2 last; rewrite ^/myaccount/subscriptions(/page/([0-9]+))?/$ /cc-core/controllers/myaccount/subscriptions.php?page=$2 last; rewrite ^/myaccount/subscribers(/page/([0-9]+))?/$ /cc-core/controllers/myaccount/subscribers.php?page=$2 last; rewrite ^/myaccount/message/inbox(/page/([0-9]+))?/$ /cc-core/controllers/myaccount/message_inbox.php?page=$2 last; rewrite ^/myaccount/message/inbox/([0-9]+)/$ /cc-core/controllers/myaccount/message_inbox.php?delete=$1 last; rewrite ^/myaccount/message/read/([0-9]+)/$ /cc-core/controllers/myaccount/message_read.php?msg=$1 last; rewrite ^/myaccount/message/send/([a-zA-Z0-9]+)/$ /cc-core/controllers/myaccount/message_send.php?username=$1 last; rewrite ^/myaccount/message/reply/([0-9]+)/$ /cc-core/controllers/myaccount/message_send.php?msg=$1 last; rewrite ^/myaccount/$ /cc-core/controllers/myaccount/myaccount.php last; } location /myaccount/profile { rewrite ^(.*)$ /cc-core/controllers/myaccount/update_profile.php last; } location /myaccount/profile/reset { rewrite ^(.*)$ /cc-core/controllers/myaccount/update_profile.php?action=reset last; } location /myaccount/message/send { rewrite ^(.*)$ /cc-core/controllers/myaccount/message_send.php last; } location /m { rewrite ^/m/v/([0-9]+)/$ /cc-core/controllers/mobile/play.php?mobile&vid=$1 last; rewrite ^/m/v/$ /cc-core/controllers/mobile/videos.php?mobile last; rewrite ^/m/s/$ /cc-core/controllers/mobile/search.php?mobile last; rewrite ^/m/$ /cc-core/controllers/mobile/index.php?mobile last; } location /system { rewrite ^/system-error/$ /cc-core/controllers/system_error.php last; } location /t { rewrite ^/t/(.*)$ /cc-core/system/translation.php last; } location / { if (!-e $request_filename){ #rewrite ^/(.*)$ /$request_uri/ permanent; rewrite ^/(.*)$ /cc-core/system/page.php last; } } location ~ \.php$ { include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_pass backend-${host}; } location ~* ^.+\.(jpg|jpeg|gif|css|png|js|ico|bmp)$ { access_log off; expires 10d; break; } location ~ \.(flv|mp4|webm|ogg|ogv|mp3)$ { mp4; mp4_buffer_size 1m; mp4_max_buffer_size 5m; } location ~ /\. { deny all; } } " echo "$directives">$sap sap_poll=/etc/php5/fpm/pool.d/$host.conf touch $sap_poll chmod 777 $sap_poll directives_poll="[${host}] listen = /var/run/php5-${host}.sock listen.mode = 0666 user = ${users} group = ${users} chdir = /var/hosting/${host} php_admin_value[upload_tmp_dir] = /var/hosting/${host}/tmp php_admin_value[soap.wsdl_cache_dir] = /var/hosting/${host}/tmp php_admin_value[date.timezone] = Asia/Yekaterinburg pm = dynamic pm.min_spare_servers = 10 pm.max_spare_servers = 20 pm.start_servers = 10 pm.max_children = 40" echo "$directives_poll">$sap_poll ln -s /etc/nginx/sites-available/$host.conf /etc/nginx/sites-enabled/$host.conf mkdir -p /var/hosting/$host/www/ mkdir -p /var/hosting/$host/tmp/ dir=/var/hosting/$host/www chown -R $users:$users "$dir"; find "$dir" -type d -exec chmod 0755 '{}' \; find "$dir" -type f -exec chmod 0644 '{}' \; /etc/init.d/nginx restart /etc/init.d/php5-fpm restart That's all. This configuration can encode video in different formats, as well as stream the stream. If this article is of interest to anyone, I will gladly give live examples of streaming.
Unfortunately, I can’t show the full stuffing, but here’s what I got at stream.etagi.com
Thanks for attention.
PS1 Found a more relaxed build path ffmpeg. Corrected below.
1. echo “deb www.deb-multimedia.org main non-free squeeze” /etc/apt/sources.list
2. apt-get update
3. apt-get install deb-multimedia-keyring
4. apt-get update
5. apt-get remove ffmpeg
6. apt-get purge ffmpeg
7. apt-get autoremove
8. apt-get install ffmpeg x264 (the installation will tighten all necessary dependencies)
9. ffmpeg -version
(in the output should not be DEPRECATED, if not from 5 to 9 point again)
PS2 Main design feature
Only works on PHP 5.3 from dotdeb
Source: https://habr.com/ru/post/188380/
All Articles