Online concert in the clouds

Resource Appleinsider.ru launches a new, unparalleled in the Russian Internet segment project for the online broadcast of live performances of both young bands and famous musicians. The frequency of the event - about 2 broadcasts per month.

The pioneers of December 6 became Fedor Chistyakov & F4BAND (ex-soloist of the ZERO group ). During the live broadcast, there were about 300 people who could appreciate the clean studio sound and atmosphere of a live concert.

For this broadcast, Clodo has allocated us its capacities and a channel width of 500 megabits / second. For a fairly high-quality bitrate of a stream of 128 kilobits / sec, this meant 4,000 customers "in a vacuum." We assumed that up to 2,000 listeners would come in and reinsured, but this time it was not possible to reach an agreement with the partners and our audience estimates were greatly underestimated. Attempt to bring the interested people from Habr ended with Dudlik ban.
')
Nevertheless, we quite successfully managed to hold a concert. Briefly about the technical implementation - we used the Icecast2 program on several servers, with load balancing using a samopisny bash script. More under the cut.

Technical implementation

To take advantage of the hosting capabilities, we needed to create 5 instances, each of which was allocated 100 megabits per second. The general scheme of interaction of the system looked like this:


We have the first server to which the source stream was broadcast (leading and musicians), and he was already engaged:
1) the actual broadcast stream for customers;
2) relay (relay) to other servers.

To distribute the audio stream, we used Icecast2 from the Ubuntu 10.04 repository. In short, here is the list of packages that had to be installed on the broadcast server:
g++ libmp3lame-dev libshout3-dev icecast2 libperl-dev libmp3-info-perl

When the source stream is absent, the first server will air the rotation of music from a specific directory. The nonstop stream is generated using Ices 0.4, which we compiled from sources:

 wget http://downloads.us.xiph.org/releases/ices/ices-0.4.tar.gz
 tar -zxvf ices-0.4.tar.gz
 cd ices-0.4
 ./configure --with-perl
 make
 make install

Music is stored in the / home / ftp directory, and ID3 tags were pulled out of them using the Perl module taken from here (for this we collected ices with the --with-perl key). Config /usr/local/etc/ices.conf:

 <? xml version = "1.0"?>
 <ices: Configuration xmlns: ices = "http://www.icecast.org/projects/ices">
   <Playlist>
     <Randomize> 1 </ randomize>
     <Type> perl </ type>
     <Module> ices </ Module>
     <Crossfade> 5 </ Crossfade>
   </ Playlist>

   <Execution>
     <Background> 0 </ Background>
     <Verbose> 0 </ Verbose>
     <BaseDirectory> / tmp </ BaseDirectory>
   </ Execution>

   <Stream>
     <Server>
       <Hostname> air1.appleinsider.ru </ Hostname>
       <Port> 1976 </ Port>
       <Password> SOURCEPASSWORD </ Password>
       <Protocol> http </ protocol>
     </ Server>

     <Mountpoint> / nonstop </ Mountpoint>
     <Name> AppleInsider.ru Radio </ Name>
     <Genre> not specified </ Genre>
     <Description> Music from our listeners </ Description>
     <URL> http://www.appleinsider.ru/ipodcast/ </ URL>
     <Public> 1 </ Public>

     <Bitrate> 128 </ Bitrate>
     <Reencode> 1 </ Reencode>
     <Samplerate> 44100 </ Samplerate>
     <Channels> 2 </ Channels>
   </ Stream>
 </ ices: Configuration>

Here are the most interesting places of the first server config (paths, logging, security sections I dropped), the /etc/icecast2/icecast.xml file:

 <icecast>
     <limits>
         <clients> 1000 </ clients>
         <sources> 6 </ sources>
         <queue-size> 524288 </ queue-size>
         <client-timeout> 30 </ client-timeout>
         <header-timeout> 15 </ header-timeout>
         <source-timeout> 10 </ source-timeout>
         <burst-on-connect> 1 </ burst-on-connect>
         <burst-size> 65535 </ burst-size>
     </ limits>

     <authentication>
         <source-password> SOURCEPASSWORD </ source-password>
         <relay-password> RELAYPASSWORD </ relay-password>
         <admin-user> admin </ admin-user>
         <admin-password> ADMINPASSWORD </ admin-password>
     </ authentication>

     <hostname> air1.appleinsider.ru </ hostname>
     <listen-socket>
         <port> 1976 </ port>
     </ listen-socket>

     <mount>
         <mount-name> / nonstop </ mount-name>
         <charset> UTF8 </ charset>
     </ mount>

     <mount>
	 <mount-name> / air </ mount-name>
	 <max listeners> 600 </ max listeners>
	 <charset> UTF8 </ charset>
	 <fallback-mount> / nonstop </ fallback-mount>
	 <fallback-override> 1 </ fallback-override>
     </ mount>

 .  .  .
 </ icecast>

Config second and subsequent servers, pay special attention to the sections of the relay and mount:

 <icecast>
     <limits>
         <clients> 1000 </ clients>
         <sources> 6 </ sources>
         <queue-size> 524288 </ queue-size>
         <client-timeout> 30 </ client-timeout>
         <header-timeout> 15 </ header-timeout>
         <source-timeout> 10 </ source-timeout>
         <burst-on-connect> 1 </ burst-on-connect>
         <burst-size> 65535 </ burst-size>
     </ limits>

     <authentication>
         <relay-password> RELAYPASSWORD </ relay-password>
         <admin-user> admin </ admin-user>
         <admin-password> ADMINPASSWORD </ admin-password>
     </ authentication>

     <hostname> air2.appleinsider.ru </ hostname>
     <listen-socket>
         <port> 1976 </ port>
     </ listen-socket>

     <relay>
	 <server> air1.appleinsider.ru </ server>
	 <port> 1976 </ port>
	 <mount> / air </ mount>
	 <local-mount> / air </ local-mount>
	 <username> relay </ username>
	 <password> RELAYPASSWORD </ password>
	 <on-demand> 0 </ on-demand>
     </ relay>

     <mount>
	 <mount-name> / air </ mount-name>
	 <max listeners> 600 </ max listeners>
	 <charset> UTF8 </ charset>
     </ mount>

 .  .  .
 </ icecast>

Thus, when icecast2 starts on the second and next servers, they automatically connect to the first server's mount / air point, copy it “on the fly” and start distributing at their same air mount point. On each server, we imposed a limit of 600 listeners, i.e. ideally, the channel of each server would be used by 75%.

Load distribution

Now about how we evenly distributed the load (although, as it turned out, this was not required).

Listeners could get to an online concert in two ways: either through a web player, by opening the podcast.appleinsider.ru page and clicking on the Play button, or by copying a link of the form air.appleinsider.ru/apple.m3u on the same page and “feeding” its your favorite desktop / mobile player.

In the first case, balancing was carried out randomly - when a client opened a page with a web player (by the way, it was built on jPlayer ), then a random number from 1 to 5 was generated using Javascript, and the stream was taken from the corresponding server (air1-air5).

In the second case, i.e. when the user gave the M3U link to his player, the following happened. Generally, when a link of the form server / mount.m3u is requested from icecast2 , it first searches for the file /usr/share/icecast2/web/mount.m3u, and if it finds it, it returns it. We could only correctly form this file.

We just once a minute for the crown polled the load of each of the servers, updated the summary statistics and redirected the symbolic link apple.m3u to one of the previously prepared files, for example for the first server, this file contained:

  http://air1.appleinsider.ru:1976/air 

Here is the actual bash script:

 #! / bin / sh

 TOTAL = 0
 IMIN = 1
 MIN = 100

 # We collect the load (ie the number of listeners) of each server
 # To do this, we query the service page and parsim it.
 for NUM in `seq 1 5`;  do
	 DATA = `curl --silent http: //air$NUM.appleinsider.ru: 1976 / status2.xsl`
	 AIR = `echo $ DATA |  sed 's /.*\/ air //' |  cut -d "," -f4`
	 NONSTOP = `echo $ DATA |  sed 's /.*\/ nonstop //' |  cut -d "," -f4`
	 if ["x $ {AIR}" = "xCurrent Listeners"];  then
		 AIR = 0
	 fi
	 if ["x $ {NONSTOP}" = "xCurrent Listeners"];  then
		 NONSTOP = 0
		 echo "air $ NUM: $ AIR"
	 else
		 if ["x $ {NONSTOP}" = "x0"];  then
			 echo "air $ NUM: $ AIR"
		 else
			 echo "air $ NUM: $ AIR / $ NONSTOP"
		 fi
	 fi

	 # NOW - server load num
	 NOW = `expr $ AIR + $ NONSTOP`
	
	 # TOTAL - total load
	 TOTAL = `expr $ TOTAL + $ NOW`

	 # Determine the number of the least loaded server
	 if [$ NOW -lt $ MIN];  then
		 MIN = $ NOW
		 IMIN = $ NUM
	 fi
 done
 echo "-------------------"
 echo "Total: $ TOTAL"

 # Update statistics on the webpage broadcast
 echo $ TOTAL> stat.txt
 scp stat.txt podcast: / var / www

 # Update M3U file
 echo air.m3u - \> http: //air$IMIN.appleinsider.ru: 1976 / air
 ssh air1 ln -sf /usr/share/icecast2/web/air.m3u_$IMIN /usr/share/icecast2/web/air.m3u

 echo "Server $ IMIN is on duty."

I agree, he is a little ax, but working. If there are suggestions for improvement, I will be happy to hear.

Results

Here is a graph of the volume of the audience during the online broadcast, 280 people in peak:


Subscribe to the podcast and download the recording of the online concert here: RSS | MP3 | iTunes

The next event is scheduled for December 25th. We are not going to talk about its participants, because negotiations are in the most active stage. All the details and announcements will be available on Appleinsider.ru