How do we build a small radio station in a large network
This article will not describe how you can quickly configure the icecast in different Linux systems for different tasks. Not. In this article, I would like to highlight the small history of one small project on the output of an on-air radio station to the global network. What problems we had to face and how we solved them.
So. There is a small prehistory - there is a radio holding from several radio music stations with its head office in the capital of our country, Moscow. The radio stations themselves broadcast in the FM band of the capital, have their own audience and are quite happy with their listeners. But bad luck - in the yard in 2012, and the direction of Internet broadcasting is not shaky. Further there will be many words and small stories in the framework of one main, if you are interested, welcome under cat.
The task definition is long overdue, both for the general director of the entire holding and for many program directors of the radio stations themselves. But there was a small problem - the holding did not in any way see ways to monetize this service, and since there is no monetization, there are no budgets to build an insane solution. Why the task is overdue - the existing service at that time (spring of 2012) did not satisfy any requirements of reliability, quality and convenience. Just in the spring of 2012, the holding changed the leadership of the technical department and IT services. Fate and my colleagues gave me the opportunity to manage the IT service of this company.
So.
Broadcasting in the company provided a small server complex:
Then the MP3 signal was sent not quite straightforward - for each of the radio stations, and there were 5 of them, a stream of 256 kbit / s was formed and published on the local icecast encoding server, after which, the main distribution server via NAT independently took these streams and transcoded them in additional (64, 96, 128, 320). This was on the server side, on the client side everything was even worse - no flash / html client existed in principle, a small FAQ was written posted on the radio stations website, how to connect to streams, and that was the end of the "licking" of clients. It is necessary to add that this construction worked not shakily, almost daily we observed “breakdowns” in the flows of the main radio station of the holding when reaching 3-3.5 thousand listeners, as a rule, “breakdowns” occurred several times during the day. The reasons for the failures were different, then the transcoder will rise, then for some reason the server decides that it does not see the remote mount point, then something else. In short, it was clearly what to do.
To my logical question, addressed to the engineers who provided support for the whole of the economy - what’s so “not logical”, is everything built up? The usual answer followed - well, that’s historically. It should be noted that the external outsourcing employee from Novosibirsk was engaged in “finishing” the icecast. He's a great guy, he knows FreeBSD up and down, he writes in C or C ++, but his understanding in multimedia is not so great, but he didn’t know exactly how to build large broadcast platforms on the network, and even without a single point of failure. Yes, and it does not need him, he does not pay money for it. Plus, the time difference, in short, the turning point was passed at the moment when the General entered the office with the head of the technical department and said - guys, my radio station doesn’t play with my iPhone on the network, let’s do something . He had nothing to answer - well, he really doesn’t play, because again, something fell off there, etc.
')
Any administrator will tell you at the sight of the icecast, what is there to think, then "file down" is necessary. But this is not our case. We have already been "historically" washed down that we need to bring these stables to some normal level. The main problems that existed at that time:
In short, where to dig, now how to bring it all to some kind of reasonable mind.
The solution required a systemic one - what comes to the mind of any IT manager when he is faced with a not entirely relevant task - is correct, but let's give it to outsourcing for money. Some managers who are not particularly good at hand also manage to shove their interest in the cost of outsourcing, but this is not about us. My boss and I considered both decisions at once; or we do it ourselves and how, or we give it to outsourcing, then the question arises - how much will it cost? So, the question of outsourcing has disappeared by itself, as soon as we figured out how much it will cost to broadcast on the network to a third party and forget about this task forever. Companies that provide such services publicly present such prices for their services that major players in the radio market will never come to them, this is a fact. The profitability of even large radio stations included in the top ten in Moscow will certainly allow them to surrender on these conditions, but it will still be a significant expense item, and this is in the absence of normal monetization. There are services offering both broadcasting and monetization in the form of placing various types of advertising, but their prices for services are quite high, and the question of return on investment is very serious, so it was said - NO. We cannot go for this, especially since they hired us to reduce operating expenses in the most accessible ways, and then everything will be exactly the opposite. In short, we abandoned this idea.
Build yourself - how?
Why is that?
Attempts to put any load balancing before the icecast carrying a large number of streams, in my opinion, are absolutely meaningless. If your server has a gigabit communication channel, with a relatively weak processor and a small amount of RAM, Icecast will clog it completely, just drag customers. So what will you balance? Several gigabit streams, and what to do with a single point of failure? What is the output? We balance only at the network level using switches, if you have a port channel. With three gigabit channels, you can count on 2.7 gigabits per second, with four channels for all 4, and so on. The load distribution between servers in order not to overload the mount points can be carried out by icecast itself, its KH version. In practice, it turned out that icecast does not hold more than 3200-3500 on one mount point of 128 kbit / s MP3 - the reason, as far as we understood, is in high load on the processor cores. The fact is that by default icecast handles one mount point on one core. If the mount points are larger than the number of cores in the system, it begins to divide them into cores, but again, not evenly, but by mount points per core. If your processors are not very fast, then you should not load more than 3000 simultaneous ones on one mount point, you can use the trick - to transfer, when this value is reached, to a hidden mount point on the same server, but with a different name. In the usual icecast, this "perverted" logic works fine. KH version requires that another mount point have another source.
Secondly, I believe that creating a monstrous server for broadcasting tasks means absolutely not taking into account the possibilities of horizontal growth of the platform as a whole, which is not entirely true, in the future it will still come around and how. Therefore, it was chosen option with horizontal scaling.
Why they wanted to encode with proprietary software - they didn’t want to make a garden in the form of separate components of FM processing and encoding, and the introduction of AXIA LiveWire was a very “tasty” addition to the solution of this problem. The advantage was that after testing several encoders it became clear that AXIA differs from free encoders by the quality of compression, i.e. compared to lame, etc. she sounds better. I would not like to explain “why?” For a long time, better then or in the comments.
What we encountered right away - it became clear that we need to somehow measure the number of listeners, and both simultaneous and accumulated per day, day, week, etc. In the process of clarifying, it became clear that we simply cannot leave such a variety of flows and many of them are extremely “thick” (320 for example). This was the first serious stumbling block, and convincing program directors and other top-players turned out to be much easier than ordinary engineers who were responsible for the implementation of this task in the past. The “audiophile” state of mind refused to understand objective reality. We managed to do this by hook or by crook, it cost a lot of blood and spiritual wounds left a lot. But thank God no one was fired and they did not spoil karma with their subordinates. Began to put everything in order. We came to the following standard streams for each of 5 radio stations - 128 & 64 kbit / s MP3, 56 & 48 kbit / s AACv2 +.
For a long time did not touch the main distributing server, put in order the internal infrastructure. Finally, one day in the spring of 2013, we and my subordinate in the morning darted to M IX and staged the collapse of everything and everything with a further uplift. It turned out well. Immediately it turned out the advantages of the introduction of Linux in conjunction with icecast-kh - it really began to load processors weaker under the same load. The difference was up to 20%. It's a lot. Plus, they immediately spread the load on 4 servers and during the day began to connect them to a single DNS record.
Due to difficulties with the development department with getting a normal player, we decided to supplement load balancing with the Round & Robin mechanism in the DNS. He is certainly not the most reliable, but not yet about this.
In November 2013, my work experience in this glorious company ended. What can I be proud of and what has not been done as part of this task.
The total farm flow of 4 servers exceeded 2.5 Gbit / s in peak during the day and continued to rise slowly from month to month. Until all gestures that were made, it did not exceed 1.2-1.3. In fact, the company did not serve everyone who wanted a service, but only those who had time to go first.
According to preliminary statistics, we were able to "break through" an indicator of half a million unik per week; these figures can already be sold somehow. A mechanism was introduced to place intro files that play a promotional video for each connected user, while without tracking who has already lost and who has not, but this was the first placement experience.
The load began to spread more evenly across the servers, and this affected the quality of the service. Interruptions occurred only as a result of real network failures and other problems, but were no longer caused by incorrect configuration of the services themselves or other problems.
Due to the emergence of high-quality streams with a low bitrate, the number of clients from mobile platforms began to shift towards their increase. This makes me happy as an engineer - after all, this is essentially the future delivery model for radio stations in general.
Domestic icecast was shot and a single internal icecast was created for distribution to all distribution servers + it was also used by some regional partners as a backup signal source. Not once or twice it saved us, and the regionals also said thanks for the reliable service of the reserve.
What can not be proud of.
Plans for the implementation of a full-fledged transition to the Omnia AX / E remained unfulfilled - they completely refused to finance. As a result, they were forced to make a fuss with the hardware FM processors found in the warehouse, as ancient as mm, you understand. The problem of monitoring remained - refusal to finance, they decided to implement PRTG, Nagios & more simply did not have human resources, this task drew the impossibility of a quick reaction to the failure of any of the servers and preventing a large number of clients from having many minutes of silence. It was not possible to implement the normal analytics system of the log files of the distributing servers - by normal is meant a system able to calculate, in addition to the usual parameters, also purely media data, ala AQH and others. The only system known to me is Casterstat, all the rest are sharpened for processing primarily Web servers and are not quite suitable. The full-fledged players were not fully developed for most mobile platforms, they could only be completed under iOS, but at least that was the case, since the task was not with us.
I do not know how to evaluate in the degree of completion, but we failed to implement a full-fledged balancing system based on the player’s pls mechanism, primarily due to problems with the department responsible for developing the player. But it doesn’t matter, we don’t blame them for anything and nothing.
It is necessary to understand that this is not the only task that the technical department and IT service have solved during my work in this company, so you should not be surprised at such a length. This type of excuse me.
If someone is interested in the topic of IT in the multimedia sphere, then I can continue the stories on some other topics, the benefit is the experience.
So. There is a small prehistory - there is a radio holding from several radio music stations with its head office in the capital of our country, Moscow. The radio stations themselves broadcast in the FM band of the capital, have their own audience and are quite happy with their listeners. But bad luck - in the yard in 2012, and the direction of Internet broadcasting is not shaky. Further there will be many words and small stories in the framework of one main, if you are interested, welcome under cat.
The task definition is long overdue, both for the general director of the entire holding and for many program directors of the radio stations themselves. But there was a small problem - the holding did not in any way see ways to monetize this service, and since there is no monetization, there are no budgets to build an insane solution. Why the task is overdue - the existing service at that time (spring of 2012) did not satisfy any requirements of reliability, quality and convenience. Just in the spring of 2012, the holding changed the leadership of the technical department and IT services. Fate and my colleagues gave me the opportunity to manage the IT service of this company.
So.
Preliminary situation
Broadcasting in the company provided a small server complex:
- 1 distributing server (FreeBSD, doped icecast, thanscoder, 3 Gbit ethernet, Butlerov M IX)
- 2 encoding servers installed in the studio complex (aka head office) in Moscow and encoding analog / AES sources in MP3 streams to the main distribution server (Windows XP, icecast, edcast)
Then the MP3 signal was sent not quite straightforward - for each of the radio stations, and there were 5 of them, a stream of 256 kbit / s was formed and published on the local icecast encoding server, after which, the main distribution server via NAT independently took these streams and transcoded them in additional (64, 96, 128, 320). This was on the server side, on the client side everything was even worse - no flash / html client existed in principle, a small FAQ was written posted on the radio stations website, how to connect to streams, and that was the end of the "licking" of clients. It is necessary to add that this construction worked not shakily, almost daily we observed “breakdowns” in the flows of the main radio station of the holding when reaching 3-3.5 thousand listeners, as a rule, “breakdowns” occurred several times during the day. The reasons for the failures were different, then the transcoder will rise, then for some reason the server decides that it does not see the remote mount point, then something else. In short, it was clearly what to do.
To my logical question, addressed to the engineers who provided support for the whole of the economy - what’s so “not logical”, is everything built up? The usual answer followed - well, that’s historically. It should be noted that the external outsourcing employee from Novosibirsk was engaged in “finishing” the icecast. He's a great guy, he knows FreeBSD up and down, he writes in C or C ++, but his understanding in multimedia is not so great, but he didn’t know exactly how to build large broadcast platforms on the network, and even without a single point of failure. Yes, and it does not need him, he does not pay money for it. Plus, the time difference, in short, the turning point was passed at the moment when the General entered the office with the head of the technical department and said - guys, my radio station doesn’t play with my iPhone on the network, let’s do something . He had nothing to answer - well, he really doesn’t play, because again, something fell off there, etc.
')
First you need to think
Any administrator will tell you at the sight of the icecast, what is there to think, then "file down" is necessary. But this is not our case. We have already been "historically" washed down that we need to bring these stables to some normal level. The main problems that existed at that time:
- low level of redundancy - in fact, not a single component of the system was reserved
- the absence of a system of at least some load balancing - all threads on one server, all clients on one server. And then we'll see if he keeps them or not.
- a large number of undiagnosed failures
- diagnostics is difficult due to insufficient documentation of customizations icecast and FreeBSD core + outdated product (FreeBSD some kind of shaggy version) + problems with this product in the form of not entirely "true" implemented parts of the TCP / IP stack that crawl out, just at high loads with a large number of sessions
- complete lack of FM signal processing - this is considered to be very unprofessional in the radio environment if clarification in comments or comments is needed
- the absence of a flash / html client for playing directly on the site of radio stations
- lack of monitoring system
In short, where to dig, now how to bring it all to some kind of reasonable mind.
Stop thinking - it's time to "gash"
The solution required a systemic one - what comes to the mind of any IT manager when he is faced with a not entirely relevant task - is correct, but let's give it to outsourcing for money. Some managers who are not particularly good at hand also manage to shove their interest in the cost of outsourcing, but this is not about us. My boss and I considered both decisions at once; or we do it ourselves and how, or we give it to outsourcing, then the question arises - how much will it cost? So, the question of outsourcing has disappeared by itself, as soon as we figured out how much it will cost to broadcast on the network to a third party and forget about this task forever. Companies that provide such services publicly present such prices for their services that major players in the radio market will never come to them, this is a fact. The profitability of even large radio stations included in the top ten in Moscow will certainly allow them to surrender on these conditions, but it will still be a significant expense item, and this is in the absence of normal monetization. There are services offering both broadcasting and monetization in the form of placing various types of advertising, but their prices for services are quite high, and the question of return on investment is very serious, so it was said - NO. We cannot go for this, especially since they hired us to reduce operating expenses in the most accessible ways, and then everything will be exactly the opposite. In short, we abandoned this idea.
Build yourself - how?
- Reservation - we increase the number of distribution servers to 4 with embedding of the balancing mechanism on the level of icecast-kh. We bring the encoding software to the virtualization platform. We take out the icecast of the first level of distribution from the encoders and put it on a separate virtual machine. We reserve channels the Internet in the central office
- Balancing - we balance the load depending on the filling of the mount points with icecast-kh
- It is proposed to use Omnia AX / E as an encoding software, it allows closing both the requirements for encoding and FM audio processing.
- a complete rejection of transcoding on the main distribution servers, the server should receive a completely ready stream and only distribute it
- Complete rejection of FreeBSD on servers and transition to one of Linux (as a result, we chose Ubuntu 12.04 LTS)
- writing by the neighboring department of a full-fledged flash / html client for playing on the site, developing clients for iOS + Andriod mobile platforms
- providing customers with not only MP3, but also AACv2 + streams for listening on the networks of mobile operators
- installation of an active monitoring system with the ability to change files by event
Why is that?
Attempts to put any load balancing before the icecast carrying a large number of streams, in my opinion, are absolutely meaningless. If your server has a gigabit communication channel, with a relatively weak processor and a small amount of RAM, Icecast will clog it completely, just drag customers. So what will you balance? Several gigabit streams, and what to do with a single point of failure? What is the output? We balance only at the network level using switches, if you have a port channel. With three gigabit channels, you can count on 2.7 gigabits per second, with four channels for all 4, and so on. The load distribution between servers in order not to overload the mount points can be carried out by icecast itself, its KH version. In practice, it turned out that icecast does not hold more than 3200-3500 on one mount point of 128 kbit / s MP3 - the reason, as far as we understood, is in high load on the processor cores. The fact is that by default icecast handles one mount point on one core. If the mount points are larger than the number of cores in the system, it begins to divide them into cores, but again, not evenly, but by mount points per core. If your processors are not very fast, then you should not load more than 3000 simultaneous ones on one mount point, you can use the trick - to transfer, when this value is reached, to a hidden mount point on the same server, but with a different name. In the usual icecast, this "perverted" logic works fine. KH version requires that another mount point have another source.
Secondly, I believe that creating a monstrous server for broadcasting tasks means absolutely not taking into account the possibilities of horizontal growth of the platform as a whole, which is not entirely true, in the future it will still come around and how. Therefore, it was chosen option with horizontal scaling.
Why they wanted to encode with proprietary software - they didn’t want to make a garden in the form of separate components of FM processing and encoding, and the introduction of AXIA LiveWire was a very “tasty” addition to the solution of this problem. The advantage was that after testing several encoders it became clear that AXIA differs from free encoders by the quality of compression, i.e. compared to lame, etc. she sounds better. I would not like to explain “why?” For a long time, better then or in the comments.
Begin to "moth"
What we encountered right away - it became clear that we need to somehow measure the number of listeners, and both simultaneous and accumulated per day, day, week, etc. In the process of clarifying, it became clear that we simply cannot leave such a variety of flows and many of them are extremely “thick” (320 for example). This was the first serious stumbling block, and convincing program directors and other top-players turned out to be much easier than ordinary engineers who were responsible for the implementation of this task in the past. The “audiophile” state of mind refused to understand objective reality. We managed to do this by hook or by crook, it cost a lot of blood and spiritual wounds left a lot. But thank God no one was fired and they did not spoil karma with their subordinates. Began to put everything in order. We came to the following standard streams for each of 5 radio stations - 128 & 64 kbit / s MP3, 56 & 48 kbit / s AACv2 +.
For a long time did not touch the main distributing server, put in order the internal infrastructure. Finally, one day in the spring of 2013, we and my subordinate in the morning darted to M IX and staged the collapse of everything and everything with a further uplift. It turned out well. Immediately it turned out the advantages of the introduction of Linux in conjunction with icecast-kh - it really began to load processors weaker under the same load. The difference was up to 20%. It's a lot. Plus, they immediately spread the load on 4 servers and during the day began to connect them to a single DNS record.
Due to difficulties with the development department with getting a normal player, we decided to supplement load balancing with the Round & Robin mechanism in the DNS. He is certainly not the most reliable, but not yet about this.
I'm leaving leaving a mountain of cigarette butts ... Results
In November 2013, my work experience in this glorious company ended. What can I be proud of and what has not been done as part of this task.
The total farm flow of 4 servers exceeded 2.5 Gbit / s in peak during the day and continued to rise slowly from month to month. Until all gestures that were made, it did not exceed 1.2-1.3. In fact, the company did not serve everyone who wanted a service, but only those who had time to go first.
According to preliminary statistics, we were able to "break through" an indicator of half a million unik per week; these figures can already be sold somehow. A mechanism was introduced to place intro files that play a promotional video for each connected user, while without tracking who has already lost and who has not, but this was the first placement experience.
The load began to spread more evenly across the servers, and this affected the quality of the service. Interruptions occurred only as a result of real network failures and other problems, but were no longer caused by incorrect configuration of the services themselves or other problems.
Due to the emergence of high-quality streams with a low bitrate, the number of clients from mobile platforms began to shift towards their increase. This makes me happy as an engineer - after all, this is essentially the future delivery model for radio stations in general.
Domestic icecast was shot and a single internal icecast was created for distribution to all distribution servers + it was also used by some regional partners as a backup signal source. Not once or twice it saved us, and the regionals also said thanks for the reliable service of the reserve.
What can not be proud of.
Plans for the implementation of a full-fledged transition to the Omnia AX / E remained unfulfilled - they completely refused to finance. As a result, they were forced to make a fuss with the hardware FM processors found in the warehouse, as ancient as mm, you understand. The problem of monitoring remained - refusal to finance, they decided to implement PRTG, Nagios & more simply did not have human resources, this task drew the impossibility of a quick reaction to the failure of any of the servers and preventing a large number of clients from having many minutes of silence. It was not possible to implement the normal analytics system of the log files of the distributing servers - by normal is meant a system able to calculate, in addition to the usual parameters, also purely media data, ala AQH and others. The only system known to me is Casterstat, all the rest are sharpened for processing primarily Web servers and are not quite suitable. The full-fledged players were not fully developed for most mobile platforms, they could only be completed under iOS, but at least that was the case, since the task was not with us.
I do not know how to evaluate in the degree of completion, but we failed to implement a full-fledged balancing system based on the player’s pls mechanism, primarily due to problems with the department responsible for developing the player. But it doesn’t matter, we don’t blame them for anything and nothing.
It is necessary to understand that this is not the only task that the technical department and IT service have solved during my work in this company, so you should not be surprised at such a length. This type of excuse me.
If someone is interested in the topic of IT in the multimedia sphere, then I can continue the stories on some other topics, the benefit is the experience.
Source: https://habr.com/ru/post/221253/
All Articles