IT at the Otkritie Arena stadium. When the adrenaline in the project is not less than during the match
As part of a team of specialists from LANIT, I participated in the project to create the IT infrastructure of the Otkrytie Arena stadium. In this article I will briefly talk about the problems that we faced, how they were solved, why they did it this way, and how it all works three years after the start of operation.
Otkrytie Arena Stadium is a cross between a shopping center, a hotel and an airport. It is designed for 45 thousand fans. This is comparable to the average throughput of Pulkovo Airport per day. To get to the stadium, these 45 thousand people must pass through the turnstiles, equipped with a ticket-passing system. Ticket system works through the LAN. Failures of the LAN or of the ticket system itself mean a crush at the turnstiles and other troubles.
')
Guests at the stadium are of four types: fans, more relaxed football fans (the so-called category "Public"), VIP and media. Journalists need Wi-Fi and wired internet access. It is important for them to send a report to the editor as quickly as possible, so that in case of the slightest problem with the Internet, they immediately start a scandal.
Media locations
In the sub-tribune space there are restaurants and trade pavilions, whose work is tied to the cash terminals. There are also televisions on which you can broadcast an advertisement or a picture from the field - this is very convenient, you can stand in line for a hot dog and watch football. If the TV breaks down, the company with which the contract for broadcasting advertising on this TV is concluded has the right to roll out a penalty to the stadium. Cash terminals also work on LAN, and worst of all, if during the match will cease to pass banking transactions. It is terrible to imagine that 45 thousand people can arrange in a confined space if they are not fed in time.
In the VIP-zone for an additional fee, you can rent a sky-box - a separate room with a view of the field, a bar, waiters and your own lodge in the stands. Wi-Fi is also working there and TVs are hanging. Renting a sky box is expensive, so here any problems with Wi-Fi and TVs will definitely turn into complaints about the stadium.
Statistics of consumption of Internet traffic at the stadium "Otkritie Arena" for the month. The most active consumers are the media and the office of Spartak football club.
Thus, 45 thousand people can come to each match, the stadium is responsible for them and makes titanic efforts so that everything goes smoothly. This is a huge stress and responsibility for everyone who works in the stadium, including IT people. The next time you cheer on your favorite team, think about it.
In January 2012, the stadium invited us to take part in a competition for SCS, LAN, PBX, server infrastructure, the system of collective reception of television (SKPT) and the system of video advertising (Digital Signage). Wi-Fi was also included there, but we will dedicate a separate post to this topic. To win the competition, it was necessary to fulfill two simple conditions: to offer the most optimal solution for the price and prove that it will work without fail during the event.
This is how the stadium looked like in December 2012, when we started designing
As the initial data there was a technical task for three sheets and a bunch of project documentation of varying degrees of responsibility. LAN and Wi-Fi were designed on Cisco.
I remembered that the Cisco Catalyst 6509 switch went to specifications without interface cards, transceivers, licenses, power supplies and SMARTNETs. Cisco was suitable for this project, but did not give advantages in price, so we decided to replace it with an analog from Huawei.
The result was cheaper and not inferior to Cisco in the reliability of operation. Of course, the counterpart from Huawei could show off a little during commissioning and was inferior to Cisco in the range of available administration guides. And so it happened, but for the sake of optimizing capital expenditures, we had to accept this.
We started thinking about how to save on servers, storage and uninterruptible power supply without sacrificing reliability. We received a call from Huawei’s Russian representative office and offered to consider the UPS2000 UPS with double conversion UPS, Huawei Tecal X86 servers and OceanStor storage systems. In 2012, it was completely unfamiliar equipment for Russia, especially when it came to UPS2000.
So it turned out a solution with an exclusive hardware, in which LAN, Wi-Fi, servers, storage and even UPS were from one vendor. In 2012, it was impossible to implement this idea with any other vendor (now from well-known vendors you can still on HPE and DELL, but on DELL - in general is exotic).
In two distant racks - UPS Huawei 2000 for server rooms. In fact, these are three 15KVA UPSs each. The total power is 45KVA. Oceanstor 5500 - on the left.
The question of reliability decided as follows. Since there was no operating experience for the proposed Huawei equipment in Russia, LANIT had to stake its 25-year-old reputation.
Under the terms of the contract, if within 5 years from the moment of commissioning the customer is not satisfied with the reliability of what we offer, we take the equipment for ourselves, put something else and pay for it ourselves.
In principle, with Huawei it was possible to go even further - take an eSpace PBX, Digital Signage vSpace, and do everything on Huawei except trays and SCS. But it did not work out. vSpace did not fit the functional (about this - just below), but for PBX the choice finally fell on Aástra (the former Ericsson, now Mitel Networks). The customer wanted to use IP DECT base stations throughout the stadium, so it was necessary to lay a station in the project with built-in support for this functionality. The cheapest of the considered and relevant requirements was Aastra.
The stadium is a very important object, and if you think that putting equipment there without operating experience in Russia is a big risk, you are right. But let's talk about IPTV / Digital Signage.
Despite the resemblance, these systems are completely different in functionality, and at the Otkrytie Arena stadium, I still have the feeling from them like a rollercoaster ride. This is because in the course of the project, the number of TV panels was constantly changing, and almost every TV has migrated from IPTV to Digital Signage or back at least once. Therefore, we decided to design both systems on the same platform.
The customer liked the idea. He also wanted with the help of such a system to control all TVs remotely, to provide the possibility of overlaying a text streamline on a TV stream, support for Adobe Flash, a custom portal for Video-on-Demand content, and much more. Such requirements from the solutions known to us met: Cisco Stadium Vision (Cisco Systems, USA), Tripleplay (Tripleplay Services, UK) and C-nario (YCD Multimedia, USA / Israel). For all three, the stadium promised to be the first project of such a scale in Russia.
Began to compare the architecture of solutions. The first to go the distance is C-nario. It has fantastic functionality, but as a set-top-box set-top box, with which the system connects to the TV, x86-based nettops were offered based on Windows, which raised questions about reliability. Another system on the C-nario turned out the most expensive.
Stadium Vision turned out to be cheaper than C-nario, used proprietary set-top-box consoles with the Cisco logo on the front panel, and this was a plus, since in theory they are more reliable than nettops on Windows. However, Cisco was a bit jealous of Huawei and hinted that if the LAN is not Cisco, it should work, but the vendor does not guarantee anything (it is still wondering why).
Tripleplay had a completely different architecture. The British guaranteed compatibility with the LAN of any vendor, and Amino products, which, in principle, could be freely purchased in Russia and used not only with Tripleplay, were used as set-top box prefixes. With a sufficient number of licenses, any prefix could be transferred from IPTV to Digital Signage with a couple of mouse clicks in the administration panel.
All this set up in a more positive way and would look absolutely gorgeous if it were not for how it was proposed to solve the problem of remote control of TV panels in Tripleplay.
In principle, Tripleplay could control TV panels either via HDMI (but without guarantee of compatibility with a particular TV model), or via RS232, but via a USB-RS232 adapter due to the lack of an RS232 port on the Amino console. We did not like both options, because the adapters did not want to, and the customer approved the specification of TV panels almost before the purchase itself. As a result, the customer chose Tripleplay and decided to organize remote control of TV panels on a separate solution in the form of an AMX controller, which interacted with TVs via LAN.
The IPTV / Digital Signage architecture is as follows:
In the yard in January 2013. There are four sectors at the stadium; according to the project, the sub-chamber space is heated only in one of them - the western one. However, even it was not heated. This had to be seen during the installation of trays and cable infrastructure.
So we went to the future stadium in the winter of 2013. Colleagues told that they saw hares who came running from the Tushino field.
Minus thirty degrees, people hang from the ceiling and pull the cistern track, which ultimately rests on the electrician's tray or fan box. This is because the representatives of related installation organizations, to keep warm, set off to build their tracks not according to the project. As a result, everyone began to interfere with each other. At each site, a consultation of engineers and technical workers of the customer and related installation organizations was held. Consilium swore for a long time, but then decided who would be laying communications where. In the next section, everything was repeated. Such an organization of work required from us the project office at the site and the landing in the form of one or two designers. Every day they went to these meetings and painted the actual lining of the trays.
The furnishings of our project office at the construction site. Photographed in the summer.
We have to deal with this kind of work organization very often. Over time, you get used to it and stop understanding people who spend a lot of time on master plans for laying engineering networks, BIM-models, wiring communications in 3D and other things that are completely useless at sites where drawings are constantly being reworked, contractors change, there is always not enough time and every week you make a new schedule.
We began to lay the cable only when it got warmer. All the same, it’s good that we laid cables in the project, which are designed for installation and operation with a frost of -40 degrees. The laying of the flute trails along the footbridges of the stadium was no less exciting, as there is very tight, slippery and a strong wind blows from the Moscow River. In just 1 year, 20 kilometers of trunk cable and 300 kilometers of twisted-pair cable were laid.
Roof walkways before the installation of low-voltage systems. Assemblers daily spent on them for 8-10 hours of working time.
When the construction work in the server rooms ended and we started installing the UPS, one piquant UPS2000 design feature surfaced. As part of the product, there was a switchboard with automatic machines, which was turned on between the UPS and power distribution units (PDU) in the racks. If we compare the values ​​of these machines, the passport currents of the PDUs, and the values ​​of the machines of the introductory switchboard of the server room, all this did not meet the requirements specified in the Electrical Installation Rules (they are also the "Bible of electricians", or just PUE). We wrote about this in Huawei, they replied that they want to look at everything with their own eyes. So I had the strangest business negotiations in my entire career.
This distribution box came with the UPS2000.
Our account manager and three Chinese appeared in the server room. I was not introduced to them, but it was clear that these were not technical support engineers. The most important of them was wearing a gray jacket in a cage and was about one and a half meters tall. The other two wore black jackets and were slightly higher than the main thing, which was clearly embarrassed. Noise punch. The air was hazy from construction dust, and we stood near the racks with UPS, because there was nowhere to sit down. My height is eighty meters. When a Chinese in a gray jacket spoke to me, he looked up at me and still looked presentable. His speech was a mixture of Chinese and English. He sincerely did not understand what the problem was. I talked about the EMP, the principles of the selectivity of automata, which must be observed by everyone who is within the borders of the Russian Federation, but his expression still has not changed. All that he answered me, sank in the rumble of a perforator. When the puncher subsided, the Chinese switched to Russian: "We can change the decision, but then we are not responsible for it." “According to our laws, you don’t answer for him anyway, but I answer as the chief engineer of the project,” I replied. As a result, we agreed that the vendor will bring the solution into compliance with the requirements of the OLC.
In fact, almost all foreign vendors with whom we deal are not always devoted to the subtleties of the Russian GOST and SNIP. In some cases, it threatens to completely replace the technical solution. It is good when the vendor understands this and goes into a constructive dialogue without unnecessary altercations and bureaucracy.
Equipment is being installed
The less time left before the end of the project, the brighter became the feats of labor builders. Brigades of painters and plasterers increasingly wielded perforators in rooms with commutators working and sucking dust. There was a threat of mass failure of LAN and data center equipment for a clearly non-guaranteed reason. They tried to get out of the situation by blowing “iron” with cylinders with compressed air and even by gluing the interface ports of the equipment.
One of the switches during commissioning. Ports unsophisticated plastered masking tape. If you look closely, you can see a layer of building dust on the top panel of the device.
In the meantime, the customer decided to connect the PBX not via E1, as it was in the project, but via SIP, for which border session controllers were required. A month after the decision was made, two Huawei AR2240 routers with voice processing cards were delivered to the stadium.
After launch, these devices categorically refused to be SIP-friendly with Aastra. The firmware change did not help, and we wrote a call to Huawei service support (TAC). At first, Huawei engineers connected to the AR2240 remotely, and when that did not help, the pilgrims of the vendor’s experts began directly to the site to fall on the console of the routers and fine-tune the SIP parameters, the total number of which exceeded 100 pcs. Everything ended with a patch for OS AR2240, which was developed personally for our project. After the patch, SIP earned, but not in the way we would like. I especially remember a bug that caused the connection to break after 30 seconds when calling from Aastra to numbers with an auto attendant (DISA). The problem was solved by another patch of the router software.
There is nothing special to remember about Tecal servers - regular X86 servers. Worked steadily, there are no complaints about the build quality. At the same time, we got acquainted with the HPE iLO analog for monitoring and remote server administration - this is called iMana for Huawei Tecal, and it works quite well.
Two Oceanstor S5500T storage systems also worked without any problems, except that they were upset by the lack of clustering support at the time, such as HP Peer Persistence or NetApp MetroCluster, but it didn’t work for us. Oceanstor currently supports clustering, a technology called HyperMetro. About her there is an article on Habré here .
Toward the close of construction, Spartak was scheduled to have an away match. The match should have been shown on TV live. The customer was inspired by the idea to display a picture of this broadcast on the video table of the stadium, the sound on the sound reinforcement system and invite fans to the stands. It was proposed to take a live TV signal from Tripleplay with our SKPT. By this time, we’ve almost finished commissioning.
Video board of the Otkritie Arena stadium during the opening ceremony. It is suitable for watching television.
The challenge was to integrate the TPCS, for which we were responsible, with a video-wall system, for which one contractor was responsible, and a sound-amplification system, for which another one was responsible. We decided to integrate not directly, but through the Media Control Room of the stadium, for which another contractor was responsible.
Media Control Room - central hardware multimedia systems of the stadium. It forms the content that is given to all the display means of the stadium.
It was necessary to somehow remove the signal from the IP-TV, feed it to the Media Control Room's matrix switcher, scale it and divide it into a video and audio component. Then the video was sent to the board controllers, and the sound to the mixing console of the sound reinforcement system. At the same time, video and sound should not lag behind each other.
As a result, we purchased two more Amino A140 consoles, registered them in the Tripleplay system and brought them to a TV channel, through which we had to broadcast the match. The HDMI interface of the set-top boxes through the HDMI-SDI converters were included in the Media Control Room's matrix switcher. And it all worked.
It was very cool to sit in the stands of the stadium and watch the match on the video table of 172 sq. M.
Now we are commissioning Huawei equipment for one of our customers. Imagine, the customer found the format of the output of one of the commands in the console of the Huawei NGFW firewall uncomfortable. We contacted Huawei, and the vendor released a personal patch for Huawei NGFW customer, which changes the output format of the console command. This is a continuation of Huawei’s flexible policy that helped us build a successful project on a new equipment for the Russian market.
The IT infrastructure construction project at the Otkritie Arena stadium took us 2 years. 4500 SCS ports were laid, SKPT was made on 180 TV panels, two storage systems for 50TB each were put into operation, four two-socket Rack-mount servers for the virtualization platform, 87 Wi-Fi points, PBX for 250 subscribers and 36 DECT base stations.
The stadium became one of the first facilities in Russia, where Huawei Oceanstor storage systems, UPS2000 UPS, Tecal servers, and DCPA / Digital Signage based on the Tripleplay solution were used. Today LANIT is engaged in service support of these solutions. These are support decisions. During operation, they managed to acquire other IT services, including a deposit club card, which can be used both to enter the stadium and to pay for purchases / additional services on the territory of the sports complex.
The stadium is actively being upgraded to receive FIFA matches in 2018. It affects the IT infrastructure. And quite seriously, because in 2011, when we were designing, the Otkrytie Arena stadium did not plan to participate in this event.
In the hands of the IT director of the Otkrytie Arena stadium, Igor Shestunov, the Talmud of FIFA requirements for stadiums (about 900 pages)
The project was pretty successful. In 2014, Huawei announced it the “Project of the Year”, and the patch written by its engineers became part of the next version of the Huawei OS AR2240. The customer also responds positively to the reliability of Huawei equipment, including UPS2000. We have not had the opportunity to evaluate the work of the Huawei service center for the delivery of new equipment instead of what we have been dusting, since during the whole period of operation no Huawei device has failed.
There are no complaints about the Tripleplay IPTV / Digital Signage solution. Positive recommendations from the IT service of the Otkrytie Arena stadium allowed us to offer a similar solution for the reconstruction of the Luzhniki stadium, which ended in May 2017.
It is nice that the British also took into account the experience of the Otkrytie Arena stadium and for three years have significantly improved the already quite good system, including in terms of architecture. At Luzhniki, set-top-box consoles are no longer needed; instead, MagicInfo players embedded in Samsung TV panels are used. Remote control of TV panels works over a LAN without an external controller and is carried out through the WEB interface of the system administrator. We will definitely tell you about Tripleplay in one of our next posts.
This project has taught us not to be afraid to offer customers new solutions, if they really suit him, and if the vendor who manufactures them is ready to support you. Everything that we offered to the stadium and introduced in 2012-2014, in 2017, is in demand and works. Perhaps this is the main indicator of the success and effectiveness of this project.
Play through
Otkrytie Arena Stadium is a cross between a shopping center, a hotel and an airport. It is designed for 45 thousand fans. This is comparable to the average throughput of Pulkovo Airport per day. To get to the stadium, these 45 thousand people must pass through the turnstiles, equipped with a ticket-passing system. Ticket system works through the LAN. Failures of the LAN or of the ticket system itself mean a crush at the turnstiles and other troubles.
')
Guests at the stadium are of four types: fans, more relaxed football fans (the so-called category "Public"), VIP and media. Journalists need Wi-Fi and wired internet access. It is important for them to send a report to the editor as quickly as possible, so that in case of the slightest problem with the Internet, they immediately start a scandal.
Media locations
In the sub-tribune space there are restaurants and trade pavilions, whose work is tied to the cash terminals. There are also televisions on which you can broadcast an advertisement or a picture from the field - this is very convenient, you can stand in line for a hot dog and watch football. If the TV breaks down, the company with which the contract for broadcasting advertising on this TV is concluded has the right to roll out a penalty to the stadium. Cash terminals also work on LAN, and worst of all, if during the match will cease to pass banking transactions. It is terrible to imagine that 45 thousand people can arrange in a confined space if they are not fed in time.
In the VIP-zone for an additional fee, you can rent a sky-box - a separate room with a view of the field, a bar, waiters and your own lodge in the stands. Wi-Fi is also working there and TVs are hanging. Renting a sky box is expensive, so here any problems with Wi-Fi and TVs will definitely turn into complaints about the stadium.
Statistics of consumption of Internet traffic at the stadium "Otkritie Arena" for the month. The most active consumers are the media and the office of Spartak football club.
Thus, 45 thousand people can come to each match, the stadium is responsible for them and makes titanic efforts so that everything goes smoothly. This is a huge stress and responsibility for everyone who works in the stadium, including IT people. The next time you cheer on your favorite team, think about it.
Space as foreboding
In January 2012, the stadium invited us to take part in a competition for SCS, LAN, PBX, server infrastructure, the system of collective reception of television (SKPT) and the system of video advertising (Digital Signage). Wi-Fi was also included there, but we will dedicate a separate post to this topic. To win the competition, it was necessary to fulfill two simple conditions: to offer the most optimal solution for the price and prove that it will work without fail during the event.
This is how the stadium looked like in December 2012, when we started designing
As the initial data there was a technical task for three sheets and a bunch of project documentation of varying degrees of responsibility. LAN and Wi-Fi were designed on Cisco.
I remembered that the Cisco Catalyst 6509 switch went to specifications without interface cards, transceivers, licenses, power supplies and SMARTNETs. Cisco was suitable for this project, but did not give advantages in price, so we decided to replace it with an analog from Huawei.
The result was cheaper and not inferior to Cisco in the reliability of operation. Of course, the counterpart from Huawei could show off a little during commissioning and was inferior to Cisco in the range of available administration guides. And so it happened, but for the sake of optimizing capital expenditures, we had to accept this.
We started thinking about how to save on servers, storage and uninterruptible power supply without sacrificing reliability. We received a call from Huawei’s Russian representative office and offered to consider the UPS2000 UPS with double conversion UPS, Huawei Tecal X86 servers and OceanStor storage systems. In 2012, it was completely unfamiliar equipment for Russia, especially when it came to UPS2000.
So it turned out a solution with an exclusive hardware, in which LAN, Wi-Fi, servers, storage and even UPS were from one vendor. In 2012, it was impossible to implement this idea with any other vendor (now from well-known vendors you can still on HPE and DELL, but on DELL - in general is exotic).
In two distant racks - UPS Huawei 2000 for server rooms. In fact, these are three 15KVA UPSs each. The total power is 45KVA. Oceanstor 5500 - on the left.
The question of reliability decided as follows. Since there was no operating experience for the proposed Huawei equipment in Russia, LANIT had to stake its 25-year-old reputation.
Under the terms of the contract, if within 5 years from the moment of commissioning the customer is not satisfied with the reliability of what we offer, we take the equipment for ourselves, put something else and pay for it ourselves.
In principle, with Huawei it was possible to go even further - take an eSpace PBX, Digital Signage vSpace, and do everything on Huawei except trays and SCS. But it did not work out. vSpace did not fit the functional (about this - just below), but for PBX the choice finally fell on Aástra (the former Ericsson, now Mitel Networks). The customer wanted to use IP DECT base stations throughout the stadium, so it was necessary to lay a station in the project with built-in support for this functionality. The cheapest of the considered and relevant requirements was Aastra.
TVs and jealousy
The stadium is a very important object, and if you think that putting equipment there without operating experience in Russia is a big risk, you are right. But let's talk about IPTV / Digital Signage.
Despite the resemblance, these systems are completely different in functionality, and at the Otkrytie Arena stadium, I still have the feeling from them like a rollercoaster ride. This is because in the course of the project, the number of TV panels was constantly changing, and almost every TV has migrated from IPTV to Digital Signage or back at least once. Therefore, we decided to design both systems on the same platform.
The customer liked the idea. He also wanted with the help of such a system to control all TVs remotely, to provide the possibility of overlaying a text streamline on a TV stream, support for Adobe Flash, a custom portal for Video-on-Demand content, and much more. Such requirements from the solutions known to us met: Cisco Stadium Vision (Cisco Systems, USA), Tripleplay (Tripleplay Services, UK) and C-nario (YCD Multimedia, USA / Israel). For all three, the stadium promised to be the first project of such a scale in Russia.
Began to compare the architecture of solutions. The first to go the distance is C-nario. It has fantastic functionality, but as a set-top-box set-top box, with which the system connects to the TV, x86-based nettops were offered based on Windows, which raised questions about reliability. Another system on the C-nario turned out the most expensive.
Stadium Vision turned out to be cheaper than C-nario, used proprietary set-top-box consoles with the Cisco logo on the front panel, and this was a plus, since in theory they are more reliable than nettops on Windows. However, Cisco was a bit jealous of Huawei and hinted that if the LAN is not Cisco, it should work, but the vendor does not guarantee anything (it is still wondering why).
Tripleplay had a completely different architecture. The British guaranteed compatibility with the LAN of any vendor, and Amino products, which, in principle, could be freely purchased in Russia and used not only with Tripleplay, were used as set-top box prefixes. With a sufficient number of licenses, any prefix could be transferred from IPTV to Digital Signage with a couple of mouse clicks in the administration panel.
All this set up in a more positive way and would look absolutely gorgeous if it were not for how it was proposed to solve the problem of remote control of TV panels in Tripleplay.
In principle, Tripleplay could control TV panels either via HDMI (but without guarantee of compatibility with a particular TV model), or via RS232, but via a USB-RS232 adapter due to the lack of an RS232 port on the Amino console. We did not like both options, because the adapters did not want to, and the customer approved the specification of TV panels almost before the purchase itself. As a result, the customer chose Tripleplay and decided to organize remote control of TV panels on a separate solution in the form of an AMX controller, which interacted with TVs via LAN.
The IPTV / Digital Signage architecture is as follows:
Frost reaction
In the yard in January 2013. There are four sectors at the stadium; according to the project, the sub-chamber space is heated only in one of them - the western one. However, even it was not heated. This had to be seen during the installation of trays and cable infrastructure.
So we went to the future stadium in the winter of 2013. Colleagues told that they saw hares who came running from the Tushino field.
Minus thirty degrees, people hang from the ceiling and pull the cistern track, which ultimately rests on the electrician's tray or fan box. This is because the representatives of related installation organizations, to keep warm, set off to build their tracks not according to the project. As a result, everyone began to interfere with each other. At each site, a consultation of engineers and technical workers of the customer and related installation organizations was held. Consilium swore for a long time, but then decided who would be laying communications where. In the next section, everything was repeated. Such an organization of work required from us the project office at the site and the landing in the form of one or two designers. Every day they went to these meetings and painted the actual lining of the trays.
The furnishings of our project office at the construction site. Photographed in the summer.
We have to deal with this kind of work organization very often. Over time, you get used to it and stop understanding people who spend a lot of time on master plans for laying engineering networks, BIM-models, wiring communications in 3D and other things that are completely useless at sites where drawings are constantly being reworked, contractors change, there is always not enough time and every week you make a new schedule.
We began to lay the cable only when it got warmer. All the same, it’s good that we laid cables in the project, which are designed for installation and operation with a frost of -40 degrees. The laying of the flute trails along the footbridges of the stadium was no less exciting, as there is very tight, slippery and a strong wind blows from the Moscow River. In just 1 year, 20 kilometers of trunk cable and 300 kilometers of twisted-pair cable were laid.
Roof walkways before the installation of low-voltage systems. Assemblers daily spent on them for 8-10 hours of working time.
Compromise
When the construction work in the server rooms ended and we started installing the UPS, one piquant UPS2000 design feature surfaced. As part of the product, there was a switchboard with automatic machines, which was turned on between the UPS and power distribution units (PDU) in the racks. If we compare the values ​​of these machines, the passport currents of the PDUs, and the values ​​of the machines of the introductory switchboard of the server room, all this did not meet the requirements specified in the Electrical Installation Rules (they are also the "Bible of electricians", or just PUE). We wrote about this in Huawei, they replied that they want to look at everything with their own eyes. So I had the strangest business negotiations in my entire career.
This distribution box came with the UPS2000.
Our account manager and three Chinese appeared in the server room. I was not introduced to them, but it was clear that these were not technical support engineers. The most important of them was wearing a gray jacket in a cage and was about one and a half meters tall. The other two wore black jackets and were slightly higher than the main thing, which was clearly embarrassed. Noise punch. The air was hazy from construction dust, and we stood near the racks with UPS, because there was nowhere to sit down. My height is eighty meters. When a Chinese in a gray jacket spoke to me, he looked up at me and still looked presentable. His speech was a mixture of Chinese and English. He sincerely did not understand what the problem was. I talked about the EMP, the principles of the selectivity of automata, which must be observed by everyone who is within the borders of the Russian Federation, but his expression still has not changed. All that he answered me, sank in the rumble of a perforator. When the puncher subsided, the Chinese switched to Russian: "We can change the decision, but then we are not responsible for it." “According to our laws, you don’t answer for him anyway, but I answer as the chief engineer of the project,” I replied. As a result, we agreed that the vendor will bring the solution into compliance with the requirements of the OLC.
In fact, almost all foreign vendors with whom we deal are not always devoted to the subtleties of the Russian GOST and SNIP. In some cases, it threatens to completely replace the technical solution. It is good when the vendor understands this and goes into a constructive dialogue without unnecessary altercations and bureaucracy.
Equipment is being installed
Dirty dancing
The less time left before the end of the project, the brighter became the feats of labor builders. Brigades of painters and plasterers increasingly wielded perforators in rooms with commutators working and sucking dust. There was a threat of mass failure of LAN and data center equipment for a clearly non-guaranteed reason. They tried to get out of the situation by blowing “iron” with cylinders with compressed air and even by gluing the interface ports of the equipment.
One of the switches during commissioning. Ports unsophisticated plastered masking tape. If you look closely, you can see a layer of building dust on the top panel of the device.
SIP Fight
In the meantime, the customer decided to connect the PBX not via E1, as it was in the project, but via SIP, for which border session controllers were required. A month after the decision was made, two Huawei AR2240 routers with voice processing cards were delivered to the stadium.
After launch, these devices categorically refused to be SIP-friendly with Aastra. The firmware change did not help, and we wrote a call to Huawei service support (TAC). At first, Huawei engineers connected to the AR2240 remotely, and when that did not help, the pilgrims of the vendor’s experts began directly to the site to fall on the console of the routers and fine-tune the SIP parameters, the total number of which exceeded 100 pcs. Everything ended with a patch for OS AR2240, which was developed personally for our project. After the patch, SIP earned, but not in the way we would like. I especially remember a bug that caused the connection to break after 30 seconds when calling from Aastra to numbers with an auto attendant (DISA). The problem was solved by another patch of the router software.
There is nothing special to remember about Tecal servers - regular X86 servers. Worked steadily, there are no complaints about the build quality. At the same time, we got acquainted with the HPE iLO analog for monitoring and remote server administration - this is called iMana for Huawei Tecal, and it works quite well.
Two Oceanstor S5500T storage systems also worked without any problems, except that they were upset by the lack of clustering support at the time, such as HP Peer Persistence or NetApp MetroCluster, but it didn’t work for us. Oceanstor currently supports clustering, a technology called HyperMetro. About her there is an article on Habré here .
Football in the distance
Toward the close of construction, Spartak was scheduled to have an away match. The match should have been shown on TV live. The customer was inspired by the idea to display a picture of this broadcast on the video table of the stadium, the sound on the sound reinforcement system and invite fans to the stands. It was proposed to take a live TV signal from Tripleplay with our SKPT. By this time, we’ve almost finished commissioning.
Video board of the Otkritie Arena stadium during the opening ceremony. It is suitable for watching television.
The challenge was to integrate the TPCS, for which we were responsible, with a video-wall system, for which one contractor was responsible, and a sound-amplification system, for which another one was responsible. We decided to integrate not directly, but through the Media Control Room of the stadium, for which another contractor was responsible.
Media Control Room - central hardware multimedia systems of the stadium. It forms the content that is given to all the display means of the stadium.
It was necessary to somehow remove the signal from the IP-TV, feed it to the Media Control Room's matrix switcher, scale it and divide it into a video and audio component. Then the video was sent to the board controllers, and the sound to the mixing console of the sound reinforcement system. At the same time, video and sound should not lag behind each other.
As a result, we purchased two more Amino A140 consoles, registered them in the Tripleplay system and brought them to a TV channel, through which we had to broadcast the match. The HDMI interface of the set-top boxes through the HDMI-SDI converters were included in the Media Control Room's matrix switcher. And it all worked.
It was very cool to sit in the stands of the stadium and watch the match on the video table of 172 sq. M.
Present continious
Now we are commissioning Huawei equipment for one of our customers. Imagine, the customer found the format of the output of one of the commands in the console of the Huawei NGFW firewall uncomfortable. We contacted Huawei, and the vendor released a personal patch for Huawei NGFW customer, which changes the output format of the console command. This is a continuation of Huawei’s flexible policy that helped us build a successful project on a new equipment for the Russian market.
The IT infrastructure construction project at the Otkritie Arena stadium took us 2 years. 4500 SCS ports were laid, SKPT was made on 180 TV panels, two storage systems for 50TB each were put into operation, four two-socket Rack-mount servers for the virtualization platform, 87 Wi-Fi points, PBX for 250 subscribers and 36 DECT base stations.
The stadium became one of the first facilities in Russia, where Huawei Oceanstor storage systems, UPS2000 UPS, Tecal servers, and DCPA / Digital Signage based on the Tripleplay solution were used. Today LANIT is engaged in service support of these solutions. These are support decisions. During operation, they managed to acquire other IT services, including a deposit club card, which can be used both to enter the stadium and to pay for purchases / additional services on the territory of the sports complex.
The stadium is actively being upgraded to receive FIFA matches in 2018. It affects the IT infrastructure. And quite seriously, because in 2011, when we were designing, the Otkrytie Arena stadium did not plan to participate in this event.
In the hands of the IT director of the Otkrytie Arena stadium, Igor Shestunov, the Talmud of FIFA requirements for stadiums (about 900 pages)
The project was pretty successful. In 2014, Huawei announced it the “Project of the Year”, and the patch written by its engineers became part of the next version of the Huawei OS AR2240. The customer also responds positively to the reliability of Huawei equipment, including UPS2000. We have not had the opportunity to evaluate the work of the Huawei service center for the delivery of new equipment instead of what we have been dusting, since during the whole period of operation no Huawei device has failed.
There are no complaints about the Tripleplay IPTV / Digital Signage solution. Positive recommendations from the IT service of the Otkrytie Arena stadium allowed us to offer a similar solution for the reconstruction of the Luzhniki stadium, which ended in May 2017.
It is nice that the British also took into account the experience of the Otkrytie Arena stadium and for three years have significantly improved the already quite good system, including in terms of architecture. At Luzhniki, set-top-box consoles are no longer needed; instead, MagicInfo players embedded in Samsung TV panels are used. Remote control of TV panels works over a LAN without an external controller and is carried out through the WEB interface of the system administrator. We will definitely tell you about Tripleplay in one of our next posts.
This project has taught us not to be afraid to offer customers new solutions, if they really suit him, and if the vendor who manufactures them is ready to support you. Everything that we offered to the stadium and introduced in 2012-2014, in 2017, is in demand and works. Perhaps this is the main indicator of the success and effectiveness of this project.
Source: https://habr.com/ru/post/335242/
All Articles