Wi-Fi under the Neva: how we built a network in the deepest metro of the world

We started our blog on Habrahabr with our first and main project - we talked about how four years ago we built the world's first free Wi-Fi network in the rolling stock of the Moscow metro. Having received the first experience of building a Wi-Fi transport network, our team implemented the project again, but for the residents of the Northern capital.

SAME-SAME BUT DIFFERENT

The network architecture in the St. Petersburg metro we did almost the same as in Moscow. In the same way, the rolling stock in the tunnels travels in the zone of continuous radio coverage, an intra-train network is created and a stationary infrastructure is deployed.

However, when planning and designing the St. Petersburg network, we:

• accurately calculated the radio coverage of the tunnels;
• changed the composition and functions of active equipment;
• used the equipment of new generations (of course, four years have passed);
• virtualized more services than in Moscow;

• used antennas of domestic development and production.

  
  

  But first things first.

  
  

SEVEN TIME

When we were planning a network in Moscow, there was no proven in our world an approved mathematical model for calculating the propagation of radio waves in tunnels. Radio planning was done semi-empirically. We have developed our own tool that can calculate the optimal placement of base stations in tunnels and with its help conducted radio network planning in St. Petersburg.

What exactly did we do?

1. We developed a theoretical model of signal propagation in tunnels and a system-level model.
2. We created several measurement systems for carrying out full-scale measurements and model calibration / adjustment.
3. We carried out several iterations of measurements and refinements of the model, until it began to give data that completely agreed with the results of measurements.

As a result, we got a tool for practical radio planning of wireless communication networks in metro, which has no analogues in the world.

At one time, when planning a network in Moscow, we only had approximate rules for setting up the BS (maximum and minimum distances, etc.) that we used. We conducted tests in the tunnels to check the assumptions made and made some adjustments to these rules based on the results. Now we know (having conducted a study of our network in Moscow with the help of our new planning tool) that many decisions made at the beginning of the project about the placement were not optimal. Now we would put many BS in a different way and in general would increase their number by about 15%. We plan to gradually upgrade the network in Moscow, optimizing the radio coverage.

In St. Petersburg, we received the target optimal BS arrangement, which allowed us to actually get the specified target indicators of speed and signal quality in the communication channels immediately after the launch of the network.

LESS IS BETTER

When planning a network in Petersburg, we tried to reduce the amount of active equipment, and also use only equipment that supports remote monitoring. This has helped us reduce operational costs.

We have abandoned the use of tunnels and switching nodes of media converters, which convert the optical signal into electrical and vice versa. Instead, they switched to a fully optical connection, which required new type base stations and more modern switches. Due to more careful design, we have reduced the number of active equipment due to the fact that now there is one switch for 2.5-3 stations. In Moscow, there is a pair of switches for every two stations. All these changes helped us save the project budget, which was also one of the important tasks.

In addition, the bandwidth between the station switches and the core was increased. And if in Moscow these are two georeserved channels of 1 Gbit / s, in St. Petersburg the capacity of each of them is increased to 10 Gbit / s.

Antenna specially designed for us by Zelenograd company Delta Satellite helped to improve the quality of the network. Thanks to them, we were not only able to significantly reduce the cost of antenna systems, but also achieved improved radio path characteristics compared to those achieved in Moscow.

We also changed the intra-train network: now all the cars are interconnected by two reserved channels of 1 Gbit / s (in Moscow, I remind you, the backup channel has a capacity of 100 Mbit / s). The more modern Cisco AIR-CAP2702i access points with support for the standard 802.11ac and MIMO 3x4 are installed in the cars, which can provide an improved quality of service for modern user devices.

CHANGE OF GENERATIONS

Another major change is the use of a new train-tunnel communications radio equipment. Since the launch of the network in Moscow, the generations of transport equipment have changed. Now available devices with much better performance than four years ago. In the project of the Petersburg Metro, we used the equipment of the company Fluidmesh, developed in Italy. As in the project in Moscow, the equipment is built on the basis of Wi-Fi-chips. But if 802.11n chipsets are used in Moscow, then in Petersburg there are new 802.11ac chipsets that support wider frequency bands (and therefore data transfer rates). In addition, the modified antenna systems in St. Petersburg provide support for the MIMO 2x2 mode better than the antenna systems of the Moscow network. Due to this, as well as due to optimal radio planning, the maximum throughput of the train-tunnel channel has increased from 120 Mbit / s in Moscow to 500 Mbit / s in St. Petersburg.

If we delve a little into the technology, then it should be noted that the communication equipment that we use in Moscow significantly modifies the standard level of access control to the medium (MAC level), transforming the TDD half-duplex Wi-Fi protocol into a synchronous TDM protocol. This leads, among other things, to static separation of capacity between the ascending and descending communication channels in the “composition-tunnel” section, and to the need for organizing the time synchronization of all base stations of the network using separate hardware clock sources. Fluidmesh equipment also modifies the Wi-Fi protocol to improve its performance with our usage model. We do not forget that standard Wi-Fi is, first of all, a protocol for accessing multiple subscribers to a shared network, and not a protocol for organizing communication between a small number of subscribers in motion (mobile PmP transport). But the Fluidmesh technology retains the principle of dynamic distribution of capacity between the descending and ascending channels and does not require time synchronization, which simplifies the structure of the network and increases the efficiency of using channel capacity.

Another major advantage of the selected equipment is the network architecture. Fluidmesh allows you to build a packet-based network on the mesh principle by organizing a transport protocol using MPLS-like packet labeling. This approach simplifies setup and management, allows you to completely avoid a number of standard Ethernet-specific problems that manifest themselves in networks with moving clients with roaming (handover), for example, problems of updating MAC address tables (MAC learning). Due to, among other things, the new architecture in St. Petersburg, the latency (delay) in the network is several times less than in Moscow, and the level of packet loss when switching trains between base stations is very low (almost zero).

VIRTUALIZE IT

New for the St. Petersburg network has become the virtualization of a part of network services (the IT layer has been fully virtualized in Moscow from the very beginning). Instead of a dedicated controller of access points in the head car, we used industrial computers and a virtual controller. From the point of view of managing a Wi-Fi network in rolling stock, nothing has changed, but other possibilities have appeared:

• In St. Petersburg, user and advertising content can be stored (cached) in the train itself on servers in the head cars, this has made it possible to speed up access to frequently required content units for subscribers and relieve the “component-tunnel” communication channel;
• now in each train, in each head car a test device is installed (the same industrial PC :). With it, we will be able to automatically measure the parameters of the channel, transmit to the network control center much more objective and accurate information about the operation of the network of the train. This can significantly improve operating procedures and enhance real user experience;
• and, finally, in case of an emergency on the network, we can activate more detailed, load and other tests in order to localize and eliminate the accident more quickly.

The industrial computer was taken with a large margin of computing power and with the possibility of an upgrade, which allows us to further increase the number of services and improve the quality of services.

FASTER, DEEPER, POWERFUL (network, of course, network)

We grow up and learn from our mistakes, as well as improve and expand our competencies. Our solutions are becoming more sophisticated and technologically advanced, and we can already say with confidence that within the framework of building wireless networks in difficult conditions, there will be no impossible tasks for us. But communication, as such, has long been no longer a service in itself, but merely acts as a transport for more and more complex both client and technological services and various other large and not very data. We are actively working in this direction. We already have a number of interesting solutions that we have created and are actively using, and we will tell about them in the next posts.