The birth of a virtual mobile operator: the Bank Tinkoff project

Today, banks are increasingly “digitizing” client services and communication channels with their clients: they personalize services based on client data, implement remote self-service services, chat bots, virtual assistants, including systems with artificial intelligence (AI) and speech recognition technologies . In an effort to expand the range of customer services, banks are developing new areas of activity for themselves. One example is mobile virtual network operators (MVNO, mobile virtual network operator). Now, together with Tele2, we are participating in the Tinkoff Bank project to create a virtual operator, Tinkoff Mobile.

What is a “virtual mobile operator”?

On Habré there are already good articles explaining the essence of the “virtual mobile operator”. Here we just say that the virtual mobile operator is a virtual (software) network managed by one company, using a physical communications network owned by another company.



Tinkoff Mobile is built according to the Full MVNO scheme. This means that when all components are put into operation, in terms of creating client services, a new virtual operator will differ little from the “real” operator. Tinkoff Mobile will lease the resources of the base station network from the partner operator, which is Tele2, with the core network infrastructure and business applications of the MVNO being its own. All client traffic will go through a virtual operator, which is actually for subscribers the point of entry into the Internet.

')

Virtual operator architecture

Our company was chosen to build elements of the mobile network core. For us, this was the first such project - today MVNO is small in the world, especially in Russia. From the start of the project to making the first client call, it took only 4 months, it is very fast by the standards of the telecom industry. The system is designed so that it can be upgraded without stopping the work, simply adding computing nodes.



What are the "cubes" of the virtual operator architecture? At the moment it looks like this:







The infrastructure consists of two identical, fully autonomous sites with geo-reserve function: when one site fails, subscribers will automatically continue servicing to the second. At the same time, both sites normally work in active mode, equally dividing the traffic of subscribers. All systems have multiple backups, as is common in the industry. In addition, high availability clusters are widely used to increase resiliency.



All the components shown in the diagram are software executed on ordinary serial servers. These are mainly solutions of not traditional large telecom vendors, but new products of companies that actively enter the market of telecom solutions and crowd out old players. In particular, PGW / GGSN nodes from Affirmed Networks, PCEF (DPI) nodes from Procera Networks and Oracle PCRF nodes are used. Of course, traditional vendors also began to move towards virtualized software solutions, but they are still not able to completely “get rid of” their old technologies, while new companies do not have such a legacy, and their solutions can be executed on any platforms, on any servers of any manufacturers.



In the telecom industry, Network Functions Virtualization (NFV), the virtualization technology of the network elements of the telecommunications network, is in fact increasingly used, in fact, the virtualization of components that provide services for telecom operator subscribers. Today, one of the main trends is the abandonment of the specialized and very expensive equipment of traditional vendors in favor of serial servers (commercial off-the-shelf, COTS) of the x86 architecture with specialized software installed on them. Moreover, it works under the control of a virtual infrastructure, that is, it is launched inside virtual machines.



NFV uses various virtualization technologies. In addition to basic hardware virtualization, which allows you to run software modules on a virtual machine that emulates interaction with a real server hardware, you can build a software-defined SDN network based on virtual elements of network functions.



With high network loads it is very important to ensure reliable interaction with the network subsystem of the software modules running inside the virtual machine. That is, the issue of speed with network interfaces is particularly acute. One way is to use the PCI Passthrough mode, in which the entire PCI device is transferred to the control of the guest operating system. This allows it to work with the device directly, without using the emulation layer on the hypervisor side. However, this is a resource-intensive way, it does not scale and binds the guest OS, and therefore the network function, to a specific instance of a PCI device.



Another disadvantage of the PCI Passthrough mode is the low density of resource allocation due to the impossibility of sharing a single device by several guest OS, because each guest OS in this mode uses the device exclusively. Therefore, we proposed an alternative approach - Single Root I / O Virtualization (SR-IOV) technology.



SR-IOV allows you to use the device directly, as in PCI Passthrough mode, bypassing the hypervisor. But at the same time, the device is available simultaneously for several virtual machines, independent processing of interrupts and DMA for each machine is performed using the technology of Virtualization Technology for directed I / O (VT-d).



When SR-IOV is enabled, the network device is “split” into one physical function (Physical Function or PF) and several virtual functions (Virtual Function or VF). The physical function (PF) remains at the level of the hypervisor and under its control. Virtual functions (VF) are transferred to the guest OS and become virtual network function (NFV) network interfaces for interacting with the outside world. The issue of VF performance within VNF is solved by using the Data Plane Development Kit (DPDK) framework. It was originally developed by Intel, and then transferred to the open community. The framework can significantly improve the performance of NFV when processing network traffic. Using a combination of DPDK and SR-IOV to virtualize network functions is a mandatory requirement when building a high-performance NFV solution, since the speed of the Internet on subscribers' smartphones depends on it.

Results

As already mentioned, the project was implemented very quickly. The composition of the equipment had to be determined at the very first stage of the project, so that by the time the work on the system design was completed, the equipment was already delivered to the sites and ready for installation. The task was also complicated by the use of solutions from different vendors, which required the coordinated work of all the teams working on the project.



After the launch of the system, the first call and the first access to the network on a smartphone with a SIM card of a new virtual mobile operator, the business rules were adjusted for two more months, and acceptance tests were carried out. As a result, the system went into commercial operation on December 14, 2017.