Distributed backends for 2GIS video advertising on .NET Core and Kubernetes
Hello!
We continue to talk about development companies that use Microsoft technology in their projects. In today's issue is the company 2GIS , whose main office is located in cold Novosibirsk and to which the author of the article feeds the warmest feelings.
2GIS recently completed a large project using the core functionality of .NET Core, Kubernetes and many Linux . I often get questions about whether .NET Core can be used in production, what is our experience of using it in a fully open source environment. I asked the guys to tell you how. Read more under the cut, it's worth it - this is one of the first public and large projects that are built on .NET Core, and even in the full Open Source around.
')
It is on behalf of the architect of the project, Denis Ivanov (on the DenisIvanov habr ), Microsoft MVP on .NET, the human back-end of the famous CodeFest conference. Inna Savchkova (on the InnaSavchkova habr ), Tech PR Manager 2GIS, provided our conversation with a beautiful and understandable language and, of course, a dash instead of hyphens. Questions asked ahriman .
Hello! What is 2GIS?
2GIS is a company that provides complete and accurate information about 330+ cities in 9 countries. This is all that may be needed in the city: streets, houses, companies operating in different areas, their contacts and useful information about them (such as, working hours and average bill in a restaurant or user reviews of travel companies), entrances to buildings , ways to travel and much more. Now 2GIS "knows" everything about all 2.3 million firms in Russia with an accuracy of 95%.
The company has a separate product team, which constantly analyzes what users are telling us, verifying the applicability of new ideas, some of which later become the basis for new 2GIS product features.
2GIS uses a wide range of technologies to build its systems and products. However, this is mainly Open Source. It's simple - it works on all platforms (and 2GIS also has a bunch of projects on GitHub - https://github.com/2gis . Author's note)
The RnD-department of the company has more than 20 teams that create tools for collecting and organizing information, cartographic systems and services, tools for selling and placing advertising, online, offline and mobile applications.
Microsoft recently released an official press release about your project . So you decided to make a new project with a small team on .NET Core, Kubernetes & Linux. Who decided to use such a bundle? Why? What for? What benefits have you seen? What are the potential problems?
Profit 2GIS receives from the sale of advertising opportunities to companies that want to be more visible to users of 2GIS products.
One of the ideas that needed to be tested was the hypothesis of advertisers' interest in placing videos. Prior to this, only text or graphic advertising could be placed in 2GIS.
The main goal was to bring the tools for the sale and placement of video advertising to the market as soon as possible. It was decided to entrust this task to developers from a team that knows the subject area well and has extensive experience in developing internal advertising sales tools. Internal systems are implemented on the .NET Framework and use the Windows platform for deployment. However, the requirements for high availability, resiliency and performance of video advertising services prompted the use of the same infrastructure and operating systems as 2GIS online products. Fortunately, at the time of development, the cross-platform .NET Core was already half a year released for widespread use, and after testing and creating a prototype, it was decided to use the .NET Core bundle, Kubernetes & Linux to implement video advertising services.
This made it possible for two programmers and an analyst to complete a task in just 2 months. This time includes all the work: starting with a detailed study of requirements and ending with release into the combat infrastructure. In addition, it became possible to use any convenient development tools and operating systems, including Windows and macOS.
Project team - from left to right engineers and project manager
Where is the “heart” of the project? How do you ensure that global customers get the same level of service in this structure?
Kubernetes cluster is used in 2GIS as a platform on which many services of the company are launched. In fact, there are three such clusters - two in Russia and one in Europe. This is a well-managed infrastructure with all the necessary properties: balancing, multi-level fault tolerance, detailed monitoring and the ability to receive and analyze any logs and custom telemetry from applications. Therefore, the choice of where to deploy backend video advertising services, in fact, did not stand.
Project architecture
From the point of view of storage, transcoding and sharing video files, everything is a little different. To solve these problems, we use the services provided by the cloud partner, specializing only in working with video. In terms of classic 3-layer architectures, the video advertising service we created at 2GIS is business logic, the API for storing, transcoding and uploading video files (CDN) is data storage, and all 2GIS products are frontend.
In this scheme, all our backends are geographically distributed, so all customers receive approximately the same level of service.
Returning to the general details of the project - tell us about the current indicators (users, workload). Is the project already in production or still being tested?
Service in production since April 2017, and now we have about 20,000 video ad requests per day. More than 80% of them we answer from the cache. This gives us the opportunity to keep quite high loads, although at the current stage we have no more than 100 RPS at the peak.
What architectural solutions used in the project?
If this is a global project, then the architecture should be suitable for high loads from different places. What do you think, what architectural patterns / approaches of the masthead for such a project? What tools are used to monitor what is happening, have you experienced difficulties? DDOS?
As for the used architectural solutions and approaches, video advertising service is not much different from other distributed applications designed for uninterrupted work with a large audience of users. There is a whole set of best practices for building distributed applications, for example, you should pay attention to the course of Jeffrey Richter .
The very first thing to consider is the proper use of server resources . The threads used should not be blocked for long; this is achieved by writing asynchronous code. If the same data is often requested, you need to use a cache. If there is little such data, then even a cache in memory will suffice. You should always keep in mind that short-term failures are possible in distributed systems (network “flashing”, restarting infrastructure services, etc.). Here quite simple patterns in use, such as Timeout, Retry, Fallback and their combinations, help. There is an excellent Polly library that provides a collection of similar off-the-shelf tools.
Difficulties, of course, are. They occur when any of the infrastructure or partner services go to prevention or when there is a disruption in the work of the network of providers. Despite the “youth” of .NET Core and the relatively small number of uses of this platform for building such solutions, we have a particularly positive experience - we did not encounter any “special effects” or strange behavior of the application due to the JIT compiler, garbage collection or other CLR components. We really hope that this trend will continue.
A few words about DDoS or similar attacks. Since we use the common infrastructure of the company, we did not have to solve all these issues. Here we completely rely on the high qualifications of our infrastructure engineers and their tracking and protection solutions.
What are the growth plans?
Video advertising is now becoming one of the basic tools for attracting users' attention to advertisers, along with text and graphic advertising. There are already several advertising positions where video advertising is used.
We are thinking about using our own infrastructure for storing source videos in order to reduce, first of all, legal risks. We also continue to improve internal processes and systems for managing and selling advertising in general, and video advertising in particular. All this provides increased efficiency within 2GIS.
The growth is fraught with the danger of a service dropping under a heavy load. Whats up?
2GIS built all the necessary infrastructure for load testing services based on Gatling. All that is needed to use it is to deploy the application on special “iron” servers running Kubernetes, and to write scripts of load tests on Scala. All necessary metrics, comparisons of results between launches, reports and graphs are provided by the platform for load testing.
Thank!
You can read more about the technologies used in 2GIS in the company's corporate blog. Technical reports, in which there is a lot of real experience of all that is listed above, are here .
We continue to talk about development companies that use Microsoft technology in their projects. In today's issue is the company 2GIS , whose main office is located in cold Novosibirsk and to which the author of the article feeds the warmest feelings.
2GIS recently completed a large project using the core functionality of .NET Core, Kubernetes and many Linux . I often get questions about whether .NET Core can be used in production, what is our experience of using it in a fully open source environment. I asked the guys to tell you how. Read more under the cut, it's worth it - this is one of the first public and large projects that are built on .NET Core, and even in the full Open Source around.
')
It is on behalf of the architect of the project, Denis Ivanov (on the DenisIvanov habr ), Microsoft MVP on .NET, the human back-end of the famous CodeFest conference. Inna Savchkova (on the InnaSavchkova habr ), Tech PR Manager 2GIS, provided our conversation with a beautiful and understandable language and, of course, a dash instead of hyphens. Questions asked ahriman .
Hello! What is 2GIS?
2GIS is a company that provides complete and accurate information about 330+ cities in 9 countries. This is all that may be needed in the city: streets, houses, companies operating in different areas, their contacts and useful information about them (such as, working hours and average bill in a restaurant or user reviews of travel companies), entrances to buildings , ways to travel and much more. Now 2GIS "knows" everything about all 2.3 million firms in Russia with an accuracy of 95%.
The company has a separate product team, which constantly analyzes what users are telling us, verifying the applicability of new ideas, some of which later become the basis for new 2GIS product features.
2GIS uses a wide range of technologies to build its systems and products. However, this is mainly Open Source. It's simple - it works on all platforms (and 2GIS also has a bunch of projects on GitHub - https://github.com/2gis . Author's note)
The RnD-department of the company has more than 20 teams that create tools for collecting and organizing information, cartographic systems and services, tools for selling and placing advertising, online, offline and mobile applications.
Microsoft recently released an official press release about your project . So you decided to make a new project with a small team on .NET Core, Kubernetes & Linux. Who decided to use such a bundle? Why? What for? What benefits have you seen? What are the potential problems?
Profit 2GIS receives from the sale of advertising opportunities to companies that want to be more visible to users of 2GIS products.
One of the ideas that needed to be tested was the hypothesis of advertisers' interest in placing videos. Prior to this, only text or graphic advertising could be placed in 2GIS.
The main goal was to bring the tools for the sale and placement of video advertising to the market as soon as possible. It was decided to entrust this task to developers from a team that knows the subject area well and has extensive experience in developing internal advertising sales tools. Internal systems are implemented on the .NET Framework and use the Windows platform for deployment. However, the requirements for high availability, resiliency and performance of video advertising services prompted the use of the same infrastructure and operating systems as 2GIS online products. Fortunately, at the time of development, the cross-platform .NET Core was already half a year released for widespread use, and after testing and creating a prototype, it was decided to use the .NET Core bundle, Kubernetes & Linux to implement video advertising services.
This made it possible for two programmers and an analyst to complete a task in just 2 months. This time includes all the work: starting with a detailed study of requirements and ending with release into the combat infrastructure. In addition, it became possible to use any convenient development tools and operating systems, including Windows and macOS.
Project team - from left to right engineers and project manager
Where is the “heart” of the project? How do you ensure that global customers get the same level of service in this structure?
Kubernetes cluster is used in 2GIS as a platform on which many services of the company are launched. In fact, there are three such clusters - two in Russia and one in Europe. This is a well-managed infrastructure with all the necessary properties: balancing, multi-level fault tolerance, detailed monitoring and the ability to receive and analyze any logs and custom telemetry from applications. Therefore, the choice of where to deploy backend video advertising services, in fact, did not stand.
Project architecture
From the point of view of storage, transcoding and sharing video files, everything is a little different. To solve these problems, we use the services provided by the cloud partner, specializing only in working with video. In terms of classic 3-layer architectures, the video advertising service we created at 2GIS is business logic, the API for storing, transcoding and uploading video files (CDN) is data storage, and all 2GIS products are frontend.
In this scheme, all our backends are geographically distributed, so all customers receive approximately the same level of service.
Returning to the general details of the project - tell us about the current indicators (users, workload). Is the project already in production or still being tested?
Service in production since April 2017, and now we have about 20,000 video ad requests per day. More than 80% of them we answer from the cache. This gives us the opportunity to keep quite high loads, although at the current stage we have no more than 100 RPS at the peak.
What architectural solutions used in the project?
If this is a global project, then the architecture should be suitable for high loads from different places. What do you think, what architectural patterns / approaches of the masthead for such a project? What tools are used to monitor what is happening, have you experienced difficulties? DDOS?
As for the used architectural solutions and approaches, video advertising service is not much different from other distributed applications designed for uninterrupted work with a large audience of users. There is a whole set of best practices for building distributed applications, for example, you should pay attention to the course of Jeffrey Richter .
The very first thing to consider is the proper use of server resources . The threads used should not be blocked for long; this is achieved by writing asynchronous code. If the same data is often requested, you need to use a cache. If there is little such data, then even a cache in memory will suffice. You should always keep in mind that short-term failures are possible in distributed systems (network “flashing”, restarting infrastructure services, etc.). Here quite simple patterns in use, such as Timeout, Retry, Fallback and their combinations, help. There is an excellent Polly library that provides a collection of similar off-the-shelf tools.
Difficulties, of course, are. They occur when any of the infrastructure or partner services go to prevention or when there is a disruption in the work of the network of providers. Despite the “youth” of .NET Core and the relatively small number of uses of this platform for building such solutions, we have a particularly positive experience - we did not encounter any “special effects” or strange behavior of the application due to the JIT compiler, garbage collection or other CLR components. We really hope that this trend will continue.
A few words about DDoS or similar attacks. Since we use the common infrastructure of the company, we did not have to solve all these issues. Here we completely rely on the high qualifications of our infrastructure engineers and their tracking and protection solutions.
What are the growth plans?
Video advertising is now becoming one of the basic tools for attracting users' attention to advertisers, along with text and graphic advertising. There are already several advertising positions where video advertising is used.
We are thinking about using our own infrastructure for storing source videos in order to reduce, first of all, legal risks. We also continue to improve internal processes and systems for managing and selling advertising in general, and video advertising in particular. All this provides increased efficiency within 2GIS.
The growth is fraught with the danger of a service dropping under a heavy load. Whats up?
2GIS built all the necessary infrastructure for load testing services based on Gatling. All that is needed to use it is to deploy the application on special “iron” servers running Kubernetes, and to write scripts of load tests on Scala. All necessary metrics, comparisons of results between launches, reports and graphs are provided by the platform for load testing.
Thank!
You can read more about the technologies used in 2GIS in the company's corporate blog. Technical reports, in which there is a lot of real experience of all that is listed above, are here .
Source: https://habr.com/ru/post/345706/
All Articles