CDN for statics and measurement: how Yandex.Mail has become faster in the regions
The speed of the web interface is a very important thing, and we at Yandex understand this very well. The feeling of lightness and the speed with which the user loads important elements may depend on how he will use the service.
We in Yandex. Mail are working a lot to accelerate the client side. Only on Habré, we talked about how to optimize the layout , abandoned XSL and switched to a new template engine .
But to increase the speed of work can not only by optimizing the frontend. Today we want to talk about moving static files Yandex. Mail on a special Yandex CDN for statics and how it accelerated its work, especially in the regions.
')
Details on where we moved, how the new hosting works, why it works that way, and how we measured how fast it has become - read on.
When a person enters the mail, the first thing that he loads is minimal HTML and the so-called bootstrap (loader). It is responsible for the asynchronous loading of all the necessary JavaScript modules, templates and CSS files for drawing the interface, as well as for the initial data loading. In Yandex.Mail, HTML is loaded from our main domain, mail.yandex.ru, and bootstrap, from the static cluster mailstatic.yandex.net.
It includes about a hundred machines located in several data centers in Moscow and the Moscow region. The only task of this cluster is to give the client the static files that are needed for the display and operation of the Mail web interface.
To see the problem of speed in full growth, you need to think about the geography of Yandex.Mail users. Even if we talk only about Russia, the largest country in the world. For example, if you focus on the length of the Trans-Siberian Railway, the distance between Moscow and Vladivostok is almost 9300 km. Packages with user data, of course, do not move by rail - they travel by lines of communication. And the data transfer rate in optical fiber is about one and a half times lower than the speed of light in a vacuum. Network equipment enters the way along the data path as well. Thus, RTT from our data centers in Moscow to Vladivostok is 120-130ms, and, for example, to Novosibirsk - about 60ms.
To solve speed problems, Yandex has CDN hosting - yandex.st. Yandex.st is several clusters of machines, most of which, like mailstatic.yandex.net, are located in Moscow data centers. But, unlike mailstatic.yandex.net, it has representative offices in many large cities of Russia, the near abroad and the world.
At the moment, our CDN-hosting supports more than three hundred providers in 11 regions. By the way, yandex.st is the third most popular hosting jquery library in the world .
It works on anycast technology. This means that the user on the file request is answered topologically by the server closest to it. That is, if a person is in Vladivostok, his request will be sent to a server located in Vladivostok, which means he will receive an answer faster than if his request went to Moscow.
Before you move, you had to evaluate how effective the move will be and what benefits it will bring. To do this, we hung the web interface of the Mail with sensors, built graphics and transported only 50% of users to yandex.st. This was done in order to be able to compare everything with the standard - mailstatic.
We have built in the client part of the web interface a special timer for loading the statics. The Navigation Timing API, which is available in most modern browsers, helped us with this. The API, up to milliseconds, knows at what point the user starts opening mail, and the javascript platform knows when all the necessary static files are loaded. If we subtract the first from the second timestamp, we can get the time for which all the static was loaded. The received number along with other metrics is sent by a GET request from the user to special servers.
Unlike the nginx access logs timings (for which, of course, we also have graphics), this metric allows you to take into account the download time of all mail, regardless of what files and how much each user loads. Roughly speaking, we can know for sure how long the user has loaded all mail, and not every file separately.
In less than one percent of cases, due to unclear reasons, these figures arrive frankly implausible. For example, negative or equal to 50 years.
In order to cut off such values, we decided to build a timeline in which 75%, 95% and 99% of downloads fit.
But these figures are not enough - the download speed varies, including due to changes in file sizes and number. In order to understand how in other regions the static of mail began to load faster, we chose to compare the download speed in each region with the download speed in Moscow as the KPI relocation. That is, we have started to take away from the loading time from the region the loading time from Moscow. If the value turned out to be negative, we considered that we were doing everything correctly and the speed in this region became faster.
This chart helped find a few bottlenecks and problems on regional machines, which we quickly fixed.
As a result of the move, Yandex.Mail started responding and loading faster: depending on the region, from several hundred milliseconds to seconds. Static loading speed in Novosibirsk and Kazan doubled, in Yekaterinburg and in Vladivostok - about one and a half.
Graph of average statics loading time in some cities
Thanks to the setting up of regional servers, we found several opportunities to increase the speed in the “home” region: Yandex.Mail has become faster not only in the regions, but also in Moscow, so we invite everyone to start using yandex.st.
If your project uses jquery or other popular javascript frameworks and libraries, you can connect them from our servers .
In addition to regionality, yandex.st has round-the-clock support by Yandex administrators. It supports the ipv6 protocol and has a minimum downtime: the more people use yandex.st, the more chances that jquery will not be “flushed” from the cache. And, of course, Yandex services already take libraries from yandex.st, so that regular users often have these versions in the cache. And this means that if you use yandex.st on your site, it will load faster.
Not all regional providers have configured routing so that their users fall on local machines. If you represent any provider and want to speed up Yandex for your users (and reduce traffic in his direction), you must ensure the topology of our CDN servers that support yandex.st (and other services). There are two ways to do this:
We in Yandex. Mail are working a lot to accelerate the client side. Only on Habré, we talked about how to optimize the layout , abandoned XSL and switched to a new template engine .
But to increase the speed of work can not only by optimizing the frontend. Today we want to talk about moving static files Yandex. Mail on a special Yandex CDN for statics and how it accelerated its work, especially in the regions.
')
Details on where we moved, how the new hosting works, why it works that way, and how we measured how fast it has become - read on.
What slows down the download?
When a person enters the mail, the first thing that he loads is minimal HTML and the so-called bootstrap (loader). It is responsible for the asynchronous loading of all the necessary JavaScript modules, templates and CSS files for drawing the interface, as well as for the initial data loading. In Yandex.Mail, HTML is loaded from our main domain, mail.yandex.ru, and bootstrap, from the static cluster mailstatic.yandex.net.
It includes about a hundred machines located in several data centers in Moscow and the Moscow region. The only task of this cluster is to give the client the static files that are needed for the display and operation of the Mail web interface.
To see the problem of speed in full growth, you need to think about the geography of Yandex.Mail users. Even if we talk only about Russia, the largest country in the world. For example, if you focus on the length of the Trans-Siberian Railway, the distance between Moscow and Vladivostok is almost 9300 km. Packages with user data, of course, do not move by rail - they travel by lines of communication. And the data transfer rate in optical fiber is about one and a half times lower than the speed of light in a vacuum. Network equipment enters the way along the data path as well. Thus, RTT from our data centers in Moscow to Vladivostok is 120-130ms, and, for example, to Novosibirsk - about 60ms.
Local servers
To solve speed problems, Yandex has CDN hosting - yandex.st. Yandex.st is several clusters of machines, most of which, like mailstatic.yandex.net, are located in Moscow data centers. But, unlike mailstatic.yandex.net, it has representative offices in many large cities of Russia, the near abroad and the world.
At the moment, our CDN-hosting supports more than three hundred providers in 11 regions. By the way, yandex.st is the third most popular hosting jquery library in the world .
It works on anycast technology. This means that the user on the file request is answered topologically by the server closest to it. That is, if a person is in Vladivostok, his request will be sent to a server located in Vladivostok, which means he will receive an answer faster than if his request went to Moscow.
Moving Yandex.Mail to yandex.st
Before you move, you had to evaluate how effective the move will be and what benefits it will bring. To do this, we hung the web interface of the Mail with sensors, built graphics and transported only 50% of users to yandex.st. This was done in order to be able to compare everything with the standard - mailstatic.
We have built in the client part of the web interface a special timer for loading the statics. The Navigation Timing API, which is available in most modern browsers, helped us with this. The API, up to milliseconds, knows at what point the user starts opening mail, and the javascript platform knows when all the necessary static files are loaded. If we subtract the first from the second timestamp, we can get the time for which all the static was loaded. The received number along with other metrics is sent by a GET request from the user to special servers.
Unlike the nginx access logs timings (for which, of course, we also have graphics), this metric allows you to take into account the download time of all mail, regardless of what files and how much each user loads. Roughly speaking, we can know for sure how long the user has loaded all mail, and not every file separately.
In less than one percent of cases, due to unclear reasons, these figures arrive frankly implausible. For example, negative or equal to 50 years.
In order to cut off such values, we decided to build a timeline in which 75%, 95% and 99% of downloads fit.
But these figures are not enough - the download speed varies, including due to changes in file sizes and number. In order to understand how in other regions the static of mail began to load faster, we chose to compare the download speed in each region with the download speed in Moscow as the KPI relocation. That is, we have started to take away from the loading time from the region the loading time from Moscow. If the value turned out to be negative, we considered that we were doing everything correctly and the speed in this region became faster.
This chart helped find a few bottlenecks and problems on regional machines, which we quickly fixed.
Result
As a result of the move, Yandex.Mail started responding and loading faster: depending on the region, from several hundred milliseconds to seconds. Static loading speed in Novosibirsk and Kazan doubled, in Yekaterinburg and in Vladivostok - about one and a half.
Graph of average statics loading time in some cities
Thanks to the setting up of regional servers, we found several opportunities to increase the speed in the “home” region: Yandex.Mail has become faster not only in the regions, but also in Moscow, so we invite everyone to start using yandex.st.
Site owners
If your project uses jquery or other popular javascript frameworks and libraries, you can connect them from our servers .
In addition to regionality, yandex.st has round-the-clock support by Yandex administrators. It supports the ipv6 protocol and has a minimum downtime: the more people use yandex.st, the more chances that jquery will not be “flushed” from the cache. And, of course, Yandex services already take libraries from yandex.st, so that regular users often have these versions in the cache. And this means that if you use yandex.st on your site, it will load faster.
Representatives of regional providers
Not all regional providers have configured routing so that their users fall on local machines. If you represent any provider and want to speed up Yandex for your users (and reduce traffic in his direction), you must ensure the topology of our CDN servers that support yandex.st (and other services). There are two ways to do this:
- Connect to traffic exchange points where Yandex is present and CDN servers are installed (see, for example, at www.peeringdb.com/private/participant_view.php?id=1751 ), and pick up with us there peering, after which CDN will automatically work ;
- Write to noc@yandex.net if the traffic between the provider and Yandex is significant (comparable or more than 500Mb / s). We are ready to consider the possibility of installing CDN servers in the provider's network.
Source: https://habr.com/ru/post/176893/
All Articles