How we implemented HTTPS on the main page of the Mail.Ru portal

Paradoxically: HTTPS has existed for many years, but over all these years it has not become the default standard for all Internet resources that work with security-critical data. Last year, we were the first among the major Russian portals to include HTTPS on the main page.

Now encryption works from the first steps of the user on the Mail.Ru portal, and is always on and on by default. In our opinion, this is just a must-have, because on the main page there is a form for entering a login and password from the mail. Although the transfer of user data during authorization for many years occurs via HTTPS, but if the main page is loaded via HTTP, then at the data entry stage there is an opportunity to intercept them during the SSLstrip attack, which is quite popular among cybercriminals. Let me remind you of its essence, taking for example the non-existent site example.com:

• Suppose a user enters the Internet through a public Wi-Fi and writes example.com in the browser;
• The request goes to the example.com servers where the response is generated. It contains including a link to the URL where authorization occurs (https://auth.example.com);
• The attacker intercepts the response and changes this URL to any other (https://some-fraudlent-server.com). Without HTTPS, a hacker can not only “listen” to traffic, but also send fake data to the client on behalf of the server;
• The attacker sends a fake page to the user's browser that displays it. It looks exactly like a regular page;
• The unsuspecting user enters his username and password there, then clicks "Login";
• The browser translates the user to the action of the authorization form. If it were not for the actions of the attacker, he would have transferred it to https: //auth.example.com, but due to the fact that the URL was changed, the browser will take it to its https://some-fraudlent-server.com page, sending there username and password.

')
Now, even if the user turns out to be very inventive (for example, he has bookmarked the http-address of the page - https://mail.ru/ and will try to enter the main page via this link), and the attacker is very persistent - even now in this case, user data will be protected by Strict Transport Security technology. This is a mechanism for activating a forced secure connection via HTTPS. It is supported by most modern browsers. That is why it was so important for us to implement HTTPS on the main page.

Thus, the transition to HTTPS protects, firstly, from listening to the traffic and, secondly, from the fact that the page code can modify the MiTM. They can be either an attacker or, for example, a public Wi-Fi provider who wants to show their own banner ads to the user. Such banners may overlap some useful content, for example, navigation elements — of course, users do not always like it. By the way, theoretically, a provider can equally well implement malicious content, intercepting user data.

And now about how we implemented HTTPS on the main page of one of the largest and most visited runet portals and what difficulties we had to face.

Content Transfer to HTTPS

First of all, it was necessary to transfer to https all the content displayed on the main page. We receive most of the information from other Mail.Ru services (News, Weather, and so on). Some of them have not yet switched to HTTPS. But the content, namely pictures taken from other services, are physically transferred to our servers, therefore there are no problems with getting them via HTTPS or with the possible increase in the load on these services.

It was more difficult with the general banner system that runs on the Mail.Ru portal, since it also has external content from partners. These are pixels (1x1 pixel transparent images) that jerk to calculate statistics when a partner banner is displayed. Even when we implemented HTTPS at the Post Office, we did two things: first, we agreed with the partners to switch to HTTPS and, second, we made it so that the display of banners with unsafe content was technically prohibited on the side of our banner system . Translating to HTTPS home page, we used these solutions.

Another example of affiliate content is classroom counters. If you will be switching to HTTPS, we recommend that you check all such content for compatibility with the secure protocol.

Before enabling HTTPS on live users, we forcibly transferred Mail.Ru Group employees to it. This could not give any noticeable load, but allowed once again to make sure that all the content of the page is loaded using a secure protocol.

Enable HTTPS on live

The main problem we were preparing for: switching to HTTPS significantly increases server load. In addition, it increases the total page load time by increasing the time to connect to the server and the time to download scripts and statics. Thus, the main tasks that we solved were, firstly, to try to run on the current amount of hardware and, secondly, to minimize the page load speed.

We decided to do a battle check and turned on HTTPS for some users, measuring how much the load on the hardware would grow. Over the course of several days, we gradually increased the proportion of HTTPS and monitored the graphs — CPU on the server, page load time per block and classroom indicators (hits and transitions to projects from the main one). Here it should be noted that for the main page we consider it necessary to keep multiple stock on load. However, a test run showed that this margin would not be present when HTTPS was launched to 100% of users. And then it is time to optimize.

First, we enabled keep-alive, thanks to which it does not take time to establish a reconnect to the server. Since the main Mail.Ru page contacts the server for updates on the current user once a minute, we use the established connection to update it. Thanks to this, keep-alive allowed to reduce the number of established connections. As a result, the load has decreased and the average time to wait for a response from the server has decreased by about 30%.

Secondly, we have reduced the number of requests for updating user data by a third. The main page makes about 900 thousand such requests per minute. However, if the user was and remains unauthorized, then absolutely nothing new we will not know how much the server is trying (this is akin to looking into the refrigerator every couple of minutes, checking if there is something new there). As a result, we began to apply for updates of user data only when we assume that this data could be updated - when the user is authorized or the authorization has changed. These changes made it possible to reduce the load to an acceptable level and enable HTTPS on all users, without having to add additional servers.

With the inclusion of HTTPS "at all", we watched the changes in the classroom indicators, because it could indicate some problems with loading the page in non-standard browsers or OS, if users had to face them.

There was an interesting point: we noticed an increase in the number of hits on the mobile version of the main page immediately after turning on HTTPS. This increase, of course, raised concerns. It turned out that with HTTPS a full page reload occurs if the user returns to it by pressing back in the browser. And users return to the main page in this way often - links from the mobile version of the main version open in the same window, unlike the large version. With HTTP, the page in mobile browsers is often shown from the cache (the behavior depends on the OS and the browser). Therefore, the number of hits counted by the main and increased. To test this hypothesis, we plotted page reloads on the back button (for some browsers), and he confirmed it.

So, we have successfully translated the main page of our portal. In addition, we implemented HTTPS on our content projects, where not so much user data is stored compared to, for example, Mail, but there is a form of authorization. Now secure protocol is enabled on News, Auto, Cars, Hi-Tech, Weather, Health and Children Mail.Ru, and in the future this list will continue to expand (in one of the following posts we will tell how their transition to HTTPS affects traffic from search engines ).

We hope that our example will inspire colleagues, and all main pages of more or less large resources will gradually turn out for HTTPS. Protect user data from SSLstrip attacks!