TLS SNI standard distribution

Experts of Akamai share their observations on the percentage of TLS SNI extensions of the TLS protocol (RFC 4366) in the total amount of encrypted HTTPS requests.

In the past few years, there has been a rapid growth in the number of clients supporting TLS SNI (a standard that makes HTTPS much more scalable). If at the beginning of 2014 only 85% of client HTTPS requests were made using TLS SNI, today many users of the Akamai service receive up to 99% of requests in the TLS SNI format.

This indicates a global trend of deployment of sites using only SNI. At the same time, 31% of sites from the Alexa Top 100k rating with valid HTTPS certificates communicate this certificate to the client only if the request is in the TLS SNI format.


')
When translating traffic to the HTTPS standard, there were two serious problems: the first was the inaccessibility of the certificates themselves, and the second was the technical difficulties with TLS virtual hosting. The readily available, self-issued Domain Validation (DV) certificates provided by services like Let's Encrypt solved the first problem. TLS Server Name Indication (SNI) is a technology that solves the second problem. Until recently, its widespread implementation was hampered by a large number of customers who did not support this standard.

The growing use of TLS SNI proves that it can already be considered a universal standard. The remaining part of clients, which does not support TLS SNI, can solve this problem for about a year, if it wants to have access to all sites at all. Especially with the growing use of HTTPS - about half of the pages are already loading over HTTPS, according to Firefox.

TLS is at the heart of HTTPS, and encryption and authentication is carried out with it. When Netscape developed SSL v3 and its fairly successful IETF-standardized descendant of TLS 0.1, they were faced with the fact that virtual hosting was not ready to use them. For authentication, the client sends the message “ClientHello” to the server and waits for the response “ServerHello”, containing a security certificate identifying the server name. If the name in the certificate does not match the name that the client expects to receive, a properly configured client should interrupt the connection and display an error message. But in the case of virtual hosting, one server serves tens, hundreds, thousands, and at least millions of sites, and in this case the server must know which certificate to return to the client.

If the ClientHello request does not contain the site name, the server serving HTTPS on port 443, there is nothing left but to allocate for each certificate (and, accordingly, the site) a separate IP. For CDN services, such as Akamai, deployed in hundreds and thousands of locations, this means that hundreds and thousands of virtual IPs must be allocated for each certificate. And due to the extremity of the pool of freely issued IPv4, it quickly turns into a fundamental problem that impedes growth and scaling.

Identification by Server Name (SNI) was introduced in TLS in 2003 in RFC 3546. This enabled clients to include the name of the server with which they would like to establish a connection in the ClientHello request, enabling the server to understand which named certificate should be communicated to the client in the ServerHello message. This SNI extension was the very route information that made virtual hosting possible, similar to the routing role of HTTP headers, which were introduced from the very beginning at the time of adopting HTTP / 1.0 in 1997. In addition, load balancers in data centers are now able to use TLS SNI to distribute traffic sent to the common IP address to the correct back-end servers.


IPv6 and HTTP / 2, of course, could help solve the problem. HTTP / 2 itself already requires clients to send TLS SNI. And the virtually infinite IPv6 address space solves the problem of virtual addresses, and for this reason, Akamai introduces dual default addressing for its IPv4 + IPv6 clients since 2016. But most of those clients that do not support TLS SNI do not support HTTP / 2 and IPv6 either.

Inadequate customer acceptance made it difficult to implement TLS SNI in practice. Previously widespread operating systems, such as Windows XP and Android 2.2, did not support TLS SNI, and the introduction of a strict requirement for an SNI request in the HTTPS protocol would leave them behind. In 2013, Akamai saw that only 80% of clients support this protocol. Even in April 2015, only 90% of HTTPS requests contained SNI. The decrease in the number of clients that do not support this standard is also caused by other reasons that change the overall situation with HTTPS protocols, for example, the discontinuation of SHA-1 certificates.

The situation has changed dramatically over the past two years. Today, Akamai, on average, 98% percent of requests contain client-side TLS SNI. However, most non-SNI-compliant clients are separate applications that have been developed privately and use old protocols on their individual routes. Therefore, the median participant of the HTTPS exchange, the equipment that gives the client a certificate, is already seen by 99% of requests including SNI. And 12% of such points already see 99.9% of the traffic with SNI authentication.

If we exclude from consideration China, where SNI is used to a much lesser extent, and the United States, where there are a number of bots / agents that do not support SNI, the median participant, based on a sample of Akamai service users, will see 99.4% of requests containing SNI. Even in the USA, the median participant sees 98.4% of SNI containing queries.

If we take a closer look at clients who do not use SNI, then, according to Akamai, these include:

• Current clients that use HTTPS traffic by intermediate proxies, for example, to exercise parental control or control corporate network traffic. These intermediate proxies degrade the reliability of TLS encryption, and the logic of their work, which forces them to substitute server names, is contrary to the SNI ideology. In addition, often these systems do not check certificates at all.
  
  
• The remaining systems are running Windows XP and even older versions of the OS. It is not quite clear how they even connect after the cancellation of SHA-1, but perhaps they are also behind intermediate proxies that do not ask for certificates.
  
  
• Some bots / robots owned by search engines. We contacted some of the top lines of the top lists, and they said that in the near future they would still introduce TLS SNI, too. It seems that there are many benign and malicious bots that continuously scan the entire IPv4 space in search of HTTPS servers.
  
  
• Various clients and bots, written on the basis of old versions of WebKit, Java, curl and python. Yes, and those customers who use more recent versions of these libraries should still have implemented mechanisms for proper initialization of the exchange based on TLS SNI. This includes a wide range of clients - from malware to various headless browsers to analyze the performance of sites.
  
  
• Some private applications (download managers, applications for game consoles, some mobile applications). They are usually isolated on certain sites.

Some sites wanted to switch to TLS SNI completely and were ready to lose from 10% to 20% of visitors. Now, the transition to TLS SNI “cuts off” only 1% of visitors, who are mostly bots, malware and insecure clients, so there will be much more people willing to switch to TLS SNI. Let's Encrypt issued about 27 million free certificates, prompting a significant number of site operators to switch to HTTPS. From the end of 2015, Akamai issues only TLS SNI-supporting certificates as part of its package offerings, prompting the sites to switch to this standard and, in fact, reserve for the future - forcing their potential users to also switch to TLS SNI.

The transition to using SNI exclusively is already underway. Today, approximately 75% of sites from Alexa top 1000 are accessible via the HTTPS protocol, 12% of them use only TLS SNI (that is, they report a certified certificate only in response to an SNI request). Among the Alexa top 100k sites, only 55% use HTTPS, but among them, 31% are exclusively in the SNI format, despite the fact that many of them are still available via HTTP.

The more sites will require TLS SNI, the more pressure customers will experience when they are forced to implement the SNI mechanism in their protocols. The decrease in the number of non-SNI clients encourages websites to switch to new certificates faster, and this, in turn, causes non-SNI clients to switch to TLS SNI, just as it was at one time, in fact, with HTTPS as such.

The trend is obvious: TLS SNI is becoming universal, generally accepted and, possibly, even the “default” standard in HTTPS. Client software developers should understand this and bear in mind the need for the correct implementation of this protocol in their packages.