In early October, GlobalSign , an authoritative center for issuing TLS certificates (CA), began restructuring its infrastructure. Among other measures, GlobalSign removed a number of cross-signatures of its root TLS certificates.

Unfortunately, in the process, Safari, Chrome, and IE11 browsers began to see GlobalSign certificates as revoked for security reasons. GlobalSign engineers promptly eliminated a critical error, but an incorrect OCSP response was cached on the CDN and spread widely. At the moment - and the next four days, before the expiration of the entry in the OCSP-browser cache - sites that are protected by a certificate from GlobalSign, may be unavailable for a significant part of users.
')
Among the affected sites are companies such as Wikipedia , Dropbox , Financial Times and others.

Bloody technical details

What is OCSP?

SSL and TLS are used to encrypt HTTP traffic on the Internet. These protocols introduce the concept of “certificate authority” (CA), or certification authority. Every operating system and every browser has a built-in cache of certificate authorities that they trust. Any HTTPS site must have a certificate issued by a trusted certificate authority, otherwise the connection will not be established and the browser will display an error.



There is one slippery moment in this concept. The fact is that if a vulnerability is detected on the server, for example, an attacker can gain access to an existing certificate and private encryption key. Having stolen the key, an attacker can use it to imitate the original site and organize a Man-in-the-Middle attack on this site. As a result, a thief could potentially gain access to passwords, plastic card data and other confidential information from site visitors.



The problem is aggravated by the fact that TLS-certificates are issued, though not permanently, but for a sufficiently long period of time (usually from a year or more), and many certification centers (including, by the way, GlobalSign) give a discount to those who buy their certificate with long validity period. This is one of the problems that the free certification center Let's Encrypt is trying to solve. Certificates issued by Let's Encrypt are valid for no more than 3 months, and the certification authority plans to reduce this period to 30 days.

Correct the current state of affairs called for mechanisms called CRL and OCSP . They allow the browser to verify whether the TLS certificate presented by the site is valid. If at some point the certificate holder suspects that the private key of the certificate fell into the wrong hands, he can contact the center that issued the certificate and revoke it. The revoked certificate will not be accepted by most modern browsers, in particular, Safari and Chrome, and in the future - by all browsers in general. Thus, confidential information will not fall into the wrong hands.

What happened in early October

GlobalSign manages multiple root trust certificates. A number of these certificates cross-sign each other, in the same way that Let's Encrypt does:



Cross-signature is needed, for example, when issuing a new root certificate. Since many people rarely update their operating systems, the newly created certificate may not be immediately cached by older browsers. So that the root certificate can be used for its intended purpose, it is signed with one of the old root certificates.

Of course, cross-signing complicates root certificate maintenance. In addition, enough time has passed since the release of some of the GlobalSign certificates so that the remaining non-upgraded systems can be neglected. In the end, these systems are still doomed - for example, the infamous Internet Explorer 6 supports out-of-the-box cryptographic protocols that are no higher than the vulnerable SSL 3.0 .

Therefore, in October 2016, GlobalSign decided to remove some of the cross-signatures between its certificates and manage them further separately and independently.

Something went wrong

On the morning of October 14, an error crept into the process of recalling cross-signatures. As a result, a number of intermediate certificates GlobalSign (in particular, inexpensive and common AlphaSSL ) began to be perceived by Safari and Chrome as revoked, and all sites that bought a certificate from AlphaSSL and others stopped to open.

GlobalSign engineers promptly fixed the problem, but the troubles did not end there . The fact is that the OCSP server is an extremely high-loaded element of the CA infrastructure; all browsers of all users of all CA clients (except for those clients who have configured OCSP stapling ) go to it . Therefore, most certificate authorities use CDN to distribute OCSP responses. In particular, GlobalSign uses Cloudflare services. There are no details yet, but, apparently, Cloudflare for some reason was unable to promptly clear its cache, and the incorrect OCSP status continued to spread on the Internet.


At the moment, the problem with the CDN cache is also solved, however, for many users, the wrong OCSP status is cached now in the browser. An entry in the OCSP cache of browsers and operating systems will be valid for the next 4 days, then the situation will be corrected.

There are also indirect reasons to believe that the problem was caused by a bug in the processing of OCSP records in Safari and Chrome browsers. However, there is no actual evidence of this yet . Update: The Register on Twitter reports : GlobalSign officials confirmed in an interview with BBC Radio that the problem was entirely on their side.

11 hours after the start of the incident, GlobalSign issued a recommendation to correct the problems, however, a number of clients during this time have already migrated to other CAs:


The most annoying in this situation: the error GlobalSign primarily affected Internet services with high attendance. In fact, the affected sites will be unavailable only for those users who came to them during those few hours while Cloudflare issued an incorrect OCSP response. The more popular the site, the greater the share of such users on it. Sites with low traffic and sites without regular visitors almost did not suffer from this incident.

If you yourself have problems accessing HTTPS sites and the browser reports that the site certificate has been revoked, try clearing the local CRL and OCSP caches. The corresponding instructions can be found on the GlobalSign website .

The most important thing

This incident is the first of its kind. For the first time in the history of the Internet, a problem of this level occurred, and there is no doubt that all players in the CA industry will do everything possible so that this does not happen again.

The global TLS and CA infrastructure is complex and voluminous, but any system is characterized not by an error, but by a reaction to an error. What happened should not be considered as a reason to stop the widespread introduction of secure protocols and ciphers. By including HTTPS, HSTS, HPKP on your website, you protect your users and make the Internet safer and, moreover, more reliable. And this is the direction in which the Internet should move.

GlobalSign is guilty in front of its customers, but this company is known for its ability to admit mistakes and draw conclusions from them . In the near future, we expect from the company a detailed analysis of all the circumstances and open work on the errors and we will keep you informed.