Security Week 20: Disable Firefox Extensions

On May 4, at four in the morning Moscow time (or a little later, depending on luck), all installed extensions stopped working for users of the Firefox browser, and installing new add-ons became impossible. The problem was on the browser side - the intermediate certificate with which all extensions are signed has expired. This event is indirectly related to information security - bad luck arose due to the quite logical desire of developers to protect users from malicious extensions (starting in 2015) and due to the fact that no one noticed that the certificate will expire soon.

Nevertheless, this is an interesting story with a moderately happy ending: a rather non-trivial problem was solved within 12 hours. This is also a detailed documented incident, with a wealth of information from both the developers and the incredible amounts of drama from the users. In the process, the problem of privacy arose, which was quite effectively solved.

The first reports of problems with extensions appeared before the certificate expired on May 3. Reddit has at least one such discussion . The author of the thread on the computer was set the wrong date, so he found out about the bug one of the first (and “fixed” it by setting the correct date, but not for long). A little later, the problem appeared for all users, but at different times: the validity of certificates signed by extensions is checked once every 24 hours. Further story with different levels of detail is described here: news , bug-report , tech support , blog post with technical details, and a story about the incident on behalf of CTO Firefox.


This is what the process of signing extensions in Firefox looks like. The root certificate at the top of the chain is in an offline repository, and once every few years it creates an intermediate certificate with which extensions are already signed. It is the period of validity of the intermediate certificate expired on May 4th. The first interim solution from the Firefox developers was the release of the patch, which temporarily stops checking the validity of the extension certificates. If such a patch came to you before checking, then the problem has passed you. Further, the developers had two ways: to release either a new release of Firefox, or a new certificate that will make valid the signatures of the extensions in the current version. It was impossible to re-sign all additions (more than 15 thousand). More precisely, perhaps, but it would take a very long time.
')

Given the tight deadlines (we learned about the problem in Firefox on the evening of May 3, of course, on Friday!), It was decided to investigate both options. Technically, the ability to issue a new certificate was required to a) generate this same certificate and b) deliver it to users as soon as possible. The first part was complicated by the fact that the root certificate is stored in a hardware module, to which you still need to get there ( to the forest? To the bank box? ). In addition, when the developers got to the root certificate, generate a new intermediate did not work out immediately, and this each time led to the loss of an hour or two on the necessary tests. How to deliver? For this purpose, the Firefox Studies mechanism was used - in fact, the add-on distribution system “from the browser developer”, in normal cases designed for experimental code. So it came out faster than building a new build and sending an update through normal channels.


But then there was the problem of privacy. Firefox Studies as an experimental system is only included with sending browser usage information back to the Firefox developers. This is logical for beta testing, but it looks a bit strange in the context of patch delivery, which is generally needed by all users. This problem was solved elegantly: Firefox decided to delete all telemetry received from May 4 to 11.

It was not a perfect solution. Those with telemetry disabled (and Studies) had to manually enable these options. In some builds of this option and not at all. The Android browser version does not support Studies. Users of old versions of Firefox who do not want to be updated, but use addons, have suffered permanently. On May 8, versions of Firefox 66.0.5 and Firefox ESR 60.6.3 were released, in which the problem with the certificate was finally solved, and Studies with telemetry are no longer necessary to include. Updates are planned for older versions of the browser, starting with Firefox 52. The problem was solved, but for many users it did not go unnoticed - there are cases of data and settings loss in extensions.

Two conclusions were drawn from this story. Firstly, Firefox promised to set up a “time bombs” tracking system in the infrastructure to prevent this from happening in the future. Secondly, it became clear that even a more or less modern system of sending updates, with different ways of delivering patches to users, is not as effective as we would like. According to CTO Firefox, browser users should be able to receive updates and hotfixes, even if they wish to disable any other experimental features and / or telemetry. This is a happy ending story that may lead to an improvement in the update mechanism of Firefox, a rare mainstream browser not tied to any large IT company. But do not forget that it began with an allegorical shooting at her own feet.

Disclaimer: The opinions expressed in this digest may not always coincide with the official position of Kaspersky Lab. Dear editors generally recommend to treat any opinions with healthy skepticism.