Why is HTTPS not commonly used yet?
Encryption. We all love him and want to use it everywhere. But why is it still not universally applied?
The first and most common barrier to the transition to HTTPS is the price of obtaining, configuring and maintaining a valid certificate. You must find a supplier of certificates, confirm your identity, pay for it and set up a server, as well as renew it in a timely manner.
Most of the proposals for the transition to ubiquitous encryption sound like this: "The NSA records all our traffic, why not encrypt it?". The purpose of such proposals is to increase the cost of passive tracking of all traffic, rather than more complex and targeted attacks that are used by attackers.
')
The guys from Let's Encrypt have already guessed that the problem with certificates is almost completely automated, and that its implementation for release, installation, configuration and renewal on several of the most common platforms can cover the vast majority of the Internet. A wonderful job, and, although much remains to be done, I think we can consider the problem of certificates solved.
Well, maybe. But maybe not. If all the HTML resources you provide have links (images, scripts ...) to the same host and you use only relative URLs, then everything is fine. Otherwise, most likely HTTPS will not work correctly.
What? HTTPS - it's good, why did it break?
Everything broke because almost certainly you have mixed content on your page.
Consider the following case when you are the “OriginA” operator. Do green circles mean resources available via http and https? red - only http. Dotted lines are content uploading via http. The usual lines are https. Suppose that all links are absolute. A red cross indicates that the download will fail.
Now let's say you configured the certificate and enabled https on OriginA. What will the chart look like now?
If you do not update the absolute links in HTML, the browser will still try to get resources via http. Most prohibit such downloads and show warnings in such cases, and over time they become only more severe.
In most web security models, the sources are responsible for their own information. Content downloaded via HTTPS can redirect the user to an insecure site, send a POST request or postMessage () to an unsafe source, and the https site can receive GET, POST or onMessage () from documents downloaded via http. With all this, why is POST allowed, but XHR prohibited?
There is a formal security rule that browsers try to apply to sites. For the first time, this rule was formulated as “Tranquility”. In simple terms, this means that a secure document will not become unsafe during an interaction.
Of all the complications and traps of Web security, browsers have come to the conclusion that there is only one more or less reliable and useful security indicator: an address bar and a lock for using HTTPS. If you type "https: //" in the address bar or see the lock icon of the document with which you interact, the browser promises you that the content is protected from outside threats.
If the https site had uploaded a script or image via http, this promise would have been broken. The specific consequences of such an action can vary greatly, but the browser does not have the right to deal with it, so it simply stops such behavior. This was done not only to protect users who could not figure it out, for example by opening a browser client in the browser, requesting a script via http, in a coffee shop, but also as an indicator for content authors who might have missed something.
This is a good solution for users, but it gives site operators some difficulties when translating sites to https. All of their HTML resources that link to unsafe content break down or scare users with warnings. This problem with dependencies, I believe, is a real obstacle that needs to be overcome to convert 100% of resources to HTTPS.
If you need to fix a problem with mixed content, then the cost of switching to https is somewhat higher. It is a bit more complicated than server configuration or obtaining a certificate; deleting mixed content is expensive and not always automated.
For a complex site, you cannot simply run s / http / https / g on all pages, or write a rule in mod_rewrite. You may encounter http resources in many places: static content, dynamic, client, stored in the database, receiving data from third-party resources, and so on.
New specifications in development are aimed at facilitating this process; they help to automatically replace http resources with https.
Before:
After:
Upgrade-Insecure-Resources helped with the AccessA Access in this example. One of the HTML resources no longer contains mixed content because the protocols were quietly replaced and remote dependencies became available via https. This example shows why many sites reluctantly provide their own content via https, in order to avoid frightening error messages and defects in functionality.
This leads to the sad circumstance that the least responsible participants at the end of the chain of dependencies can hinder progress. Cyclic dependencies (since they definitely exist in the huge structure of the web) can create interlocks that cannot be resolved without coordination.
There is a lack of an intermediate state between http and https. It would be ideal if this condition had the following properties:
1. Allow safe sources that depend on your resources to receive them without violating the principle of tranquility.
2. It would not force other resources to make hasty conclusions about security or the absence of mixed content.
3. It would be extremely cheap and not creating any risks for implementation. Ideally, add an http header.
4. It would allow to determine dependencies that violate the principle of tranquility or create errors of mixed content.
OriginB goes into "https-transitional" mode. This means that the resource is still not available at https: //, but will be available in the future through TLS with full guarantees, including a valid certificate. This should not cost anything to OriginB, since the browser and users do not know anything about the security status of this resource.
Now OriginB resources are available through the "https-transitional". OriginA includes HTTPS. The browser knows that it can initiate a TLS connection to the OriginB and request resources via the traditional http protocol. If this protocol change fails, the resource will be marked as unsafe and a warning about mixed content will appear. If everything ends successfully, all guarantees about the safety of OriginA declared by the browser will be true, warnings about mixed content will not appear, despite the fact that the file was transmitted via http.
After updating, OriginA may also update Origin. OriginB is still dependent on OriginD, so it cannot yet go to https, but including the “transit” mode, it allows resources A and B to access resources via https.
To resolve circular dependencies without creating documents with mixed content, you need to configure a filter that returns 404 to all requests, except those for which you can return Content-Type text / html. Pretty good solution, excluding some CORS-related cases. Of the four properties we need, this approach implements three.
The fourth property is the ability to determine the states of your dependencies in order to know the moment when you can enable full HTTPS. What if browsers could use Content-Security-Policy-Report-Only with “upgrade-insecure-requests” for http documents?
It is possible that you do not need to do anything other than the above steps. This configuration, with upgrade-insecure-requests, implemented for A, B, and C:
Unfortunately, simply switching to html html resources via https is not enough, it does not work with HTML dependencies to HTML via iframe. It is quite often used.
It is necessary to implement the ability to download the HTML resource by replacing the protocol with TLS, but only on a secure page. This does not violate security because without the substitution of the protocol the content would already be unsafe, so we do not create mixed content for users of OriginB.
ALPN allows the client to communicate with the server via TLS by specifying another protocol that the client would like to use.
Let's introduce a new type of ALPN protocol: “https-transitional”. The server from which the client requests data in this way understands it as: “connect me to the server via http, but via TLS, not via https”. As described in the HTTP Alt-Svc draft, a TLS connection should occur, the server should present a certificate suitable for the domain.
The resource opened through the “https-transitional” will have the following properties:
Upgrade-Insecure-Requests will be modified as follows:
The blocking of mixed content must also be modified. Browsers are not currently trying to automatically replace http with https, since there is no guarantee that both schemes provide the same content. The https-transitional scheme provides such a guarantee. A servet can also report on the availability of a transit mode using the HTTP Alt-Svc header.
After downloading the document in transit mode, the browser should try to replace all connections blocked as mixed content, as if upgrade-insecure-requests were enabled, but also should seamlessly download content via https if the previous attempt fails. It should also output an error to the console.
The diagram above demonstrates the implementation of the above rules. Resources downloaded from OriginA, whose iframe points to OriginB will never contain mixed content, because OriginB supports transit mode. However, if OriginB resources are loaded from a document that blocks mixed content and depends on resources for which protocol replacement is not possible (for example, a JS file with OriginD), such requests will be silently blocked. Partial failure is preferable to complete. Resources downloaded directly from OriginB, which depend on the same on OriginD, will not be blocked, since the tranquility principle is not required.
For https resources, this offer should not create any new delays compared to using Upgrade-Insecure-Requests. Browsers will prefer TLS, so no new requests will be required to determine support for https-transitional.
The only place that can lose in performance is to replace the connection due to the Alt-Svc header. The browser will attempt to recursively update all dependent resources, some of which may be unavailable via https, which can create long delays. They can also fix this by remembering that TLS was unavailable for a certain source for a certain amount of time, or else parallelize requests. Perhaps some experiments are needed to determine the best strategy.
At some point, all sites will want to leave http. Downloading via https never rolls back to http, but the above script is not enough to protect against hackers who use attacks to remove encryption. How can we go from a transit web to a completely secure web, especially if the existence of the first means avoiding the need to change the query scheme for each link on the site?
We can begin to develop patterns based on HTTP Strict Transport Security, which was created to solve the problems of sites that want to completely abandon the http and Alt-Svc header.
The first thing an administrator can do is to install Alt-Svc in infinite, which will be a signal for browsers not to try to connect via http, use only https-transitional and https, always. After that, the site can leave http support for legacy clients, or turn it off completely.
The next step is to apply the principle tranquility for transit content. Perhaps the best way to do this is the HTTP header. Setting the tranquil bit will mean banning mixed content. This may also allow this resource to get the lock icon, or to cause similar changes in the interface.
One day, if transit mode propagation is sufficient, browsers will be able to start dropping http. Perhaps the beginning will be the appearance of the item in the settings, then showing a big warning. Many sites are likely to remain in transit for an unlimited time, but users will still benefit from using TLS.
Problem in certificates?
The first and most common barrier to the transition to HTTPS is the price of obtaining, configuring and maintaining a valid certificate. You must find a supplier of certificates, confirm your identity, pay for it and set up a server, as well as renew it in a timely manner.
Most of the proposals for the transition to ubiquitous encryption sound like this: "The NSA records all our traffic, why not encrypt it?". The purpose of such proposals is to increase the cost of passive tracking of all traffic, rather than more complex and targeted attacks that are used by attackers.
')
The guys from Let's Encrypt have already guessed that the problem with certificates is almost completely automated, and that its implementation for release, installation, configuration and renewal on several of the most common platforms can cover the vast majority of the Internet. A wonderful job, and, although much remains to be done, I think we can consider the problem of certificates solved.
Now we have a certificate, we can enable HTTPS, right?
Well, maybe. But maybe not. If all the HTML resources you provide have links (images, scripts ...) to the same host and you use only relative URLs, then everything is fine. Otherwise, most likely HTTPS will not work correctly.
What? HTTPS - it's good, why did it break?
Everything broke because almost certainly you have mixed content on your page.
What is mixed content and why should I care?
Consider the following case when you are the “OriginA” operator. Do green circles mean resources available via http and https? red - only http. Dotted lines are content uploading via http. The usual lines are https. Suppose that all links are absolute. A red cross indicates that the download will fail.
Now let's say you configured the certificate and enabled https on OriginA. What will the chart look like now?
If you do not update the absolute links in HTML, the browser will still try to get resources via http. Most prohibit such downloads and show warnings in such cases, and over time they become only more severe.
Why do browsers block mixed content? Why can't I influence this as a site owner?
In most web security models, the sources are responsible for their own information. Content downloaded via HTTPS can redirect the user to an insecure site, send a POST request or postMessage () to an unsafe source, and the https site can receive GET, POST or onMessage () from documents downloaded via http. With all this, why is POST allowed, but XHR prohibited?
There is a formal security rule that browsers try to apply to sites. For the first time, this rule was formulated as “Tranquility”. In simple terms, this means that a secure document will not become unsafe during an interaction.
Of all the complications and traps of Web security, browsers have come to the conclusion that there is only one more or less reliable and useful security indicator: an address bar and a lock for using HTTPS. If you type "https: //" in the address bar or see the lock icon of the document with which you interact, the browser promises you that the content is protected from outside threats.
If the https site had uploaded a script or image via http, this promise would have been broken. The specific consequences of such an action can vary greatly, but the browser does not have the right to deal with it, so it simply stops such behavior. This was done not only to protect users who could not figure it out, for example by opening a browser client in the browser, requesting a script via http, in a coffee shop, but also as an indicator for content authors who might have missed something.
This is a good solution for users, but it gives site operators some difficulties when translating sites to https. All of their HTML resources that link to unsafe content break down or scare users with warnings. This problem with dependencies, I believe, is a real obstacle that needs to be overcome to convert 100% of resources to HTTPS.
We mix mixed content
If you need to fix a problem with mixed content, then the cost of switching to https is somewhat higher. It is a bit more complicated than server configuration or obtaining a certificate; deleting mixed content is expensive and not always automated.
For a complex site, you cannot simply run s / http / https / g on all pages, or write a rule in mod_rewrite. You may encounter http resources in many places: static content, dynamic, client, stored in the database, receiving data from third-party resources, and so on.
New specifications in development are aimed at facilitating this process; they help to automatically replace http resources with https.
Before:
After:
Upgrade-Insecure-Resources helped with the AccessA Access in this example. One of the HTML resources no longer contains mixed content because the protocols were quietly replaced and remote dependencies became available via https. This example shows why many sites reluctantly provide their own content via https, in order to avoid frightening error messages and defects in functionality.
This leads to the sad circumstance that the least responsible participants at the end of the chain of dependencies can hinder progress. Cyclic dependencies (since they definitely exist in the huge structure of the web) can create interlocks that cannot be resolved without coordination.
None of these sites can enable https
Eliminate cyclic dependencies
There is a lack of an intermediate state between http and https. It would be ideal if this condition had the following properties:
1. Allow safe sources that depend on your resources to receive them without violating the principle of tranquility.
2. It would not force other resources to make hasty conclusions about security or the absence of mixed content.
3. It would be extremely cheap and not creating any risks for implementation. Ideally, add an http header.
4. It would allow to determine dependencies that violate the principle of tranquility or create errors of mixed content.
OriginB goes into "https-transitional" mode. This means that the resource is still not available at https: //, but will be available in the future through TLS with full guarantees, including a valid certificate. This should not cost anything to OriginB, since the browser and users do not know anything about the security status of this resource.
Now OriginB resources are available through the "https-transitional". OriginA includes HTTPS. The browser knows that it can initiate a TLS connection to the OriginB and request resources via the traditional http protocol. If this protocol change fails, the resource will be marked as unsafe and a warning about mixed content will appear. If everything ends successfully, all guarantees about the safety of OriginA declared by the browser will be true, warnings about mixed content will not appear, despite the fact that the file was transmitted via http.
After updating, OriginA may also update Origin. OriginB is still dependent on OriginD, so it cannot yet go to https, but including the “transit” mode, it allows resources A and B to access resources via https.
How can we create this state?
To resolve circular dependencies without creating documents with mixed content, you need to configure a filter that returns 404 to all requests, except those for which you can return Content-Type text / html. Pretty good solution, excluding some CORS-related cases. Of the four properties we need, this approach implements three.
The fourth property is the ability to determine the states of your dependencies in order to know the moment when you can enable full HTTPS. What if browsers could use Content-Security-Policy-Report-Only with “upgrade-insecure-requests” for http documents?
- We try to change the protocol
- In case of failure: Rollback on http, message to the administrator
It is possible that you do not need to do anything other than the above steps. This configuration, with upgrade-insecure-requests, implemented for A, B, and C:
iframe
Unfortunately, simply switching to html html resources via https is not enough, it does not work with HTML dependencies to HTML via iframe. It is quite often used.
It is necessary to implement the ability to download the HTML resource by replacing the protocol with TLS, but only on a secure page. This does not violate security because without the substitution of the protocol the content would already be unsafe, so we do not create mixed content for users of OriginB.
https-transitional with ALPN and HTTP Alt-Svc
ALPN allows the client to communicate with the server via TLS by specifying another protocol that the client would like to use.
Let's introduce a new type of ALPN protocol: “https-transitional”. The server from which the client requests data in this way understands it as: “connect me to the server via http, but via TLS, not via https”. As described in the HTTP Alt-Svc draft, a TLS connection should occur, the server should present a certificate suitable for the domain.
The resource opened through the “https-transitional” will have the following properties:
- The resource should not be locked as mixed content.
- Document settings transmitted via the “https-transitional” should not prohibit mixed content.
- The previous properties are executed until the parent frame of the document prohibits; in this case, the upgrade-insecure-requests should automatically occur
Upgrade-Insecure-Requests will be modified as follows:
- In the case of protocol substitution for a dependent resource, we connect to https, add a new “https-transitional” ALPN protocol as preferred, in addition to http / spdy / h2.
- If the server understands the "https-transitional", respond to this protocol and deliver the http resource via TLS.
- If the server does not understand “https-transitional” - reply by https.
The blocking of mixed content must also be modified. Browsers are not currently trying to automatically replace http with https, since there is no guarantee that both schemes provide the same content. The https-transitional scheme provides such a guarantee. A servet can also report on the availability of a transit mode using the HTTP Alt-Svc header.
After downloading the document in transit mode, the browser should try to replace all connections blocked as mixed content, as if upgrade-insecure-requests were enabled, but also should seamlessly download content via https if the previous attempt fails. It should also output an error to the console.
The diagram above demonstrates the implementation of the above rules. Resources downloaded from OriginA, whose iframe points to OriginB will never contain mixed content, because OriginB supports transit mode. However, if OriginB resources are loaded from a document that blocks mixed content and depends on resources for which protocol replacement is not possible (for example, a JS file with OriginD), such requests will be silently blocked. Partial failure is preferable to complete. Resources downloaded directly from OriginB, which depend on the same on OriginD, will not be blocked, since the tranquility principle is not required.
Performance impact
For https resources, this offer should not create any new delays compared to using Upgrade-Insecure-Requests. Browsers will prefer TLS, so no new requests will be required to determine support for https-transitional.
The only place that can lose in performance is to replace the connection due to the Alt-Svc header. The browser will attempt to recursively update all dependent resources, some of which may be unavailable via https, which can create long delays. They can also fix this by remembering that TLS was unavailable for a certain source for a certain amount of time, or else parallelize requests. Perhaps some experiments are needed to determine the best strategy.
At some point, all sites will want to leave http. Downloading via https never rolls back to http, but the above script is not enough to protect against hackers who use attacks to remove encryption. How can we go from a transit web to a completely secure web, especially if the existence of the first means avoiding the need to change the query scheme for each link on the site?
We can begin to develop patterns based on HTTP Strict Transport Security, which was created to solve the problems of sites that want to completely abandon the http and Alt-Svc header.
The first thing an administrator can do is to install Alt-Svc in infinite, which will be a signal for browsers not to try to connect via http, use only https-transitional and https, always. After that, the site can leave http support for legacy clients, or turn it off completely.
The next step is to apply the principle tranquility for transit content. Perhaps the best way to do this is the HTTP header. Setting the tranquil bit will mean banning mixed content. This may also allow this resource to get the lock icon, or to cause similar changes in the interface.
One day, if transit mode propagation is sufficient, browsers will be able to start dropping http. Perhaps the beginning will be the appearance of the item in the settings, then showing a big warning. Many sites are likely to remain in transit for an unlimited time, but users will still benefit from using TLS.
Source: https://habr.com/ru/post/275539/
All Articles