Beginner's HTTP caching headers guide
The article contains information on caching headers (CK) for HTTP and the corresponding behavior of content delivery networks (CDN). If you want to figure out how caching headers fit into the modern web, or you just wonder what your colleagues are talking about - this article is for you.
If you already understand the benefits of HK, and want to expand your knowledge, I recommend that you refer to the documentation from W3.
Simply put, caching allows you to store web resources at remote locations along the path from your server to a custom browser. The browser also keeps a cache so that clients do not constantly request the same resources from you.
')
Web traffic caching settings are extremely important for the sites you visit. If you pay for traffic, generate e-commerce revenue, or just want to maintain your reputation as a good web developer, you need to understand how caching works.
In the case of resources such as your company's logo, website favicon, or basic CSS files that do not change from request to request, you can allow the requestor to keep copies of these files for a while. If your visitors were children in the backseat of a car who asked you all the time, “Have we already arrived?”, This permission would be akin to the answer “No, and we need to travel another 20 minutes, so remember my answer.”
By reducing the number of requests to the server, you increase the number of requests that it can handle. Pictures, scripts and style sheets can usually be cached in the tail and mane, and dynamically created pages (forums, web applications) are usually not worth it. If you’re primarily concerned about performance, all of your dynamic content should be minimized to AJAX resources, and the rest should be cached to the maximum.
Historically, the cache settings related only to the client browser, so do not forget about them. But today, due to the proliferation of content delivery networks, CDN, it is more important to understand how caching works at intermediate points on the web.
When properly configured, the CDN transfers your content to clients via the fastest and closest server to it. In addition, the CDN works as a buffer between you and the users. We are interested in the percentage of the number of cached requests - those requests that were processed by the CDN, not pulling our server. Depending on the traffic and architecture, this number can reach up to 90%, although you will notice an effect with smaller numbers. It should be noted that with a small number of requests, most of them will be sent to your server - so this percentage makes sense only along with the caching time and the total load on the site. But if you configure only one cache, and the caching headers do not work correctly, the totals may even become worse than they were.
Your servers provide content to intermediate servers that are present in different regions.
In addition to caching, the CDN has a pleasant side effect: if suddenly your servers crash, the CDN in some cases buffers the requests so that users may not notice.
The most important of all. Usually you specify its parameters in the string, something like:
cache-control: private, max-age = 0, no-cache
These settings are called cache response directives, and are as follows:
private | public
Indicates whether the content is intended for a specific user. If so, you do not need to cache it.
no-cache
By itself, the directive says that this request needs to be done anew each time. The etag header is usually used, which is described below. The fun begins when you give the field name after this directive. Then the caching servers understand that the answer can be cached, but it is necessary to delete the specified fields. This, for example, is useful for proper operation of cookies. However, some old programs do not know how to work with this trick.
no-store
Tells that this answer does not need to be stored. Surprisingly, but true. If the cache works according to the rules, it will make sure that no part of the request will be stored. This is necessary in order to protect all sensitive information.
max-age
Usually the lifetime of the resource is given in expires, but if you need to be more specific, you can set max-age in seconds. And this directive takes precedence over expires.
s-maxage
A bit like the previous one, but s here means shared cache, and is needed for a CDN. This directive takes precedence over max-age and expires when it comes to CDN servers.
must-revalidate
He says that every request needs to be done anew, and under no circumstances provide the user with cached content. It has an advantage over all other directives that allow caching. It is mainly used in some special protocols (for example, remittances).
no-transform
Some proxies can compress and convert content to speed up work. This directive prohibits such behavior.
proxy-revalidate
Approximately the same as must-revalidate, but for intermediate CDN servers. Why wasn’t she called s-mustrevalidate? Who knows. The point is to check if the content has not been updated, it is necessary for each new user only once.
Initially, it was the standard method for determining when a resource is aging. Today, max-age and s-maxage take precedence over it, but it is always useful to set this header for backward compatibility.
By setting the date to more than a year, you violate the header specification.
Abbreviation from entity tag. This is the unique identifier of the requested resource - usually, a certain hash of its content, or a hash of its update time. In general, the client's way to request from the CDN is “give me the X resource if he has an etag different from mine”.
Very powerful stuff. IE handled it incorrectly in the past, and even now it’s not quite correct. At some point, even Chrome was buggy with it. In essence, the header tells caching systems which of the headers can be used to determine if they have content in the cache. If we consider the cache as a data store of the key-value type, the use of vary adds these values ​​to the keys.
Often, you can come across an Accept-Encoding header that verifies that your gzip-compressed resources will be accepted by the client. It saves great traffic. In addition, the setting
vary: User-Agent
will make your website more SEO friendly if you distribute different HTML / CSS depending on the User-Agent. Google will notice this little thing and Googlebot will process your mobile content.
A rather old directive that can do a lot of things, which, however, is already being processed with the help of more modern ones. We are most interested in the form
pragma: no-cache
that in modern customers turns into
cache-control: no-cache
Not all CDNs or programs for customers work according to specifications. If you are a web developer, you know this problem. Therefore, before starting the service you should always test it to make sure that everything works as it should.
In addition, you may have noticed that some of the headers overlap or that their applications overlap. This is due to small differences between the different methods, or due to the transition of the web from HTTP / 1.0 to the HTTP / 1.1 protocol, in which the caching is described in more detail.
CDN providers that receive a request in gzip as an acceptable compression should also request compressed content from the server, or provide the client with a compressed version of the resource. Modern CDNs can compress resources on their own.
When testing the performance of the CDN servers, you need to take into account that some are configured to check for the presence of both compressed and uncompressed versions of resources. This check slightly increases the response time.
A CDN is a man-in-the-middle. You need to think about the organization of HTTPS traffic and how it goes to your server. Many CDNs will redirect the request to somesite.com/asset to somesite.com/asset , so if the logic of your server depends on the protocol, either change the logic or ask the CDN to redirect to HTTPS.
Usually in this case you need to set cache-control: no-cache so that the CDNs do not cache it. If you are interested in how to speed up a little and its work, continue reading.
Typical dynamic content
Most of the dynamic content is not so inconsistent. The list of active users, for example, has a lifetime of 10-20 seconds. Pages with schedules for certain can live some minutes. The news feed can be cached for some time, especially if there is an etag. If the load on your site is great, you should try to cache some of its resources for a while.
So what is the best time to store resources? It depends on the amount of traffic, size of resources, cache size.
In addition, you need to think about the main lack of caching - reducing control over resources. If you need to update the resource instantly, you will have problems if you asked him a time to live a year a while ago. Especially if you set this time not only for CDN (s-maxage), but also for users (max-age).
The longest interval for a cache is a year, or 31536000. But this is a bad idea. It's like a tattoo on the face. If your servers do not withstand even daily requests from the CDN about whether the resource has changed, it’s time to change the servers.
Headers for static content
An example of cache settings for static content with S3. The cache is recommended to keep the resource 900 seconds, user programs - 300 seconds. The last header indicates which CDN processed the request.
One exception to the commandment is “do not cache resources for a year,” or rather, one hack that allows you to bypass the problems of a long cache. You can rename a resource every time you release a new version of it. An increasing version number, time label, and hash can be added to its name.
The following table reflects the loss of time on the cache. Assuming that a resource receives 500 requests per minute, for different variants of the resource storage time, the following percentage of cache requests are obtained:
What percentage of requests to the cache will suit you? Usually from 60 seconds to 60 minutes is the normal lifetime for the resource. For pseudo-dynamic content, you can set the caching time in the interval up to 60 seconds.
Check that the headers go through the CDN the way you expect it. Typically, a CDN inserts some X-header where it specifies details on a given resource. Here are some tools that I found quite helpful.
The easiest method is to right-click on a page in Chrome, select Inspect Element, go to the Network tab and click on the HTML resource. If this is not selected by default, select the Headers tab to view all headers. Chrome also has the ability to set the user agent and not use the local cache.
Through such a proxy, you can redirect traffic and manipulate DNS queries, record headers, view performance statistics — all in a clear and simple GUI.
Request from the command line and view the response headers. Useful keys: -A view user anget, -b - cookies, -F - form data, -H - assign headers, -I - request headers only.
Roughly speaking, this is a cURL with a clear interface. You can set headers and view response headers and response body.
Requests is a Python library for web requests. With it, you can write automated tests for your site.
Now that you've read the entire guide, a few notes.
Most web servers, such as Apache and Nginx, do most of the work for you. You only have to work with the cache-control header. Browsers are usually set to strong content caching, so you often have to deal with excessive caching rather than the other way around. Typically, you specify a path like “/ static” that is cached for some reasonable time, for example, 300 seconds. Then you need to make sure that the root path "/" is given with the header “cache-control: no-cache”, and it is better to redirect the user for dynamic content directly to your servers, and leave for CDN only what is in “/ static”.
A CDN usually has the ability to break caching rules and cache as much as you need, regardless of headers. Also, they often freely manage with the protocols, so it is important to test what the CDN servers give and compare them with the headers of your servers.
For additional materials to speed up the site we recommend to refer to the Mobile Web Performance documentation.
If you already understand the benefits of HK, and want to expand your knowledge, I recommend that you refer to the documentation from W3.
What can ZK do for you?
Simply put, caching allows you to store web resources at remote locations along the path from your server to a custom browser. The browser also keeps a cache so that clients do not constantly request the same resources from you.
')
Web traffic caching settings are extremely important for the sites you visit. If you pay for traffic, generate e-commerce revenue, or just want to maintain your reputation as a good web developer, you need to understand how caching works.
In the case of resources such as your company's logo, website favicon, or basic CSS files that do not change from request to request, you can allow the requestor to keep copies of these files for a while. If your visitors were children in the backseat of a car who asked you all the time, “Have we already arrived?”, This permission would be akin to the answer “No, and we need to travel another 20 minutes, so remember my answer.”
By reducing the number of requests to the server, you increase the number of requests that it can handle. Pictures, scripts and style sheets can usually be cached in the tail and mane, and dynamically created pages (forums, web applications) are usually not worth it. If you’re primarily concerned about performance, all of your dynamic content should be minimized to AJAX resources, and the rest should be cached to the maximum.
For customers and CDN
Historically, the cache settings related only to the client browser, so do not forget about them. But today, due to the proliferation of content delivery networks, CDN, it is more important to understand how caching works at intermediate points on the web.
What is a CDN?
Not to read the entire Wikipedia article : these are servers (plural), sitting between your server and your client. Each of them caches your content according to the rules that you specify in the various HTTP headers.
When properly configured, the CDN transfers your content to clients via the fastest and closest server to it. In addition, the CDN works as a buffer between you and the users. We are interested in the percentage of the number of cached requests - those requests that were processed by the CDN, not pulling our server. Depending on the traffic and architecture, this number can reach up to 90%, although you will notice an effect with smaller numbers. It should be noted that with a small number of requests, most of them will be sent to your server - so this percentage makes sense only along with the caching time and the total load on the site. But if you configure only one cache, and the caching headers do not work correctly, the totals may even become worse than they were.
Your servers provide content to intermediate servers that are present in different regions.
In addition to caching, the CDN has a pleasant side effect: if suddenly your servers crash, the CDN in some cases buffers the requests so that users may not notice.
Main headlines
1. cache-control
The most important of all. Usually you specify its parameters in the string, something like:
cache-control: private, max-age = 0, no-cache
These settings are called cache response directives, and are as follows:
private | public
Indicates whether the content is intended for a specific user. If so, you do not need to cache it.
no-cache
By itself, the directive says that this request needs to be done anew each time. The etag header is usually used, which is described below. The fun begins when you give the field name after this directive. Then the caching servers understand that the answer can be cached, but it is necessary to delete the specified fields. This, for example, is useful for proper operation of cookies. However, some old programs do not know how to work with this trick.
no-store
Tells that this answer does not need to be stored. Surprisingly, but true. If the cache works according to the rules, it will make sure that no part of the request will be stored. This is necessary in order to protect all sensitive information.
max-age
Usually the lifetime of the resource is given in expires, but if you need to be more specific, you can set max-age in seconds. And this directive takes precedence over expires.
s-maxage
A bit like the previous one, but s here means shared cache, and is needed for a CDN. This directive takes precedence over max-age and expires when it comes to CDN servers.
must-revalidate
He says that every request needs to be done anew, and under no circumstances provide the user with cached content. It has an advantage over all other directives that allow caching. It is mainly used in some special protocols (for example, remittances).
no-transform
Some proxies can compress and convert content to speed up work. This directive prohibits such behavior.
proxy-revalidate
Approximately the same as must-revalidate, but for intermediate CDN servers. Why wasn’t she called s-mustrevalidate? Who knows. The point is to check if the content has not been updated, it is necessary for each new user only once.
2. expires
Initially, it was the standard method for determining when a resource is aging. Today, max-age and s-maxage take precedence over it, but it is always useful to set this header for backward compatibility.
By setting the date to more than a year, you violate the header specification.
3. etag
Abbreviation from entity tag. This is the unique identifier of the requested resource - usually, a certain hash of its content, or a hash of its update time. In general, the client's way to request from the CDN is “give me the X resource if he has an etag different from mine”.
4. vary
Very powerful stuff. IE handled it incorrectly in the past, and even now it’s not quite correct. At some point, even Chrome was buggy with it. In essence, the header tells caching systems which of the headers can be used to determine if they have content in the cache. If we consider the cache as a data store of the key-value type, the use of vary adds these values ​​to the keys.
Often, you can come across an Accept-Encoding header that verifies that your gzip-compressed resources will be accepted by the client. It saves great traffic. In addition, the setting
vary: User-Agent
will make your website more SEO friendly if you distribute different HTML / CSS depending on the User-Agent. Google will notice this little thing and Googlebot will process your mobile content.
5. pragma
A rather old directive that can do a lot of things, which, however, is already being processed with the help of more modern ones. We are most interested in the form
pragma: no-cache
that in modern customers turns into
cache-control: no-cache
Cautions
Not all CDNs or programs for customers work according to specifications. If you are a web developer, you know this problem. Therefore, before starting the service you should always test it to make sure that everything works as it should.
In addition, you may have noticed that some of the headers overlap or that their applications overlap. This is due to small differences between the different methods, or due to the transition of the web from HTTP / 1.0 to the HTTP / 1.1 protocol, in which the caching is described in more detail.
1. Compression
CDN providers that receive a request in gzip as an acceptable compression should also request compressed content from the server, or provide the client with a compressed version of the resource. Modern CDNs can compress resources on their own.
When testing the performance of the CDN servers, you need to take into account that some are configured to check for the presence of both compressed and uncompressed versions of resources. This check slightly increases the response time.
2. SSL
A CDN is a man-in-the-middle. You need to think about the organization of HTTPS traffic and how it goes to your server. Many CDNs will redirect the request to somesite.com/asset to somesite.com/asset , so if the logic of your server depends on the protocol, either change the logic or ask the CDN to redirect to HTTPS.
And what about dynamic content?
Usually in this case you need to set cache-control: no-cache so that the CDNs do not cache it. If you are interested in how to speed up a little and its work, continue reading.
Typical dynamic content
HTTP/1.1 200 OK Server: Apache X-Rack-Cache: miss ETag: "e6811cdbcedf972c5e8105a89f637d39-gzip" Status: 200 Content-Type: text/html; charset=utf-8 Expires: Mon, 29 Apr 2013 21:44:55 GMT Cache-Control: max-age=0, no-cache, no-store Pragma: no-cache Date: Mon, 29 Apr 2013 21:44:55 GMT Most of the dynamic content is not so inconsistent. The list of active users, for example, has a lifetime of 10-20 seconds. Pages with schedules for certain can live some minutes. The news feed can be cached for some time, especially if there is an etag. If the load on your site is great, you should try to cache some of its resources for a while.
Caching time analysis
So what is the best time to store resources? It depends on the amount of traffic, size of resources, cache size.
In addition, you need to think about the main lack of caching - reducing control over resources. If you need to update the resource instantly, you will have problems if you asked him a time to live a year a while ago. Especially if you set this time not only for CDN (s-maxage), but also for users (max-age).
The longest interval for a cache is a year, or 31536000. But this is a bad idea. It's like a tattoo on the face. If your servers do not withstand even daily requests from the CDN about whether the resource has changed, it’s time to change the servers.
Headers for static content
HTTP/1.1 200 OK Cache-Control: no-transform,public,max-age=300,s-maxage=900 Content-Type: text/html; charset=UTF-8 Date: Mon, 29 Apr 2013 16:38:15 GMT ETag: "bbea5db7e1785119a7f94fdd504c546e" Last-Modified: Sat, 27 Apr 2013 00:44:54 GMT Server: AmazonS3 Vary: Accept-Encoding X-Cache: HIT An example of cache settings for static content with S3. The cache is recommended to keep the resource 900 seconds, user programs - 300 seconds. The last header indicates which CDN processed the request.
One exception to the commandment is “do not cache resources for a year,” or rather, one hack that allows you to bypass the problems of a long cache. You can rename a resource every time you release a new version of it. An increasing version number, time label, and hash can be added to its name.
The following table reflects the loss of time on the cache. Assuming that a resource receives 500 requests per minute, for different variants of the resource storage time, the following percentage of cache requests are obtained:
| Caching time (min) | Cache query percentage | Requests to the original per hour |
| one | 99.8% | 60 |
| five | 99.96% | 12 |
| 20 | 99.99% | 3 |
| 60 | 99.997% | one |
| 86400 | 99.9998% | <1 |
What percentage of requests to the cache will suit you? Usually from 60 seconds to 60 minutes is the normal lifetime for the resource. For pseudo-dynamic content, you can set the caching time in the interval up to 60 seconds.
CDN check
Check that the headers go through the CDN the way you expect it. Typically, a CDN inserts some X-header where it specifies details on a given resource. Here are some tools that I found quite helpful.
1. Web Inspector
The easiest method is to right-click on a page in Chrome, select Inspect Element, go to the Network tab and click on the HTML resource. If this is not selected by default, select the Headers tab to view all headers. Chrome also has the ability to set the user agent and not use the local cache.
2. Charles Proxy
Through such a proxy, you can redirect traffic and manipulate DNS queries, record headers, view performance statistics — all in a clear and simple GUI.
3. cURL
Request from the command line and view the response headers. Useful keys: -A view user anget, -b - cookies, -F - form data, -H - assign headers, -I - request headers only.
4. hurl.it
Roughly speaking, this is a cURL with a clear interface. You can set headers and view response headers and response body.
5. Python and Requests
Requests is a Python library for web requests. With it, you can write automated tests for your site.
Margin notes
Now that you've read the entire guide, a few notes.
Most web servers, such as Apache and Nginx, do most of the work for you. You only have to work with the cache-control header. Browsers are usually set to strong content caching, so you often have to deal with excessive caching rather than the other way around. Typically, you specify a path like “/ static” that is cached for some reasonable time, for example, 300 seconds. Then you need to make sure that the root path "/" is given with the header “cache-control: no-cache”, and it is better to redirect the user for dynamic content directly to your servers, and leave for CDN only what is in “/ static”.
A CDN usually has the ability to break caching rules and cache as much as you need, regardless of headers. Also, they often freely manage with the protocols, so it is important to test what the CDN servers give and compare them with the headers of your servers.
For additional materials to speed up the site we recommend to refer to the Mobile Web Performance documentation.
Source: https://habr.com/ru/post/253121/
All Articles