Frontend performance. Part 2 - we cache dynamic content with etagify

From the translator: This is the sixth article from the Node.js series from the Mozilla Identity team, which deals with the Persona project.

---

You may know that Connect puts ETag on static content, but not dynamic. Unfortunately, if you dynamically generate localized versions of static pages, they are not cached at all, unless you decide to generate them all in advance, at the assembly stage of the project. This can be avoided.
')
This article is devoted to etagify , a middleware module for Connect, which generates ETags on the fly based on MD5 hashes of answers and stores these hashes in memory. Etagify eliminates unnecessary routine when building a project, is extremely simple to use and increases productivity more than you would expect (in our tests we received page loading acceleration by 9%):

myapp = require('express').createServer(); myapp.use(require('etagify')()); ... app.get('/about', function(req, res) { res.etagify(); var body = ejs.render(template, options); res.send(body); }); 

Let's take a closer look at how etagify works, when it is worth it and when it should not be used, and how to measure the result of its use. In case you don’t have a very good idea of ​​what Etag is, and how to work with them, I wrote a small cheat sheet .

How etagify works

Due to the fact that etagify is aimed at one specific use case, this tiny library has only a hundred lines along with documentation. Let's take a look at the key fifteen lines, leaving behind the processing of boundary cases with Vary headers.

Etagify does two things - it considers the hashes for the resources given to the client and stores them in memory in order to check the version of the resource when processing conditional GET requests.

Here is how an array of hashes is formed:

 //   etagify.js //     . //     : // '/about': { md5: 'fa88257b77...' } var etags = {}; var _end = res.end; res.end = function(body) { var hash = crypto.createHash('md5'); //    ,   if (body) { hash.update(body); } //      etags[req.path] = { md5: hash.digest('hex') }; //    _end.apply(res, arguments); } 

And this is how the version of the resource is checked:

 return function(req, res, next) { var cached = etags[req.path]['md5']; //      ETag,     if (cached) { res.setHeader('ETag', '"' + cached + '"' } //     GET, if (connect.utils.conditionalGET(req)) { // ,    If-None-Match  ETag if (!connect.utils.modified(req, res)) { //       . //    ETag     304 Not modified res.removeHeader('ETag'); return connect.utils.notModified(res); } } } 

When can and when not to use etagify

The etagify approach is extremely simple and works well for dynamically created pages that do not change during server operation, for example, multilingual static content. In other cases, using etagify may cause problems.

• If the content of the page does change, the user will see the version from the cache.
• The behavior of personalized pages will depend on the state of the Vary header.
  • If Vary:cookie used Vary:cookie for separate caching of individual pages, the array of etagify hashes will quickly grow to huge sizes.
  • If Vary:cookie missing, then all users will be shown the same version, first hit the cache.

Measure performance improvement

We did not expect a serious increase in speed from using etagify, since we still have to drive conditional GET requests, and the savings are only a few kilobytes per page. However, optimizing with etagify is very simple, so even a small increment will justify the effort.



We launched the Persona instance on awsbox (we’ll talk about awsbox in more detail in the last article of the series), opened firebug and made 50 measurements of the loading speed of the “about” page with etagify and without (The page load time was important for us. In other projects, there are other metrics - the time before the content is displayed, or the time before the first advertising message is displayed, etc.).

We analyzed the data - we calculated the average value and standard deviation for both data sets, assuming that they are distributed normally.

To our surprise, we found that etagify reduces the page load time by 9% from 1.65 (with a deviation of 0.19) to 1.50 (with a deviation of 0.13) seconds. Not a bad result, considering how little effort he demanded.

We then applied the Student's t-test to find out how likely such a result could have been achieved without etagify. The p-value turned out to be less than 0.01, that is, the probability of a random distribution of the same distribution is less than 1%. Thus, the measured improvement is statistically significant.

Here is how it looks on the chart:



etagify is a tiny but very useful module for us. Even if it doesn’t find application in your project, we hope that our approach to resolving problems, which is to create highly specialized tools for each specific task and a thorough measurement of their effectiveness, will give you inspiration and food for the mind.

---