How incremental updates affect download speed. Experience Yandex. Mail
Yandex.Mail is a large and complex web application. For the initial load, it needs more than 1 MB of static resources (JS / CSS / Templates). At the same time, Yandex.Mail is updated twice a week, and sometimes more often.
But when updating from version to version, not so much code changes - especially in the case of hotfixes. This is shown and friezes. To reduce mail loading time when new versions are released, we already do the following:
But this is not enough for us. Even with a frieze, if the release changes only one file in which there are several lines, the hash from the content of this file changes and the cache becomes invalid, therefore the file is reloaded completely. To avoid this problem and even more efficiently load new resources, we have invented an incremental update mechanism.
We thought: “What if you store somewhere the old version of the files (for example, in localStorage), and when a new one comes out, transfer only the diff between it and the one that the user has?” In the browser, you just need to patch the client. That came out of this and what conclusions we came with Panya , read under the cut.
In fact, this idea is not new. Standards already exist for HTTP — for example, RFC 3229 “Delta encoding in HTTP” and Google SDHC — but for various reasons they have not been properly distributed in browsers and on servers.
')
We decided to make our own analog on JS. To implement this update method, we started looking for a diff implementation on JS. On popular hosting code found the library: VCDiff , google-diff-patch-match , jsdiff , Pretty Diff and jsdifflib .
The last two libraries (jsdifflib and Pretty Diff) didn’t fit right away, because they don’t know how to apply a patch, but show only the changes between the lines. And jsdiff generates a patch in a format similar to google diff patch match, but imposes it five times slower. As a result, we have two candidates left.
For the final choice of the library, we need to compare them on two key metrics for us. The first is the size of the generated patch. We generated patches for different resources of different versions and compared google diff patch match and vcdiff with different block sizes.
As seen in the results table, vcdiff with a block size of 20 bytes has the smallest patch size. The second key metric for us is patch imposition on the client.
Here, too, vcdiff wins by a large margin. In IE 9 and below, google-diff-patch-match applies a patch in more than a second.
We have a winner - vcdiff. This algorithm was proposed in 2002 ( RFC3284 ). It is quite popular and has many implementations in different languages, including C ++, Java and JS.
Once we have decided on the library for diff, we need to decide on where and how to store the static on the client. Yandex.Mail is a modern web application, we don’t have old browsers, so almost everyone supports localStorage. It is a convenient place to store static. No personal user data is there, so there are no security issues. Each file is stored in a separate key. This key contains not only the name of the resource, but also its version. This allows you to avoid problems when the version was able to register in localStorage, and the file itself is not (or vice versa), in the variant when the file and its metadata are stored in different keys.
When building a new release, we generate patches for several previous versions of each resource - for the beginning, we chose three. Information about the presence of a patch for previous versions is given in the bootloader.
The file format with patches for the project looks like this:
That is, it is an ordinary array of objects. Each object is a separate resource. Each object has three properties. "K" is the key name in localStorage for this resource. “P” is a patch for the resource that was generated by vcdiff. “S” is a cheksumma for the resource of the current version, so that later you can check the correctness of the patch application on the client. Cheksumma is calculated by the Fletcher algorithm .
Why the Fletcher algorithm, and not other popular algorithms like CRC16 / 32 or md5? Because it is fast, compact and easy to implement.
At the very beginning the loader looks in localStorage. If the user has nothing in it or the versions are so old that we have no patches for them, then all the files are downloaded.
If there is a saved old version in localStorage, send a request after the file with the patch. Next, update all the resources and check cheksummu. If all checksums match, update the contents of the keys in localStorage.
This completes the update process. Further we take all necessary resources from localStorage. The code of JS modules and compiled templates we execute via new Function (), and CSS is connected via dynamic tag insertion
But when updating from version to version, not so much code changes - especially in the case of hotfixes. This is shown and friezes. To reduce mail loading time when new versions are released, we already do the following:
- enable gzip;
- set caching headers;
- Frizim CSS, JS, templates and images ;
- we use CDN ;
- we load statics in parallel through cross-domain AJAX requests.
But this is not enough for us. Even with a frieze, if the release changes only one file in which there are several lines, the hash from the content of this file changes and the cache becomes invalid, therefore the file is reloaded completely. To avoid this problem and even more efficiently load new resources, we have invented an incremental update mechanism.
We thought: “What if you store somewhere the old version of the files (for example, in localStorage), and when a new one comes out, transfer only the diff between it and the one that the user has?” In the browser, you just need to patch the client. That came out of this and what conclusions we came with Panya , read under the cut.
In fact, this idea is not new. Standards already exist for HTTP — for example, RFC 3229 “Delta encoding in HTTP” and Google SDHC — but for various reasons they have not been properly distributed in browsers and on servers.
')
We decided to make our own analog on JS. To implement this update method, we started looking for a diff implementation on JS. On popular hosting code found the library: VCDiff , google-diff-patch-match , jsdiff , Pretty Diff and jsdifflib .
The last two libraries (jsdifflib and Pretty Diff) didn’t fit right away, because they don’t know how to apply a patch, but show only the changes between the lines. And jsdiff generates a patch in a format similar to google diff patch match, but imposes it five times slower. As a result, we have two candidates left.
For the final choice of the library, we need to compare them on two key metrics for us. The first is the size of the generated patch. We generated patches for different resources of different versions and compared google diff patch match and vcdiff with different block sizes.
| Vcdiff (block size 3) | Vcdiff (block size 10) | Vcdiff (block size 20) | google diff patch match |
|---|---|---|---|
| 13957 | 3586 | 3431 | 9297 |
| 865 | 367 | 309 | 910 |
| 4615 | 1854 | 1736 | 6740 |
As seen in the results table, vcdiff with a block size of 20 bytes has the smallest patch size. The second key metric for us is patch imposition on the client.
| Library | IE 9 | Opera 12 | Firefox 19 | Chrome |
|---|---|---|---|---|
| vcdiff (block size 10) | eight | five | five | 3 |
| google diff patch match | 1363 | 76 | 43 | 35 |
Here, too, vcdiff wins by a large margin. In IE 9 and below, google-diff-patch-match applies a patch in more than a second.
We have a winner - vcdiff. This algorithm was proposed in 2002 ( RFC3284 ). It is quite popular and has many implementations in different languages, including C ++, Java and JS.
Once we have decided on the library for diff, we need to decide on where and how to store the static on the client. Yandex.Mail is a modern web application, we don’t have old browsers, so almost everyone supports localStorage. It is a convenient place to store static. No personal user data is there, so there are no security issues. Each file is stored in a separate key. This key contains not only the name of the resource, but also its version. This allows you to avoid problems when the version was able to register in localStorage, and the file itself is not (or vice versa), in the variant when the file and its metadata are stored in different keys.
When building a new release, we generate patches for several previous versions of each resource - for the beginning, we chose three. Information about the presence of a patch for previous versions is given in the bootloader.
The file format with patches for the project looks like this:
[ { "k": "jane.css", "p": [patch], "s": 4554 }, { “k”: “jane.js”, “p”: [patch], “s”: 4423 }, ...]That is, it is an ordinary array of objects. Each object is a separate resource. Each object has three properties. "K" is the key name in localStorage for this resource. “P” is a patch for the resource that was generated by vcdiff. “S” is a cheksumma for the resource of the current version, so that later you can check the correctness of the patch application on the client. Cheksumma is calculated by the Fletcher algorithm .
Why the Fletcher algorithm, and not other popular algorithms like CRC16 / 32 or md5? Because it is fast, compact and easy to implement.
Update process
At the very beginning the loader looks in localStorage. If the user has nothing in it or the versions are so old that we have no patches for them, then all the files are downloaded.
If there is a saved old version in localStorage, send a request after the file with the patch. Next, update all the resources and check cheksummu. If all checksums match, update the contents of the keys in localStorage.
This completes the update process. Further we take all necessary resources from localStorage. The code of JS modules and compiled templates we execute via new Function (), and CSS is connected via dynamic tag insertion
.
80-90% . :
7.7.20 — 7.7.21 397 174 549 7.7.21 — 7.7.22 383 53 995 7.7.22 — 7.8 18 077 611 378 7.8 — 7.8.50 2 817 137 820 7.8.50 — 7.8.8000 14 868 443 159
?
. ?
. . 20, Opera 12 IE9 — 100. . , JavaScript — , , 120, — .
, — , , . , , , — , .
, , , JS . JS , — . CSS . CSS , , ( ). , CSS , — .
. 30% . , . HTTP-. , , — , , . , , . . « », , , .
VS. — . , . , , . , . , .
, , localStorage, , , .
, 146% . , — , . , , .
, HTTP/2 , RFC3229, . . ..
80-90% . :
7.7.20 — 7.7.21 397 174 549 7.7.21 — 7.7.22 383 53 995 7.7.22 — 7.8 18 077 611 378 7.8 — 7.8.50 2 817 137 820 7.8.50 — 7.8.8000 14 868 443 159
?
. ?
. . 20, Opera 12 IE9 — 100. . , JavaScript — , , 120, — .
, — , , . , , , — , .
, , , JS . JS , — . CSS . CSS , , ( ). , CSS , — .
. 30% . , . HTTP-. , , — , , . , , . . « », , , .
VS. — . , . , , . , . , .
, , localStorage, , , .
, 146% . , — , . , , .
, HTTP/2 , RFC3229, . . .
80-90% . :
7.7.20 — 7.7.21 397 174 549 7.7.21 — 7.7.22 383 53 995 7.7.22 — 7.8 18 077 611 378 7.8 — 7.8.50 2 817 137 820 7.8.50 — 7.8.8000 14 868 443 159
?
. ?
. . 20, Opera 12 IE9 — 100. . , JavaScript — , , 120, — .
, — , , . , , , — , .
, , , JS . JS , — . CSS . CSS , , ( ). , CSS , — .
. 30% . , . HTTP-. , , — , , . , , . . « », , , .
VS. — . , . , , . , . , .
, , localStorage, , , .
, 146% . , — , . , , .
, HTTP/2 , RFC3229, . . .
.
80-90% . :
7.7.20 — 7.7.21 397 174 549 7.7.21 — 7.7.22 383 53 995 7.7.22 — 7.8 18 077 611 378 7.8 — 7.8.50 2 817 137 820 7.8.50 — 7.8.8000 14 868 443 159
?
. ?
. . 20, Opera 12 IE9 — 100. . , JavaScript — , , 120, — .
, — , , . , , , — , .
, , , JS . JS , — . CSS . CSS , , ( ). , CSS , — .
. 30% . , . HTTP-. , , — , , . , , . . « », , , .
VS. — . , . , , . , . , .
, , localStorage, , , .
, 146% . , — , . , , .
, HTTP/2 , RFC3229, . . ..
80-90% . :
7.7.20 — 7.7.21 397 174 549 7.7.21 — 7.7.22 383 53 995 7.7.22 — 7.8 18 077 611 378 7.8 — 7.8.50 2 817 137 820 7.8.50 — 7.8.8000 14 868 443 159
?
. ?
. . 20, Opera 12 IE9 — 100. . , JavaScript — , , 120, — .
, — , , . , , , — , .
, , , JS . JS , — . CSS . CSS , , ( ). , CSS , — .
. 30% . , . HTTP-. , , — , , . , , . . « », , , .
VS. — . , . , , . , . , .
, , localStorage, , , .
, 146% . , — , . , , .
, HTTP/2 , RFC3229, . . ..
80-90% . :
7.7.20 — 7.7.21 397 174 549 7.7.21 — 7.7.22 383 53 995 7.7.22 — 7.8 18 077 611 378 7.8 — 7.8.50 2 817 137 820 7.8.50 — 7.8.8000 14 868 443 159
?
. ?
. . 20, Opera 12 IE9 — 100. . , JavaScript — , , 120, — .
, — , , . , , , — , .
, , , JS . JS , — . CSS . CSS , , ( ). , CSS , — .
. 30% . , . HTTP-. , , — , , . , , . . « », , , .
VS. — . , . , , . , . , .
, , localStorage, , , .
, 146% . , — , . , , .
, HTTP/2 , RFC3229, . . .
80-90% . :
7.7.20 — 7.7.21 397 174 549 7.7.21 — 7.7.22 383 53 995 7.7.22 — 7.8 18 077 611 378 7.8 — 7.8.50 2 817 137 820 7.8.50 — 7.8.8000 14 868 443 159
?
. ?
. . 20, Opera 12 IE9 — 100. . , JavaScript — , , 120, — .
, — , , . , , , — , .
, , , JS . JS , — . CSS . CSS , , ( ). , CSS , — .
. 30% . , . HTTP-. , , — , , . , , . . « », , , .
VS. — . , . , , . , . , .
, , localStorage, , , .
, 146% . , — , . , , .
, HTTP/2 , RFC3229, . . .
.
80-90% . :
7.7.20 — 7.7.21 397 174 549 7.7.21 — 7.7.22 383 53 995 7.7.22 — 7.8 18 077 611 378 7.8 — 7.8.50 2 817 137 820 7.8.50 — 7.8.8000 14 868 443 159
?
. ?
. . 20, Opera 12 IE9 — 100. . , JavaScript — , , 120, — .
, — , , . , , , — , .
, , , JS . JS , — . CSS . CSS , , ( ). , CSS , — .
. 30% . , . HTTP-. , , — , , . , , . . « », , , .
VS. — . , . , , . , . , .
, , localStorage, , , .
, 146% . , — , . , , .
, HTTP/2 , RFC3229, . . ..
80-90% . :
7.7.20 — 7.7.21 397 174 549 7.7.21 — 7.7.22 383 53 995 7.7.22 — 7.8 18 077 611 378 7.8 — 7.8.50 2 817 137 820 7.8.50 — 7.8.8000 14 868 443 159
?
. ?
. . 20, Opera 12 IE9 — 100. . , JavaScript — , , 120, — .
, — , , . , , , — , .
, , , JS . JS , — . CSS . CSS , , ( ). , CSS , — .
. 30% . , . HTTP-. , , — , , . , , . . « », , , .
VS. — . , . , , . , . , .
, , localStorage, , , .
, 146% . , — , . , , .
, HTTP/2 , RFC3229, . . .
80-90% . :
7.7.20 — 7.7.21 397 174 549 7.7.21 — 7.7.22 383 53 995 7.7.22 — 7.8 18 077 611 378 7.8 — 7.8.50 2 817 137 820 7.8.50 — 7.8.8000 14 868 443 159
?
. ?
. . 20, Opera 12 IE9 — 100. . , JavaScript — , , 120, — .
, — , , . , , , — , .
, , , JS . JS , — . CSS . CSS , , ( ). , CSS , — .
. 30% . , . HTTP-. , , — , , . , , . . « », , , .
VS. — . , . , , . , . , .
, , localStorage, , , .
, 146% . , — , . , , .
, HTTP/2 , RFC3229, . . .
.
80-90% . :
7.7.20 — 7.7.21 397 174 549 7.7.21 — 7.7.22 383 53 995 7.7.22 — 7.8 18 077 611 378 7.8 — 7.8.50 2 817 137 820 7.8.50 — 7.8.8000 14 868 443 159
?
. ?
. . 20, Opera 12 IE9 — 100. . , JavaScript — , , 120, — .
, — , , . , , , — , .
, , , JS . JS , — . CSS . CSS , , ( ). , CSS , — .
. 30% . , . HTTP-. , , — , , . , , . . « », , , .
VS. — . , . , , . , . , .
, , localStorage, , , .
, 146% . , — , . , , .
, HTTP/2 , RFC3229, . . .Source: https://habr.com/ru/post/242503/
All Articles