Generation of site pages by means of service workers

( C )

From this article, you will learn how to create a page with a list of previously cached site materials right on your mobile device, in your browser, so that the conditional user stuck in the elevator will not get bored without the Internet. As we approach the goal, we will touch on the following topics:

• caching site pages for offline access;
• keeping records of offline pages, forwarding necessary data;
• monitoring network status, online or offline;
• communication service worker with the pages and tabs that he serves;
• interception by the service worker of the request to open the address /offline/ and generate a new page directly on the device, without a request to the server.

If the topic of service workers and Progressive Web Apps (PWA) is new for you, then you should get to know them better before reading this article.
')
My name is Pavel Rybin, I work in the front-end development of Mail.Ru Group Media projects. This guide helped me write rakes, stuffed cones and pitfalls that I came across when implementing PWA for the mobile version of Auto Mail.Ru.

In the text there will be small code examples illustrating the story. An enhanced demo version can be viewed on GitHub .

Connecting a service worker

A service worker serving the entire site should be located at the root. For example, have the address /service-worker.js . In our case, this is required. If you give a service worker file from the /js/ directory, for example /js/service-worker.js , then it will be able to process only those network requests that start with /js/ ...

We connect a service worker from our website:

 // app.js -     //      service worker if ('serviceWorker' in navigator) { window.addEventListener('load', registerServiceWorker); } function registerServiceWorker () { //     ,   //       navigator.serviceWorker.register('/service-worker.js') .then(registration => { if (!registration.active) { //    return; } // - ,    . //        . }); } 

In our example, the service worker's initialization code should contain a complete list of resources needed to correctly render the future page /offline/ , all styles, images, etc. We pre- cache them with the install event, the first one in the chain of life cycle events.

 // service-worker.js // ,    const dependencies = [ '/css/app.css', '/js/offline_page.js', '/img/logo.png', '/img/default_thumb.png' ]; //  , -    self.addEventListener('install', event => { //   ,    offline- const loadDependencies = self.caches.open('myApp') .then(cache => cache.addAll(dependencies)); // -      , //         event.waitUntil(loadDependencies); }); 

Next activate event. It is useful to us in order to clear the old cache and records in the database. In our example, a simple idb-keyval helper is used to work with IndexedDB . He and his more advanced idb brother are convenient wrappers that allow to work with the morally outdated API IndexedDB .

 // service-worker.js import { clear } from 'idb-keyval'; // ,    const dependencies = [/* ... */]; //  self.addEventListener('activate', event => { //    IndexedDB const promiseClearIDB = clear(); //    const promiseClearCache = self.caches.open(cacheName) .then((cache) => cache.keys() .then((cacheKeys) => Promise.all(cacheKeys.map((request) => { //  ,     , //    const canDelete = !dependencies.includes(request.url); return canDelete ? cache.delete(request, {ignoreVary: true}) : Promise.resolve(); })))); const promiseClearAll = Promise.all([promiseClearIDB, promiseClearCache]) .catch(err => console.error(error)); //   - , //      IndexedDB event.waitUntil(promiseClearAll); }); 

After activation, the service worker is ready to go. It will be able to process network requests and receive messages from all pages of our site that were opened after its activation. Just add the appropriate event handlers. This is where we will catch the page request /offline/ .

 // service-worker.js //     self.addEventListener('fetch', event => { const { request } = event; const url = new URL(request.url); if (url.origin !== self.location.origin) { //  ,     return fetch(request).catch(err => console.log(err)); } // ,     /offline/ const isOfflineListRequested = /^\/offline\//.test(url.pathname); const response = isOfflineListRequested //   response   //  ,   ? createOfflineListResponse() //   . //         //   , -  " ", // -     .. : fetchWithOneOrAnotherCacheStrategy(event.request); event.respondWith(response); }); 

What are “caching strategies” and why are they needed?

The resources we download play a different role on the page. This could be a logo image or some kind of shared JS library, which most likely will never change. This can be JSON with comments that are updated every five minutes.

Files and documents involved in the construction of the page, in its life cycle, depending on the purpose can be divided into groups:

• can be cached "forever";
• cannot be cached for long;
• can be cached, but update when appropriate;
• and so on, limit this list only to your imagination and business objectives.

If you implement filtering of such groups by address, file type, anything, then you can apply your own logic for the mutual operation of network requests and the local cache to each of them. You can see several examples of different caching strategies in the sample repository .

Now we have already implemented support for offline mode for cached pages. They will open when activated in the phone mode "in the plane." Now you need to collect them all in one place, on a separate page.

Offline registration of pages

To draw a page of the list of materials available for viewing offline, you must first create this list, and then, each time you open a new page, update it. The logic for registering pages will be as follows:

• When you open the page, we will form a data object that describes this page (address, title, preview address for output in the list).
• After generating the data, send it to the service worker via postMessage .
• The service worker will receive the data and add it to the general list.

The script that collects data about the page will be executed on it, being part of the page. So comfortable. This will allow to forward necessary information, for example, through the head block and receive it from the layout. Or in any other suitable way for you.

Let's use Open Graph microdata . Today it is difficult to imagine a site without it. In addition, it can be used to transfer all the necessary information in our case:

 <meta property="og:title" content="Homer Simpson" /> <meta property="og:url" content="http://example.com/homer.html" /> <meta property="og:image" content="http://example.com/homer.png" /> 

Why transfer the page address through layout? Why not get it in JS through the location object?

Today, most sites use for analytics all sorts of get-parameters, marking, for example, the source of traffic. The result is that the addresses /homer.html , /homer.html?utm_source=vk and /homer.html?utm_source=email actually lead to the same page, which means they must be registered in the list once. Here the “canonical” address transmitted via og:url will help us, it will always be the same. Most likely you already have all the necessary og-markup, you can check its completeness with the help of Google Chrome extension .

So, let's teach the page to tell the service worker that it is loaded. We will finalize the registerServiceWorker function (see above).

 // app.js -     function registerServiceWorker () { navigator.serviceWorker.register('/service-worker.js') .then(registration => { if (!registration.active) { //    return; } // - ,     // ,       registerPageAsCached(); }); } /** *    ,    */ function registerPageAsCached () { //  getPageInfoFromHtml    , // ,       : // url - ""   // title -   // thumb -    const page = getPageInfoFromHtml(); //   - postMessage({ action: 'registerPage', page }); } /** *    service worker * @param {object} message */ function postMessage (message) { const {controller} = navigator.serviceWorker; if (controller) { controller.postMessage(message); } } 

Note: in the message, in addition to the page data, we pass an action field describing the type of the message. This will allow us in the future to transmit different data for different purposes.

Someone will ask, how do we know that the page is cached?

All requests from our site go through one of the caching strategies that we entered into the work earlier, which means we accept that everything displayed in the browser passed through the cache.

Get data from the page in the service worker:

 // service-worker.js import {get, set} from 'idb-keyval'; /* *     */ self.addEventListener('message', event => { const {data = {}} = event; const {page} = data; //    , // ,  action switch (data.action) { case 'registerPage': addToOfflineList(page); break; } }); /** *  ,    * @param {object} pageInfo * @return {Promise} */ export function addToOfflineList (pageInfo) { //   ,   , //    offline  if (pageInfo.thumb) { fetchWithOneOrAnotherCacheStrategy(pageInfo.thumb); } //      IndexedDB return get('cachedPages') .then((pages = {}) => set('cachedPages', { ...pages, [pageInfo.url]: pageInfo })); } 

Page registered.

In this example, we used the page title, address, and image for the page description, but the list of data can be expanded. For example, it makes sense to specify the timestamp of the last update. This will allow you to sort articles by download date, as well as remove old materials from the cache.

Monitoring network connectivity status

While the page is open, we will teach it to monitor the status of the network, or rather, the availability of our server. When the server stops responding, a corresponding message appears with a link to /offline/ , which we will do later. Also, for convenience, we will highlight the available links directly on the page.

You can make inaccessible materials dull by visually highlighting cached ones:



And on the contrary, it is possible to select articles stored in the cache with a special icon, as was done on Auto Mail.Ru :



In the script that runs on the page, we will create a ping function that will periodically be called at a specified interval and send a message to the service worker.

 // app.js -     const PING_INTERVAL = 10000; // 10  function registerServiceWorker () { navigator.serviceWorker.register('/service-worker.js') .then(registration => { if (!registration.active) { //    return; } // - ,     registerPageAsCached(); // .  //   ping(); }); } /** *     ( ) */ function ping() { postMessage({ action: 'ping', }); setTimeout(ping, PING_INTERVAL); } /** *    service worker * @param {object} message */ function postMessage (message) { const {controller} = navigator.serviceWorker; if (controller) { controller.postMessage(message); } } 

On the side of the service worker, we will receive a message, check the server availability and send back a report. For verification, you can request any URL, it is better if it will be some kind of static, for example, a traditional pixel.

 // service-worker.js import {get} from 'idb-keyval'; /* *     */ self.addEventListener('message', event => { const {data = {}} = event; const {page} = data; //    , // ,  action switch (data.action) { case 'ping': ping(); break; } }); /** *    */ export function ping () { fetch('/ping.gif').then( () => pingHandler(true), () => pingHandler(false) ); } /** *       *      * @param {boolean} isOnline */ function pingHandler (isOnline) { postMessage({ action: 'ping', online: isOnline, }); } /** *       , *  - * @param {object} message */ function postMessage (message) { //         self.clients.matchAll().then(clients => { //       //    const offlinePagesPromise = message.online ? Promise.resolve() : get('cachedPages'); offlinePagesPromise.then(offlinePages => { if (offlinePages) { message.offlinePages = offlinePages; } clients.forEach(client => { //   ,   if (client) { client.postMessage(message); } }); }); }); } 

In the browser, several tabs with our site can be opened at once. Each will call its own ping method. Therefore, it is better not to load a pixel from the page, but through a service worker who can control the frequency of network check requests, for example, through the throttle micro pattern . Also, the knowledge of the status can be useful to the service worker himself.

The page, having received the report, performs the necessary manipulations with its contents:

 // app.js -     let isOnline = true; function registerServiceWorker () { navigator.serviceWorker.register('/service-worker.js') .then(registration => { if (!registration.active) { //    return; } //     - serviceWorker.addEventListener('message', handleMessage); registerPageAsCached(); // .  ping(); // .  }); } /** *    service worker- * @param {MessageEvent} e */ function handleMessage (e) { const {data} = e; if (data.action === 'ping' && isOnline !== data.online) { isOnline = data.online; toggleNetworkState(data); } } /** *   /  * @param {object} params */ function toggleNetworkState (params) { const {online, offlinePages = {}} = params; //   , //        "" document.body.classList.toggle('offline', !online); //       if (!online) { Array.from(document.links).forEach(link => { const href = link.getAttribute('href'); const isCached = !!offlinePages[href] || href === '/offline/'; link.classList.toggle('cached', isCached); }); } } 

Creating a page / offline / service worker

So, we got to the main, to create a page inside the service worker without contacting the server. We will need a template that draws HTML, and data about cached pages.

I used a simple pug template engine in the demo version . However, you can use any other one up to the “server renderer” for an isomorphic React application.

My template looks like this:

 html(lang="en") head title Available offline link(rel="stylesheet" href="/css/app.css") body section.layout header.layout__header a.layout__header__logo(href="/") h1 You can read it offline ul.articles-list each page in pages li.articles-list__item a(href=page.url) if page.thumb img.avatar(src=page.thumb alt="") else img.avatar(src="/img/default_thumb.png" alt="") span=page.title 

In the service worker, in the fetch event handler, select the request to /offline/ and return the newly created page to the unsuspecting browser:

 // service-worker.js import {get} from 'idb-keyval'; const template = require('offlinePage.pug'); //     self.addEventListener('fetch', event => { const { request } = event; const url = new URL(request.url); if (url.origin !== self.location.origin) { //  ,     return fetch(request).catch(err => console.log(err)); } // ,     /offline/ const isOfflineListRequested = /^\/offline\//.test(url.pathname); const response = isOfflineListRequested //  ! //   response   //  ,   ? createOfflineListResponse() //   : fetchWithOneOrAnotherCacheStrategy(event.request); event.respondWith(response); }); /** *   response    , *   * @return {Promise<Response>} */ function createOfflineListResponse () { //      return get('cachedPages') .then((pagesList = {}) => { //       const html = template({ pages: Object.values(pagesList) }); //      const blob = new Blob([html], { type: 'text/html; charset=utf-8' }); return new Response(blob, { status: 200, statusText: 'OK' }); }).catch(err => console.error(err)); } 

Result:

At last

In order not to inflate this manual, some topics had to be omitted. However, they are extremely important. The most important is clearing the cache. This should be done regularly and independently, otherwise the space provided by the browser will end.

It makes sense to keep the resources that are required regularly in the cache: CSS, JS, images of interface elements. For the rest, you should come up with some "rule of swelling." For example, delete everything that was not requested for more than three (five, ten, year?) Days.

For easier debugging, detailed logging of each of the steps is useful. To do this, you can create your own utility log , which inside is able to turn on / off the flag from the environment, and display information through it. Unlike the pages, the service worker continues to live between their reloads and closing, so I recommend turning on the Preserve log checkbox in the developer’s tools console.

Thank you for reading this line. Write questions, ideas and insights from personal experience in the comments. If it turns out that the topic is in demand (I have fears that it is too narrow), I will continue it.

useful links

• Demo version of the tutorial on GitHub
• Service Worker API
• Cache
• IndexedDB
• Utilities for working with IndexedDB: idb and idb-keyval
• Expansion for checking open graph markup
• JS Optimization Micropatters