Task

Sometimes owners of services, project managers and SEO-specialists have a desire to peek at the user, how he presses the buttons and what he splits his forehead. It happens that such a desire to pry allows you to identify interface problems, which may indirectly affect the efficiency of the service, and even profit.

I know several ways to solve this problem:

1. recruit a group of test subjects for usability testing;
2. integrate a third-party service to the site that records user actions;
3. write your decision.

Testing group

Quite an option that requires, nevertheless, the solution of a number of issues:

1. It is necessary to carefully select the composition of the testing team so that it includes as many users as possible from different age groups (from the target audience, of course), different qualifications and experience of use. At the same time, the size of the team also plays a role - a larger number will allow to get more data for analysis, but it will also be costly;
2. need to equip a place for research. At a minimum, office space, computing, audio and video;
3. it is necessary to distract employees from the main process or to attract additional ones to organize the process and analyze the results.

Even the successful solution of the above questions does not guarantee a more or less reliable result, and the costs of organizing the process are already frightening. In addition, there is a risk of loss of boundary cases, which are not so rare when used in combat.

Perhaps not this time.

Third-party analytics service

Most services offer heat maps of clicks and scrolling states, which allows you to more or less reliably determine what users saw on the site and what they noticed.

Some offer analysis of forms, key logging, and tracking of selection and copying, which also makes it possible to track complex human-machine interaction schemes.

Offhand found a couple of services:

1. WebVisor from Yandex;
2. HotJar.

For lack of an economically sound goal, take WebVisor. The service promised to remember and lose for us the entire user session, i.e. repeat user behavior on the resource from start to finish. Primary integration is trivial, just add the code on the page.

Immediately it should be noted that requests that are deliberately destructive for the session state (that is, all but GET) are ignored.

There are two versions. The principle of their work is about the same. First, we get the user authentication parameters and transfer to the server, which downloads the current page and saves it. On the client side, it collects user actions: mouse movements, clicks, data entry, and so on. The embedded code transmits actions to a server with a timestamp (delta from the beginning or date / time, did not figure out).

Later, you can go to the user sessions page in the Yandex.Metric service and play user actions. It seems all is well, but there are a number of problems.

Version one:

1. incorrectly transmits click events to a complex, modifiable interface. In the particular case, clicking on the cross icon of the modal window does not close this modal window;
2. any actions leading to AJAX requests, with the exception of GET, break off and the interface behavior is played incorrectly, which again does not give an understanding of what the user saw;
3. arbitrarily loses user actions.

The second version is in beta testing and contains the same flaws, plus sometimes, like a decent beta, it does not work from the word “at all”.

Hands did not reach HotJar. If you have experience of integration with SPA - share, please, but I have suspicions that there will be the same suffering.

Attempt to collect your decision

And here we smoothly approach the moment “a, give, I will try!”.

What you need to implement:

1. getting the initial state of the page. Do not lose the user's context (for different users the page may differ significantly);
2. tracking the size of the window object;
3. tracking the scrolling of page elements, including the document itself;
4. registration of user activity. Follow the movement of the mouse, clicks, data entry in the form fields;
5. compact exchange protocol with the server.

Collecting analytics should be considered for two options for the quality of the network connection - for a low-speed unstable channel and for more favorable cases.

Initial state

The initial state can be obtained in two ways:

1. take HTML and styles on the window.load event. This will create a load on the network, so if the quality of the network is poor, this option will have to be abandoned;
2. request a page on the server side. You must pass the user context. You can implicitly steal a cookie or ask you to explicitly pass the URL and authorization parameters at the time of library initialization.

In both cases, you need to disable all scripts from the saved page, since we will do the interface response to user actions yourself by recording user actions.

The question remains with styles and static data. In general, they can be left as they are, but special effects are possible if the statics changes at the time between receiving the initial state and when viewing the recorded activity.

At the time of loading the page save the current size of the window object.

Event Tracking

You need to keep track of all DOM subtree events regardless of the use of stopImmediatePropagation () / stopPropagation (). For this we will use addEventListener () with the parameter useCapture = true.

To track changes in the document structure and attributes, you can use the mechanism - MutationObserver. DOM Node for registering handlers will be taken in the initialization parameters, by default - document.

You also need to keep track of the scrolling position of the document.scroll window and the resizing of the window.resize window.

To identify the event object, we will build a CSS selector.

Exchange protocol

For lack of practical data, we will use the JSON format for exchange. To fight with ad blockers and any other things, you should use GET requests and return a small image, say, GIF format. For example, the URL / path / to / api / [JSON string] .gif.

It is worth remembering that the URL is not rubber, it has a limit of 2000 characters. It is a rather small value, taking into account the sending of information about the change in the document structure, so you should immediately attend to data compression, for example, using the GZIP algorithm, a JavaScript implementation of which exists. To transfer compressed data, you will have to additionally encode them in BASE64. You also need to provide for transmission in parts, but to test the concept it will be unnecessary at this stage.

Bottom line

The source code for the client library prototype is here .

For experiments the real project was chosen:

1. adaptive layout (HTML5, CSS3);
2. Elm. Widgets (registration / authorization forms, multi-step forms for creating custom content) and user profile (SPA);
3. built-in google maps.

Integration of problems did not cause, the performance of the browser is not visually observed.

The generated traffic was not as great as expected. Taking into account the limitations, save once every 10 seconds or, according to accumulation, 100 events, the data did not exceed 4 kilobytes. The compression ratio floats heavily from a few to dozens, but for fairly large volumes (tens of kilobytes) lies around tens, which is logical, since data is text and there are many duplicate substrings.

In the dry residue

The prototype showed its viability. To make it completely clear, you need to implement the player and the server side, where a number of problems are foreseen:

1. getting an initial state, fighting duplicate data. You may need to cache related data. There is a suspicion that you will have to use a headless browser for the initial playback of user activity in order to determine static elements;
2. cross browser support player. Zoo and interversion variety has not been canceled.

You should not forget about the features when changing attributes. For example, the style attribute cannot simply be taken through a collection of attributes (Element.attributes); you will have to use the HTMLElement.style.cssText property. The number of such nuances at the moment can not be assessed.

If you use a headless browser with pre-playing user activity, you should consider recording video. In this case, there is no need for a player, but the required amount of computing resources and the size of the storage of the resulting data increase, which may not always be rational.