Infinite scrolling in web applications with examples on AngularJS

Misko Heveri, the main developer of Angulyar, once mentioned that the application is guaranteed to work without brakes, if there are no more than 100 active scopes in it. This approach is generally applicable to any applications. In games, they do not render for a long time what the player does not see and only on the web is still considered the norm to display the entire list of several thousand items. With the advent of js frameworks, the situation should change and the best solution would be to remove from the DOM what is not on the screen, rather than reject intermediate tags, binding and other things that facilitate development. Therefore, I conducted a small analysis of solutions for displaying large lists. Stumbled upon a couple of articles:

1. The Infinite Path of Scrolling

In it, the guy says that he had an internship at Google on Angulyar’s team and was assigned to investigate this issue. (I am glad that the developers are interested in this. I hope that we will soon see the native support of the endless scroll).
')
There are two solutions: good old pagination and endless scrolling. Moreover, the infinite scrolling can both load the necessary data and delete what has already been viewed, what is implemented, for example, in the UITableView component from IOS. The author tried to recreate this component for Angulyar.



In his scroller, he separated the logic of the main model from the logic of the actually displayed data and tried to create several predefined implementations for the list, the grid of icons, the list scrolling through the columns. But for each implementation, I had to write a lot of my code, which confused him.

Initially, the CSS transformation was used for animation, but problems arose with it when the user returned to the data already viewed. Therefore, he left this venture. In order not to tinker with the indices of individual elements, it was decided to use page identifiers (a page is a portion of the data received), where the offset is specified as the identifier of the previous (or subsequent if we request new elements) element and the maximum number of elements we would like . Thus, the pagination is not tied to the original state.

It is worth noting that the component should be universal and handle the scrolling of both the entire browser window and internal areas , if the scrolling is local.



The idea is to insert the first portion of the data set into an empty container (the number of elements in the portion requested from the server is configurable). Then, when the user scrolls to the end of the buffer, a new chunk is requested. A preloader may be displayed at the bottom at this time.

When the user scrolls below, the items above the viewport are removed from the DOM and transferred to the pool of unused items. When a new data arrives in response to a request to the server, the component first looks to see if there are reusable elements and either replace existing elements with new ones or create a new element. These elements are then added to the end of the container.

The user can scroll back (up) and the same happens in reverse order. Items that are not visible at the end are pooled. Other items are taken from the data store and placed before the first visible item, and deleted items are reused when we receive data.

Deleting items as soon as they go out of the viewport is not a good idea, because you will have to request them again if the user scrolls backwards through several elements. It makes sense to put the recently loaded items in the cache and only very old again to request from the server.



One of the most important features of the scroller is the support of several templates , so that there can be different elements in the stream, for example, elements with text, photos, links, etc., as in Facebook.

Minimum HTML required for the scroller:

<div ng-scroller> <div ng-scroller-repeat=”post in posts”> <div>{{post.author}}: {{post.text}}</div> </div> </div> 

The element with ng-scroller is the viewport, the element with ng-scroller-repeat is the container (the blue area in the diagram above) and the internal element of the template that will be duplicated and linked to the child scope created for each element.

The posts key in scope, in this case, must point to a data storage object that implements the following interface for querying items. If the key points to an array, for convenience, it gets a wrapped object that implements the required interface.

 IScrollerDataStore { void getRangeAfter(prev_id, length, function(Error, Array)); void getRangeBefore(next_id, length, function(Error, Array)); } 

If you need to display multiple templates, you should use the directives ng-if and ng-switch . In most cases this is enough.

 <div ng-scroller> <div ng-scroller-repeat=”post in posts”> <div ng-class=”{ link: post.link, photo: post.photo }”> {{post.author}}: <a ng-if="post.link" href=”{{post.link}}”>{{post.link}}</a> <img ng-if="post.photo" ng-src=”{{post.photo}}”> </div> </div> 

Initially, he wanted to allow developers to expand the scroller, but decided not to complicate people's lives and implement the main cases on their own. For example, a common case is the case with the “Load More” button, etc.



In scrolling containers, often make flowing elements, such as row headers, tables, footers, rows and columns of tables, headers of a contact list, as on Instagram, which stick only when the element is the topmost visible element in the viewport.



The first two types can be done in pure CSS. The latter is more difficult, because stickiness dynamically changes while scrolling.

When he discussed a scroller with the same developer who is working on a similar component inside Google, he advised to focus on working with the state of deleted items , which will have to be restored later. An example with tweets: you can click on a tweet and expand the entire conversation. What if, scroll the open tweet down so that it is removed from the DOM, and then scroll back. It will be necessary to restore his condition.

You can handle this in two ways. Either store the state in the scope of the element created using ng-scroller-repeat (ie tweet.open = true ) or somehow track the changes in the scope itself by creating a correspondence table with individual elements and restore them if the element is used again.

At the moment, he chose the first approach as the simplest.

Project on Gitkhab , demos
The project is obviously not finalized, but the ideas embodied in it are quite sound.

2. AngularJS Virtual Scrolling. Part 1 , Part 2

Here the person rewrote the ng-repeat directive, which he called sf-repeat . Parses the problem by the example of the log of logs.

 <div style="overflow: scroll; height:200px;"> <ol> <li ng-repeat="event in eventLog">{{event.time}}: {{event.message}} </ol> </div> 

He says that pagination in a dynamic application is a bad thing, because when deleting / adding elements, you need to monitor so that the elements are correctly transferred through the pages, there are no duplicates or lost records.

Trying to fix a problem with a filter.

 angular.module('sf.virtualScroll').filter('sublist', function(){ return function(input, range, start){ return input.slice(start, start+range); }; }); 

 <div style="overflow: scroll; height:200px;"> <ol> <li ng-repeat="event in eventLog|sublist:rows:offset">{{event.time}}: {{event.message}}</li> </ol> </div> 

Passing an expression to a directive is divided into two parts: the event value identifier and the collection eventLog|sublist:rows:offset identifier eventLog|sublist:rows:offset . The collection ID is calculated each time during the dirty check and compared with the previous value. Thus, only the visible range is calculated. If the collection has changed the screen is updated and if the value in the scope has changed, the visible position of the list changes.

It remains to give the user the ability to change the position of the scroll.

To ensure that the range of the scrollbar attached to the container (i.e. DIV with overflow: scroll ) does not change, you need to deceive the browser by adding an area with empty content. By changing the height of the empty content, we control the scroll range. The problem is how to get the height of the content.

Here the author proposes to take the average height of the element and multiply by the number of elements. This rolls, but the problem in situations where the dimensions of the elements are very different, remains.

Makes your scroll widget:

 <div style="overflow: scroll; height:200px;"> <div sf-scroller="y = 0 to eventLog.length" ng-model="slicePosition"></div> </div> 

Parsing a range expression (y = 0 to eventLog.length)

 function parseRangeExpression (expression) { var match = expression.match(/^(x|y)\s*(=|in)\s*(.+) to (.+)$/); if( !match ){ // throw an informative Error. } return { axis: match[1], lower: match[3], upper: match[4] }; } 

Directive

 var mod = angular.module('sf.virtualScroll'); mod.directive("sfScroller", function(){ //function parseRangeExpression ... return function(scope, element, attrs){ // ... }; }); 

He notes the shortcomings of this approach: the developer will have to make his own scroll bar, but for the user it will not look native anyway.

Decides to remake ng-repeat , parses it .

The first thing to note is the use in the transclude directive and compilation functions. Simpler directives use only the linking function, but ng-repeat needs access to the linking function, which will link the compiled elements to the new scope for each element in the collection (see the section “Reasons for separation of the compilation and linking stages” of the developer’s guide ).

In essence, the connecting function parses the expression in ng-repeat and sets watch-watchers. Observers are needed to add and remove items and synchronize with the collection, but you need to be careful when tracking the movements of items. If you have an element that corresponds to an element in the collection, and it has some state in the DOM (a good example is a form element), then you do not want the element to be deleted and re-created just because the underlying object has moved into the collection. This is not a trivial task, but after we have taken care of all the permutations, the code for adding new and removing existing elements becomes relatively simple.

Decides not to save the state of the DOM element when deleting, because the state should be stored in the model (the previous author did the same)

Another subtlety of ng-repeat is that the collection can be an object and the elements will be shown using for(key in collection) . Since because of this, there are problems with the indexes and the location of elements, decides to do only arrays.

Describes his sf-repeat , says that elements should come from the server in fixed portions.



Explains that you need to define marks after and before which elements are added or removed. To calculate the mark you need to specify the height of the line. There are problems with this again, because in the compilation / assembly phase it is impossible to determine that the element is fully rendered. Decides to calculate the height of the CSS (explicit or maximum).

The height of the viewport takes the same from CSS.

Directive

 var mod = angular.module('sf.virtualScroll'); mod.directive("sfVirtualRepeat", function(){ return { transclude: 'element', priority: 1000, terminal: true, compile: sfVirtualRepeatCompile }; // ... }); 

Compilation function

 function sfVirtualRepeatCompile(element, attr, linker) { var ident = parseRepeatExpression(attr.sfVirtualRepeat), LOW_WATER = 100, HIGH_WATER = 200; return { post: sfVirtualRepeatPostLink }; // ... } 

The link function evaluates the expression that defines the data set and we connect and set the observers as before, but these observers follow the upper and lower level labels that define the active range and, accordingly, the size of the collection. In addition, the link function sets the handler for the scroll event in view, which will adjust the current active range.

The observer algorithm can be reduced to comparing the new active range with the existing one and deciding which lines to destroy and which ones to create. The location of elements on a specific place can be made absolute positioning or indentation of the first element. The author has done with indentation.

The sf-virtual-repeat directive is part of the sf.virtualScroll module on GitHub. Source , bower-component , demo .

The main problem with this solution is that you must use CSS.

---

The articles describe well the approaches to solving the problem of endless scrolling, but have not yet found an optimal solution. Hope to see something useful in the comments.

PS LinkedIn Developer Study
PPS Question on Toaster . There are interesting answers