Why write your React Data Grid in 2019

Hi, Habr! I am involved in the development of an ECM system. And in a small series of articles I want to share our experience and history of developing my React Data Grid (hereinafter referred to as a grid), namely:

• why we abandoned the finished component
• what problems and tasks we faced when developing our grid
• What profit does the development of its grid

Prehistory

Our system has a web application in which users work with lists of documents, search results, directories. Moreover, the lists can be either small (10 employees) or very large (50,000 contractors). To display these lists, we developed our grid:

When we first started developing a web application, I wanted to find a ready library for displaying a grid that can do everything we need: sort and group records, drag and stretch columns, work with multiple selections, filter and calculate totals by columns, portionwise upload data from the server and display tens of thousands of records.

Let me explain the last requirement “to display tens of thousands of records”. In grids this requirement is implemented in several ways: paging, infinity scrolling, virtual scrolling.

The paging and infinity scrolling approaches are common on web sites, you use them every day. For example, paging on Google:

Or infinity scrolling in the same Google in pictures, where the next portion of pictures is loaded when you scroll through the first portion:

But virtual scrolling (I will call virtual scrolling hereafter) is rarely used on the web, its main difference from infinity scrolling is the ability to quickly scroll very large lists to any place. In this case, only the data visible to the user will be loaded and displayed.

For our web application, we wanted to use virtual scrolling. I agree that scrolling to any place in the list of 10,000 entries is rather a fictional case. However, arbitrary scrolling within 500–1000 records is a live case.

When virtual scrolling is implemented, the software management API of this scrolling is often implemented. This is a very important feature. Software scrolling is used, for example, to position a highlighted entry in the middle of the screen when opening a directory:

Let's return to the requirements. What else we needed:

• Virtual scrolling management software API
• Customizing the appearance of the grid (rows, columns, context menu) so that the grid does not look foreign in our application.
• Support technologies we use: react, redux and flexbox
• That grid worked in ie11

In general, there were many requirements.

Attempt the first (2016). DevExtreme JavaScript Data Grid

Not long examining existing libraries, we stumbled upon DevExtreme JavaScript Data Grid. According to the functional requirements, this grid covered all our needs and had a very presentable appearance. However, the technological requirements are not suitable (not react, not redux, not flexbox). At that time, DevExtreme had no react grid.

Well, let not react, we decided, for that the grid is beautiful and functional, we will use it. And they added the library to their project. It turned out we added 3 MB of scripts.

For a couple of weeks, we integrated the grid into our web application and enhanced the basic functionality:

• Wrapped over the grid to make it friends with react and redux
• We lifted virtual scrolling and portion loading of data from our web server
• Implemented sorting and selection

In the course of screwing up the grid, two serious problems emerged and a whole heap of less serious ones.

The first serious problem

Making friends with DevExtreme JavaScript Data Grid with redux is very difficult. We managed to manage the settings of the columns and the selection of records through redux, but storing the batch-loaded data in redux and performing CRUD operations on them through redux is unrealistic. I had to make a crutch that, bypassing redux, manipulated the grid data. The crutch turned out to be complex and fragile. It was the first alarm bell that the grid does not suit us, but we continued to screw it.

The second serious problem

No virtual scrolling management API. We couldn’t refuse software scrolling management, we had to override the DevExtreme sources and find the internal scrolling management API. Of course, this API had a mountain of restrictions, because it was designed for internal use. As a result, we achieved that the internal API more or less worked on our cases, but, again bypassing redux, and again a bunch of crutches.

Less serious problems

Less serious problems surfaced constantly, because the standard DevExtreme JavaScript Data Grid functionality weren’t completely matched, and we tried to correct it:

1. Stretching the DevExtreme grid height doesn't work. I had to write a hack to teach DevExtreme to do it (perhaps in recent versions there are no problems with this).
2. When the focus is not in the grid, it is impossible to control the selection of lines through the keyboard (and we needed it). I had to write my keyboard controls.
3. When changing the composition of columns and changing data, we had a problem of data blinking (with virtual scrolling turned on).
4. The problem of a large number of requests at the first showing of the grid. It was especially noticeable when we managed scrolling through the internal API.
5. It is difficult to customize some parts of the UI grid. For example, there was a desire to draw line management actions over the selected line of the grid (delete a line, copy, open a card). But how to twist it in DevExtreme was not clear, and even using react:
   
6. It is difficult to categorize the sorting (we wanted to sort by data that is not displayed in the grid, and not buried in the columns).
7. Crutches are required for screwing the react components into grid cells (because the grid is not on react).
8. No DevExtreme code typing (flow / typescript).
9. The problem of speed during long virtual scrolling.
10. The problem of speed when stretching / rearranging columns (after a long virtual scrolling).
11. Grid scripts size - 3 MB.

Although the DevExtreme grid in terms of functionality contained everything we needed, but almost all the standard functionality wanted to be rewritten. During its use, hundreds of lines of difficult to understand code were added, which tried to solve problems of interaction with redux and react, it was difficult to use a non-react grid in the react application.

Rejection of DevExtreme. Search for alternatives

After some time using DevExtreme, it was decided to abandon it. Throw out all the hacks, complex code, as well as 3 MB of DevExtreme scripts. And find or write a new grid.

This time, we were more attentive to the study of existing grids. MS Fabric DetailsList, ReactVirtualized Grid, DevExtreme React Grid, Telerik Grid, KendoUI Grid were studied.
The requirements remain the same, but they have already formed into a list that is understandable to us.

Technology Requirements:

• react
• redux
• flexbox

Requirements for functionality:

• Virtual scrolling (with the ability to show tens of thousands of entries)
• Scrolling management API
• Grid data and settings storage in redux
• Portion loading of data from the web server
• Column management (stretching / rearranging / visibility control)
• Sorting + Filtering by Column
• Multiple selection
• Like-search backlit
• Horizontal scrolling
• Keyboard operation
• Context menu (on line, on empty area, on columns)
• Support ie11, edge, chrome, ff, safari

By this time, the first version of DevExtreme React Grid has already appeared, but we immediately discarded it for the following reasons:

• Virtual scrolling is not supported in ie11
• Virtual scrolling does not work in conjunction with batch loading of data from the server (although there seems to be some workaround).
• And most importantly, I didn’t want to step on the same rake when I wanted to rewrite half of the standard third-party grid functionality.

Analysis of existing solutions showed that there is no “silver bullet”. Grid, which would close all our requirements, does not exist. It was decided to write our grid, which we will develop in terms of functionality in the direction we need, and be friends with the technologies needed by our product.

Develop your React Data Grid

Grid development started with prototypes, where we tried the most difficult topics for us:

• virtual scrolling
• storing all grid data in redux

Virtual scrolling

The most difficult was to make a virtual scrolling. In a big way, it is made in one of 3 ways:

1. Page-based virtualization
Data is drawn in chunks - pages. When scrolling, visible pages are added, invisible ones are deleted. The page consists of 20-60 lines (usually the size is customized). In this way, the products went: DevExtreme JavaScript Data Grid, MS Fabric DetailsList.

2. Line Virtualization
Only visible lines are drawn. As soon as the line leaves the screen, it is immediately deleted. This way went the products: ReactVirtualized Grid, DevExtreme React Grid, Telerik Grid.

3. Canvas
All lines and their contents are drawn using Canvas. So did Google Docs.

When developing the grid, we made prototypes for all three variants of virtualization (even for Canvas). And they chose paged virtualization.

Why abandoned other options?

Line-by-line virtualization had problems with rendering speed in the prototype. As soon as the contents of the lines became complicated (a lot of text, highlighting, trimming, icons, a large number of columns, and flexbox everywhere), it became expensive to add / delete lines several times a second. Of course, the results also depend on the browser (we did support, including for ie11, edge):

The Canvas version was very seductive in rendering speed, but time consuming. It was suggested to draw everything: text, text wrap, text trimming, highlighting, icons, dividing lines, selection, indents. Make a reaction to pressing the mouse buttons on the Canvas, highlighting the lines when you hover the cursor. At the same time, on top of the Canvas should be put some Dom-elements (showing hints, "pop-up actions" above the line). Still needed to solve the problem of blur text and icons in the Canvas. All this is long and difficult to do. Although we mastered the prototype. In this case, any customization of rows and cells in the future we would have resulted in greater complexity.

Pluses of virtualization virtualization

The selected page-based virtualization had advantages over line-by-line, which determined its choice:

• If the page is already rendered, then scrolling inside the page is cheap (the DOM tree does not change when scrolling). Line-by-line virtualization with any minor scrolling requires changing the DOM tree, and this is expensive when the DOM tree is complex and flexbox is used everywhere.
• For small lists (<200 entries), pages can be not deleted, only added. Sooner or later, all pages will be built, and scrolling will be completely free (in terms of rendering time).

Select page size

A separate question is the choice of page size. Above, I wrote that the size is customizable and is usually 20-60 lines. A large page is drawn for a long time, a small one leads to frequent display of a “white screen” when scrolling. Experimentally, the page size was 25 lines. However, for ie11, the size has been reduced to 5 lines. It feels like the interface in IE is more responsive if you draw many small pages with small delays than one large one with a large delay.

React and virtual scrolling

Page-based virtualization needed to be implemented using react. For this, several tasks had to be solved:

Task 1. How to add / delete pages through react when scrolling?

To solve this problem introduced the concept:

• page model
• page view

A model is information on which to build a presentation. A view is a React component.

In essence, the task of virtualization after that was reduced to manipulating page models: keeping a list of page models, adding and deleting models when scrolling. And according to the list of models through react, build / rebuild the mapping:

In the course of implementation, the rules for working with page models were formed:

• Pages should be added one by one. After each addition, give time to draw. It is acceptable to add 1 page every 300-500ms - this is a situation of fast scrolling. If you add, for example, 5 pages at once, then the user will hang up the interface on their construction.
• Pages do not need to keep dozens. Example of a problem situation: 20 pages are displayed, the user goes to another list and all 20 pages need to be deleted at once. Deleting a large number of pages is an expensive operation, stripping the DOM of a tree will take 1 second. To avoid this, it is better to keep no more than 10 pages at a time.
• For any manipulation of the columns (permutation, addition, deletion, stretching) it is better to remove the pages invisible to the user in advance. This will avoid expensive rebuilding of all rendered pages.

Task 2. How to display scollbar?

Virtual scrolling assumes that a scrollbar is available, which takes into account the sizes of the list and allows scrolling to any place:

How to display such a scollbar? The simplest solution is to draw an invisible div of the desired size instead of real data. And already on top of this div display the visible pages:

Task 3. How to monitor the size of the viewport?

Viewport is the visible data area of ​​the grid. Why keep track of its size? To calculate the number of pages to display to the user. Suppose we have a small page size (5 lines) and a large screen resolution (1920x1080). How many pages should be displayed to the user to close the entire viewport?

You can solve this problem if you know the height of the viewport and the height of one page. Now let's complicate the task, suppose the user dramatically changes the scale in the browser - sets 50%:

The situation with the scale shows that it is not enough to know the size of the viewport once, the size must be monitored. And now we’ll completely complicate the task: html elements don’t have a resize event to which you can subscribe and track the size. Resize is only for the window object.

The first thing that comes to mind is to use a timer and constantly poll the height of the html element. But there is an even better solution that we saw in DevExtreme JavaScript Data Grid: create an invisible iframe, stretch it to the grid size and subscribe to the resize event in iframe.contentWindow:

Summary

PS This is not the end. In the next article I will tell how we made friends with our redux grid.

To get a full-fledged virtual scrolling, many other tasks had to be solved. But those described above were themselves interesting. Here are a few more tasks that came up too:

• Take into account the direction and speed of scrolling when adding / deleting pages.
• Consider changing data to minimize rebuilding of page models. For example, they deleted one line, or added a line, what to do with the already rendered pages? All thrown out, or some leave? There is room for optimization.
• When changing the selection rebuild the minimum required number of pages.

If you have questions about the implementation, you can write them in the comments.