Effective way to load a large number of points (geometries) on the map

Hi, Habr.

Today I will tell you how you can efficiently download a large number of points on the map (web or in mobile applications).

Quick dive

From the point of view of most modern mapping solutions, several basic approaches to drawing a map can be distinguished:

• raster (static files - pictures with tiles stored on a CDN and having their own TTL)
• vector

Here you need to make a remark, a tile, for those who are not vkurse, this is a special block grid (along with tiles, static images are also called), which is characterized by three coordinates (z - zoom, x, y), at each level the number of tiles $$$2 ^ {(2 \ times zoom)}$

The most interesting today is the vector format.

Why? Actually, you need to start with the problems associated with the raster format:

• Discrete zoom levels (from 0 to 20 in most cases)
• Hard binding of style and data on tile layers
• Classic raster flaws

What perspectives does the vector format offer? Actually, there are several:

• The zoom is not clearly associated with tiles on a certain layer, but may change with any accuracy and smoothness.
• Since it is often the client (without considering the option of intermediate rendering) that renders the map, it is he who determines what data and in what style he wants to draw
• Excellent quality regardless of resolution, etc., which in general is a classic vector advantage over raster

Actually, everything with a vector format is good, but there are also disadvantages. At a minimum, the vector format is more difficult to cache, it is not as obvious as in most cases with a raster format, where enough pre-generated tiles are folded onto a CDN, set up TTL and regeneration and no longer think about it.

In general, it is possible to talk about this for a very long time, my task was to give a brief description of the actual work mechanisms, now we can proceed to the topic of the post.

Decision

Actually, in terms of possible solutions, there are several approaches that have different support among the ready-made libraries (services):

1. Download the whole GeoJSON and render it (most fall off on this option :)). Actually, the advantage is that almost all popular libraries of Google Maps, Yandex Maps, Mapbox, Leaflet, etc., are supported in one way or another. However, not everyone does it efficiently, for example, on request to Google about an effective solution for loading a large number of points in Mapbox, the first link will be their solution - supercluster, which in fact is a server utility that loads, prepares GeoJSON data and returns it to the client.

   
   

2. Download only located in viewport. The idea is quite simple, we only load what is in the viewport (* in some cases, nearby tiles are lazily loaded). Realization of this is in the box, for example in Yandex Maps - LoadingObjectManager , including it can do clustering on the client. They also have RemoteObjectManager, which already expects clustered data from the server.

   
   

3. Pregenerating static tiles and loading them. It also takes place, especially in the case of high-load services, but not very interesting. Tiles are most often generated using a tool, such as Mapnik, and uploaded to a CDN, or a third-party utility / library is used, which does all this work on its own.

   
   
4. Mapbox Vector Tiles, is actually one of the vector formats for maps, which describes various geometries, their style, etc ... The idea of ​​using this method is to form your MVT endpoint, which itself can:
   
   • Return Geometries
   • Cluster points
   • Simplify geometry at low zoom levels
   • Other

This format is perfectly supported by solutions from Mapbox, there are also libraries for Leaflet.

From the point of view of storage for GIS data, which would also make it possible to effectively carry out operations on them, you can choose PostGIS, which by the way gives all the necessary operations for the above, including the generation of MVT (Mapbox Vector Tiles):

• Inclusion of geometry in a tile (rectangle) using the @ operator
• MVT Generation - ST_AsMVT, ST_AsMVTGeom

Thank you for your attention, I hope for a constructive feedback, if it is interesting, I can tell you about some points of working with maps!

Examples of the above are available in my GitHub.