Geekly Articles each Day
📜 ⬆️ ⬇️

Create a map of the whole world based on OpenStreetMap data

Prehistory


When developing a project for online navigation (in the future and offline) it became necessary to develop your own online maps. Services from Google, Bing, etc. have restrictions on using their cards for commercial purposes. Free, such as OpenStreetMap, did not satisfy customer requirements.
There was a question about creating an online map (tile maps) for the whole world. Style close to Google and Skobbler. Data based on OpenStreetMap.


Restrictions


As you know, file systems have limitations in the number of files and directories. Therefore, storing each tile (card tile) separately rests on these limitations. For example, only for the 14th zoom you need 267,583,488 tiles (~ 4 14 ).
View the maximum number of files in Linux: df -i
The solution is to store tiles in the database.

Iron and axis


Server generation and download data: Supermicro server. 24 cores at 2GHz, 64Gb RAM, HDD 2.7Tb. With Ubuntu 13.04 on board.
The server on which the generation results are used is much more modest: 8 cores at 1.8GHz, 27Gb RAM, 130Gb SSD.
')

Installation and Setup


Separately dwell on the installation will not. Since it is well described in other articles (see links below).
Established:


Development


In order not to reinvent the wheel turned to open sources. The best option for storing tiles in the database format is MBTiles. This is a simple SQLite database that allows you to store millions of tiles in a single file.
The TileMill editor is well suited for developing and editing your own style, and it also allows you to save maps in the MBTiles format.
First you need to download a separate section (country) and work out styles. It is advisable to work out several countries with different alphabets: Latin, Cyrillic, Arabic, and hieroglyphs. So you will not face the unpleasant surprise of the lack of some names or incorrect display of the style for different countries. I do not advise you to load the whole world and work on styles, since the dynamic generation time in the editor is greatly increased.

Loading data into PostgreSQL

PostgreSQL tuning

To speed up the boot process, let's make some changes in the /etc/postgresql/9.1/main/postgresql.conf file .
max_connections = 150 - calculated empirically, depending on the number of processors. The default is 100.
shared_buffers = 7GB - <25% of system RAM. For this parameter, you need to increase the kernel.shmmax.
temp_buffers = 512MB - used to access temporary tables, needed for the slim mode utility osm2pgsql.
work_mem = 3GB - used to sort “ORDER BY” and combine “JOIN”.
maintenance_work_mem = 16GB - used for VACUUM, CREATE INDEX, ALTER TABLE ADD FOREIGN KEY commands.
random_page_cost = 3.0 - this is the processor load settings. Be careful with this parameter!
effective_cache_size = 42GB - up to 66% of RAM.
checkpoint_segments = 50
autovacuum = off

Download using osm2pgsql

Download data on planet.osm.org.
The utility osm2pgsql was used to load data into PostgreSQL. To load the whole world, the slim mode option was used, it allows you to limit the use of RAM, while creating temporary tables to store intermediate results.
Before downloading, we will fix the stylesheet file, which in Ubuntu is located in /usr/local/share/osm2pgsql/default.style . For non-Latin names, I added three fields: int_name (original), name: en (name in English), is_in: country (country).
Team to download the whole world:
 time sudo -u postgres osm2pgsql -r pbf -sc -d gis -C 40000 --number-processes 24 --cache-strategy dense --unlogged planet-latest.osm.pbf

I will explain some parameters:
-r pbf - pbf format is used;
-sc - slim mode ( s ) with the creation of new tables ( c ), to add, use a ;
--number-processes 24 - the number of parallel processes, by the number of CPU-s;
--unlogged - do not log transactions.
Download the whole world took 28 hours.

Creating your own styles in TileMill

Preparation for non-Latin names

As I wrote above to get started, I downloaded several individual countries into PostgreSQL. Since in the world many alphabets are used, it was necessary to process and inscriptions not written in Latin (after the original in brackets indicate the name in Latin). To do this, create a nonlatin field in the planet_osm_point table:
 ALTER TABLE planet_osm_point ADD COLUMN nonlatin boolean; UPDATE planet_osm_point SET nonlatin='1' WHERE name similar to '%[^\x20-\x7e]+%';


Editing Styles

In order to have default styles you need to install OSM Bright for TileMill.
TileMill can work with three data sources: files, SQLite, PostGIS. For styling maps using CartoCSS, if you are familiar with CSS, you will find a lot in common.
Be careful, some filters (pre-PostGIS PostgreSQL queries) from OSM Bright styles are quite heavy and take a long time to complete. Therefore, you need to optimize SQL queries for acceleration.
Sampling of Latin and non-Latin names in queries:
 SELECT .., CASE WHEN nonlatin=true AND "name:en" IS NOT NULL THEN CONCAT("name:en",' (',name,')') ELSE name END as en_name ...


MBTiles generation

In TileMill choose Export / MBTiles. Specify the file name, zooms (map approximation), center coordinates and borders.
The boundaries of the whole world: -180.0, -85, 180.0, 85
For convenience, generated a map of individual zooms.

Zoom 15, 16, 17 decided to leave the dynamic generation. To speed up the work use caching.

Merge individual MBTiles files

To merge individual MBTiles files, you can use the command:
 echo '.dump' | sqlite3 file1.mbtiles | sqlite3 file2.mbtiles

The result will be in file2.mbtiles.

Using

Run MBTiles

To run MBTiles, use Node.js:
 nohup node /usr/share/tileserver/mbtiles-server/server.js /media/data/mbtiles/filename.mbtiles PORTNUMBER &

Where PORTNUMBER is the port number by which the map tiles (tiles) will be available.

NGINX configuration file

To access the map use nginx.
Sample configuration file:
 server { listen 80; server_name your_server_map_tiles_name.com; #dinamically generation location ^~ /15/ { rewrite ^/15/(.*)$ http://localhost:20008/tile/projectName/15/$1 last; } ... #using MBTiles via runned port location / { proxy_pass http://localhost:PORTNUMBER; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; } }

Thus, your map tile will be available at your_server_map_tiles_name.com/zoom/x/y.png .

Features and wishes


It is best to place the MBTiles file on the SSD for faster download.
For iPhone applications, non-standard Retina tiles are used, 512x512 pixels in size.
To reduce the load for dynamic generation (zooms 15-17), caching is used in Redis.

Result


The result can be seen here: tourstart.org/drive
Or download the iPhone application: itunes.apple.com/app/tourstart/id586049610?mt=8

Articles used in the process


habrahabr.ru/post/144675
switch2osm.org/serving-tiles/manually-building-a-tile-server-12-04
wiki.openstreetmap.org/wiki/Osm2pgsql
github.com/mapbox/mbtiles-spec
www.mapbox.com/tilemill/docs/linux-install
www.mapbox.com/tilemill/docs/guides/osm-bright-mac-quickstart
www.mapbox.com/carto/api/2.1.0

Excuse me. There is not enough karma to put it in the OpenStreetMap hub.
I will be glad to answer your questions.

Source: https://habr.com/ru/post/203212/

More articles:


All Articles