📜 ⬆️ ⬇️

Maps in a browser without a network: open source strikes back

Some time ago I wrote about how it is possible to use maps on the web without a network and tried to do this with the help of Google maps. Unfortunately, the terms of use prohibited the modification of resources, and the code I wrote worked only with localStorage , so I decided to switch to the bright side of power, where the code is open, simple and understandable.

What do I want?


I want to make a map caching for full work without a network, for the first time you load a map, view tiles of interest (which will be cached in this case) and the next time a map with viewed tiles will be fully accessible without a network.

In principle, it is not necessary to cache on the fly and you can do the same for a specific region separately. But I just want to show you the approach.
')

What do we have?


In a modern web for storage of our data can approach:
Application Cache - for statics, but not for tiles.
Local Storage - using base64 data uri, synchronously, supported everywhere, but very little space.
Indexed DB - using base64 data uri, asynchronously, is supported in full and mobile chrome, ff, ie10.
Web SQL - using base64 data uri, asynchronously, designated as obsolete, supported in full-featured and mobile chrome, safari, opera, android browser.
File Writer is only chrome.

You can also try using blobs and blobs to reduce the space occupied by tiles, but this can only work together with Indexed DB . I will leave this venture for now.

So, if you combine Application Cache , Indexed DB and Web SQL , then you can solve the problem of storing tiles sufficient for normal use in modern browsers, including mobile ones.

Theory


In theory, we need:
  1. take an API;
  2. add all static in application cache;
  3. redefine the tile layer so that it loads data from our asynchronous storages;
  4. add logic for loading tiles into repositories.

Storage


To begin with we will organize key-value storage with basic operations add, delete, get for Indexed DB and Web SQL. There is one magic construct emr.fire('storageLoaded', storage); , which will be called after the storage is initialized and ready to use, so that the card does not fall when accessing the storage.

Implementing storage using Indexed DB
 var getIndexedDBStorage = function () { var indexedDB = window.indexedDB || window.mozIndexedDB || window.webkitIndexedDB || window.msIndexedDB; var IndexedDBImpl = function () { var self = this; var db = null; var request = indexedDB.open('TileStorage'); request.onsuccess = function() { db = this.result; emr.fire('storageLoaded', self); }; request.onerror = function (error) { console.log(error); }; request.onupgradeneeded = function () { var store = this.result.createObjectStore('tile', { keyPath: 'key'}); store.createIndex('key', 'key', { unique: true }); }; this.add = function (key, value) { var transaction = db.transaction(['tile'], 'readwrite'); var objectStore = transaction.objectStore('tile'); objectStore.put({key: key, value: value}); }; this.delete = function (key) { var transaction = db.transaction(['tile'], 'readwrite'); var objectStore = transaction.objectStore('tile'); objectStore.delete(key); }; this.get = function (key, successCallback, errorCallback) { var transaction = db.transaction(['tile'], 'readonly'); var objectStore = transaction.objectStore('tile'); var result = objectStore.get(key); result.onsuccess = function () { successCallback(this.result ? this.result.value : undefined); }; result.onerror = errorCallback; }; }; return indexedDB ? new IndexedDBImpl() : null; }; 


Implementing storage using Web SQL
 var getWebSqlStorage = function () { var openDatabase = window.openDatabase; var WebSqlImpl = function () { var self = this; var db = openDatabase('TileStorage', '1.0', 'Tile Storage', 5 * 1024 * 1024); db.transaction(function (tx) { tx.executeSql('CREATE TABLE IF NOT EXISTS tile (key TEXT PRIMARY KEY, value TEXT)', [], function () { emr.fire('storageLoaded', self); }); }); this.add = function (key, value) { db.transaction(function (tx) { tx.executeSql('INSERT INTO tile (key, value) VALUES (?, ?)', [key, value]); }); }; this.delete = function (key) { db.transaction(function (tx) { tx.executeSql('DELETE FROM tile WHERE key = ?', [key]); }); }; this.get = function (key, successCallback, errorCallback) { db.transaction(function (tx) { tx.executeSql('SELECT value FROM tile WHERE key = ?', [key], function (tx, result) { successCallback(result.rows.length ? result.rows.item(0).value : undefined); }, errorCallback); }); }; }; return openDatabase ? new WebSqlImpl() : null; }; 


Creating a repository
 var storage = getIndexedDBStorage() || getWebSqlStorage() || null; if (!storage) { emr.fire('storageLoaded', null); } 


I propose to consider this implementation very schematic, I think there is something to think about, for example, in order not to block the initialization of the map while the storage is initialized; remember which tiles are in the repository without directly accessing the API; try to combine multiple save operations into a single transaction to reduce the number of writes per disk; try using blobs wherever they are supported. Perhaps the implementation of Indexed DB in old browsers will fall, onupgradeneeded event may not be implemented in them.

IMG to data URI & CORS


In order to store the tiles, we need to convert them to the data URI, that is, the base64 representation. To do this, use the canvas and its methods toDataURL or getImageData :

 _imageToDataUri: function (image) { var canvas = window.document.createElement('canvas'); canvas.width = image.width; canvas.height = image.height; var context = canvas.getContext('2d'); context.drawImage(image, 0, 0); return canvas.toDataURL('image/png'); } 

Since the html img element can take as a picture any available resource, including on authorized services and the local file system, the ability to send this content to a third party is a security risk, therefore the pictures do not allow Access-Control-Allow-Origing for Your domain will not be saved. Fortunately, mapnik tiles or tile.openstreetmap.org have an Access-Control-Allow-Origing: * , but for everything to work, you need to set the flag of the img.crossOrigin element to Anonymous .

The operation of CORS in this implementation in all mobile browsers is not guaranteed, so the easiest way is to configure a proxy for your site on your domain or turn off CORS checking for example for Phoengap adherents. Personally, my code did not take off in the default Androids browser (androin 4.0.4 sony xperia active), and in the opera some tiles were kept in a strange way (compare what sometimes happens and what should really be , but it looks like an opera bug ).

Here you can try using WebWorkers + AJAX instead of canvas .

Leaflet


So we need the popular open source JS API, one such candidate is Leaflet .

After a bit of looking at the sources, you can find a tile layer method that is responsible for directly specifying src for tiles:

 _loadTile: function (tile, tilePoint) { tile._layer = this; tile.onload = this._tileOnLoad; tile.onerror = this._tileOnError; tile.src = this.getTileUrl(tilePoint); } 

That is, if you override this class and this method directly to load data into src from storage, then we will do what we need. We also implement adding data to the repository, if they were downloaded from the network and we get full caching.

Implementation for Leaflet
 var StorageTileLayer = L.TileLayer.extend({ _imageToDataUri: function (image) { var canvas = window.document.createElement('canvas'); canvas.width = image.width; canvas.height = image.height; var context = canvas.getContext('2d'); context.drawImage(image, 0, 0); return canvas.toDataURL('image/png'); }, _tileOnLoadWithCache: function () { var storage = this._layer.options.storage; if (storage) { storage.add(this._storageKey, this._layer._imageToDataUri(this)); } L.TileLayer.prototype._tileOnLoad.apply(this, arguments); }, _setUpTile: function (tile, key, value, cache) { tile._layer = this; if (cache) { tile._storageKey = key; tile.onload = this._tileOnLoadWithCache; tile.crossOrigin = 'Anonymous'; } else { tile.onload = this._tileOnLoad; } tile.onerror = this._tileOnError; tile.src = value; }, _loadTile: function (tile, tilePoint) { this._adjustTilePoint(tilePoint); var key = tilePoint.z + ',' + tilePoint.y + ',' + tilePoint.x; var self = this; if (this.options.storage) { this.options.storage.get(key, function (value) { if (value) { self._setUpTile(tile, key, value, false); } else { self._setUpTile(tile, key, self.getTileUrl(tilePoint), true); } }, function () { self._setUpTile(tile, key, self.getTileUrl(tilePoint), true); }); } else { self._setUpTile(tile, key, self.getTileUrl(tilePoint), false); } } }); 


The card itself in this case will be initialized as follows:

 var map = L.map('map').setView([53.902254, 27.561850], 13); new StorageTileLayer('http://{s}.tile.osm.org/{z}/{x}/{y}.png', {storage: storage}).addTo(map); 

We will also add our resources to Application Cache so that the map can work fully without a network with cached tiles:

Application Cache manifest for Leaflet
 CACHE MANIFEST NETWORK: * CACHE: index.html style.css event.js storage.js map.js run.js leaflet.css leaflet.js images/layers.png images/marker-icon.png images/marker-icon@2x.png images/marker-shadow.png 


An example and its githaba code .

Mapbox (modesmaps)


Another candidate open map JS API is the mapbox based on modesmaps .

Having looked at the sources of the mapbox, we will not find anything interesting for us, so let's move on to the sources of modestmaps . Let's start with TemplatedLayer , which is a regular map layer with a template provider, the code that we need will be in the layer class:

 MM.TemplatedLayer = function(template, subdomains, name) { return new MM.Layer(new MM.Template(template, subdomains), null, name); }; 

Having found the use of the template provider in the map layer, you can see that our provider can return either the tile URL or the finished DOM element, with the DOM element being positioned immediately, and the requestManager URL being sent to the requestManager :

 if (!this.requestManager.hasRequest(tile_key)) { var tileToRequest = this.provider.getTile(tile_coord); if (typeof tileToRequest == 'string') { this.addTileImage(tile_key, tile_coord, tileToRequest); } else if (tileToRequest) { this.addTileElement(tile_key, tile_coord, tileToRequest); } } 

 addTileImage: function(key, coord, url) { this.requestManager.requestTile(key, coord, url); } 

 addTileElement: function(key, coordinate, element) { element.id = key; element.coord = coordinate.copy(); this.positionTile(element); } 

The requestManager itself is initialized in the map layer's constructor. Creating an img element img and setting its src occurs in the processQueue method, which also jerks from the map layer:

 processQueue: function(sortFunc) { if (sortFunc && this.requestQueue.length > 8) { this.requestQueue.sort(sortFunc); } while (this.openRequestCount < this.maxOpenRequests && this.requestQueue.length > 0) { var request = this.requestQueue.pop(); if (request) { this.openRequestCount++; var img = document.createElement('img'); img.id = request.id; img.style.position = 'absolute'; img.coord = request.coord; this.loadingBay.appendChild(img); img.onload = img.onerror = this.getLoadComplete(); img.src = request.url; request = request.id = request.coord = request.url = null; } } } 

That is, if we override this method, we will also get the desired result.

Implementation for mapbox (modestmaps)
 var StorageRequestManager = function (storage) { MM.RequestManager.apply(this, []); this._storage = storage; }; StorageRequestManager.prototype._imageToDataUri = function (image) { var canvas = window.document.createElement('canvas'); canvas.width = image.width; canvas.height = image.height; var context = canvas.getContext('2d'); context.drawImage(image, 0, 0); return canvas.toDataURL('image/png'); }; StorageRequestManager.prototype._createTileImage = function (id, coord, value, cache) { var img = window.document.createElement('img'); img.id = id; img.style.position = 'absolute'; img.coord = coord; this.loadingBay.appendChild(img); if (cache) { img.onload = this.getLoadCompleteWithCache(); img.crossOrigin = 'Anonymous'; } else { img.onload = this.getLoadComplete(); } img.onerror = this.getLoadComplete(); img.src = value; }; StorageRequestManager.prototype._loadTile = function (id, coord, url) { var self = this; if (this._storage) { this._storage.get(id, function (value) { if (value) { self._createTileImage(id, coord, value, false); } else { self._createTileImage(id, coord, url, true); } }, function () { self._createTileImage(id, coord, url, true); }); } else { self._createTileImage(id, coord, url, false); } }; StorageRequestManager.prototype.processQueue = function (sortFunc) { if (sortFunc && this.requestQueue.length > 8) { this.requestQueue.sort(sortFunc); } while (this.openRequestCount < this.maxOpenRequests && this.requestQueue.length > 0) { var request = this.requestQueue.pop(); if (request) { this.openRequestCount++; this._loadTile(request.id, request.coord, request.url); request = request.id = request.coord = request.url = null; } } }; StorageRequestManager.prototype.getLoadCompleteWithCache = function () { if (!this._loadComplete) { var theManager = this; this._loadComplete = function(e) { e = e || window.event; var img = e.srcElement || e.target; img.onload = img.onerror = null; if (theManager._storage) { theManager._storage.add(this.id, theManager._imageToDataUri(this)); } theManager.loadingBay.removeChild(img); theManager.openRequestCount--; delete theManager.requestsById[img.id]; if (e.type === 'load' && (img.complete || (img.readyState && img.readyState === 'complete'))) { theManager.dispatchCallback('requestcomplete', img); } else { theManager.dispatchCallback('requesterror', { element: img, url: ('' + img.src) }); img.src = null; } setTimeout(theManager.getProcessQueue(), 0); }; } return this._loadComplete; }; MM.extend(StorageRequestManager, MM.RequestManager); var StorageLayer = function(provider, parent, name, storage) { this.parent = parent || document.createElement('div'); this.parent.style.cssText = 'position: absolute; top: 0px; left: 0px;' + 'width: 100%; height: 100%; margin: 0; padding: 0; z-index: 0'; this.name = name; this.levels = {}; this.requestManager = new StorageRequestManager(storage); this.requestManager.addCallback('requestcomplete', this.getTileComplete()); this.requestManager.addCallback('requesterror', this.getTileError()); if (provider) { this.setProvider(provider); } }; MM.extend(StorageLayer, MM.Layer); var StorageTemplatedLayer = function(template, subdomains, name, storage) { return new StorageLayer(new MM.Template(template, subdomains), null, name, storage); }; 


The card itself in this case will be initialized as follows:

 var map = mapbox.map('map'); map.addLayer(new StorageTemplatedLayer('http://{S}.tile.osm.org/{Z}/{X}/{Y}.png', ['a', 'b', 'c'], undefined, storage)); map.ui.zoomer.add(); map.ui.zoombox.add(); map.centerzoom({lat: 53.902254, lon: 27.561850}, 13); 

We will also add our resources to Application Cache so that the map can work fully without a network with cached tiles:

Application Cache manifest for Mapbox (modestmaps)
 CACHE MANIFEST NETWORK: * CACHE: index.html style.css event.js storage.js map.js run.js mapbox.css mapbox.js map-controls.png 


An example and its githaba code .

Openlayers


And the last candidate of the open JS API maps is OpenLayers .

I had to spend some time to figure out how to run the minimum view, in the end my assembly file acquired the following form:

 [first] [last] [include] OpenLayers/Map.js OpenLayers/Layer/OSM.js OpenLayers/Control/Zoom.js OpenLayers/Control/Navigation.js OpenLayers/Control/TouchNavigation.js [exclude] 

I will use OpenLayers.Layer.OSM , so I’ll start the search from it:

 url: [ 'http://a.tile.openstreetmap.org/${z}/${x}/${y}.png', 'http://b.tile.openstreetmap.org/${z}/${x}/${y}.png', 'http://c.tile.openstreetmap.org/${z}/${x}/${y}.png' ] 

OpenLayers.Layer.OSM inherited from OpenLayers.Layer.XYZ with redefined URLs. The getURL method is interesting here:

 getURL: function (bounds) { var xyz = this.getXYZ(bounds); var url = this.url; if (OpenLayers.Util.isArray(url)) { var s = '' + xyz.x + xyz.y + xyz.z; url = this.selectUrl(s, url); } return OpenLayers.String.format(url, xyz); } 

Also interesting is the getXYZ method, which can be used to create a key:

 getXYZ: function(bounds) { var res = this.getServerResolution(); var x = Math.round((bounds.left - this.maxExtent.left) / (res * this.tileSize.w)); var y = Math.round((this.maxExtent.top - bounds.top) / (res * this.tileSize.h)); var z = this.getServerZoom(); if (this.wrapDateLine) { var limit = Math.pow(2, z); x = ((x % limit) + limit) % limit; } return {'x': x, 'y': y, 'z': z}; } 

OpenLayers.Layer.XYZ itself is inherited from OpenLayers.Layer.Grid , which has the addTile method and which internally creates tiles using tileClass , which is OpenLayers.Tile.Image :

 addTile: function(bounds, position) { var tile = new this.tileClass( this, position, bounds, null, this.tileSize, this.tileOptions ); this.events.triggerEvent("addtile", {tile: tile}); return tile; } 

In OpenLayers.Tile.Image src is set in the setImgSrc method:

 setImgSrc: function(url) { var img = this.imgDiv; if (url) { img.style.visibility = 'hidden'; img.style.opacity = 0; if (this.crossOriginKeyword) { if (url.substr(0, 5) !== 'data:') { img.setAttribute("crossorigin", this.crossOriginKeyword); } else { img.removeAttribute("crossorigin"); } } img.src = url; } else { this.stopLoading(); this.imgDiv = null; if (img.parentNode) { img.parentNode.removeChild(img); } } } 

But it does not specify onload and onerror . The method itself is twitching from initImage , where these handlers are hung:

 initImage: function() { this.events.triggerEvent('beforeload'); this.layer.div.appendChild(this.getTile()); this.events.triggerEvent(this._loadEvent); var img = this.getImage(); if (this.url && img.getAttribute("src") == this.url) { this._loadTimeout = window.setTimeout( OpenLayers.Function.bind(this.onImageLoad, this), 0 ); } else { this.stopLoading(); if (this.crossOriginKeyword) { img.removeAttribute("crossorigin"); } OpenLayers.Event.observe(img, "load", OpenLayers.Function.bind(this.onImageLoad, this) ); OpenLayers.Event.observe(img, "error", OpenLayers.Function.bind(this.onImageError, this) ); this.imageReloadAttempts = 0; this.setImgSrc(this.url); } } 

You can see that the layer's getURL class method, as well as initImage , is jerked from the renderTile :

 renderTile: function() { if (this.layer.async) { var id = this.asyncRequestId = (this.asyncRequestId || 0) + 1; this.layer.getURLasync(this.bounds, function(url) { if (id == this.asyncRequestId) { this.url = url; this.initImage(); } }, this); } else { this.url = this.layer.getURL(this.bounds); this.initImage(); } } 

So, if we override this class, we also get the desired result.

Implementation for OpenLayers
 var StorageImageTile = OpenLayers.Class(OpenLayers.Tile.Image, { _imageToDataUri: function (image) { var canvas = window.document.createElement('canvas'); canvas.width = image.width; canvas.height = image.height; var context = canvas.getContext('2d'); context.drawImage(image, 0, 0); return canvas.toDataURL('image/png'); }, onImageLoadWithCache: function() { if (this.storage) { this.storage.add(this._storageKey, this._imageToDataUri(this.imgDiv)); } this.onImageLoad.apply(this, arguments); }, renderTile: function() { var self = this; var xyz = this.layer.getXYZ(this.bounds); var key = xyz.z + ',' + xyz.y + ',' + xyz.x; var url = this.layer.getURL(this.bounds); if (this.storage) { this.storage.get(key, function (value) { if (value) { self.initImage(key, value, false); } else { self.initImage(key, url, true); } }, function () { self.initImage(key, url, true); }); } else { self.initImage(key, url, false); } }, initImage: function(key, url, cache) { this.events.triggerEvent('beforeload'); this.layer.div.appendChild(this.getTile()); this.events.triggerEvent(this._loadEvent); var img = this.getImage(); this.stopLoading(); if (cache) { OpenLayers.Event.observe(img, 'load', OpenLayers.Function.bind(this.onImageLoadWithCache, this) ); this._storageKey = key; } else { OpenLayers.Event.observe(img, 'load', OpenLayers.Function.bind(this.onImageLoad, this) ); } OpenLayers.Event.observe(img, 'error', OpenLayers.Function.bind(this.onImageError, this) ); this.imageReloadAttempts = 0; this.setImgSrc(url); } }); var StorageOSMLayer = OpenLayers.Class(OpenLayers.Layer.OSM, { async: true, tileClass: StorageImageTile, initialize: function(name, url, options) { OpenLayers.Layer.OSM.prototype.initialize.apply(this, arguments); this.tileOptions = OpenLayers.Util.extend({ storage: options.storage }, this.options && this.options.tileOptions); }, clone: function (obj) { if (obj == null) { obj = new StorageOSMLayer(this.name, this.url, this.getOptions()); } obj = OpenLayers.Layer.Grid.prototype.clone.apply(this, [obj]); return obj; } }); 


The card itself in this case will be initialized as follows:

 var map = new OpenLayers.Map('map'); map.addLayer(new StorageOSMLayer(undefined, undefined, {storage: storage})); var fromProjection = new OpenLayers.Projection('EPSG:4326'); var toProjection = new OpenLayers.Projection('EPSG:900913'); var center = new OpenLayers.LonLat(27.561850, 53.902254).transform(fromProjection, toProjection); map.setCenter(center, 13); 

We will also add our resources to Application Cache so that the map can work fully without a network with cached tiles:

Application Cache manifest for OpenLayers
 CACHE MANIFEST NETWORK: * CACHE: index.html style.css event.js storage.js map.js run.js theme/default/style.css OpenLayers.js 


An example and its githaba code .

I also found a ready-made implementation of the OpenLayers.Control.CacheWrite cache, but only using localStorage , which is not very interesting.

Conclusion


Actually, I got what I wanted. The examples work smartly in chrome or if you have an SSD, there is a tormaz in the fox and the opera while saving to disk, the donkey is not at hand. Brakes also appear in mobile browsers, and even when reading, which upset me a bit, this task is more relevant for mobile devices.

The standard size of the IndexedDB or WebSQL is enough to cache a city or more, which makes the application approach more interesting than in the version with localStorage .

In my examples, you can work with asynchronous storages, but for more comfort, you need to work on their implementation to improve performance compared to what is now.

Source: https://habr.com/ru/post/170129/


All Articles