IndexedDB - unlimited data storage
Good afternoon, dear community.
For those who do not know what IndexedDB is and what it is eaten with, you can read here .
And we go further.
In the office in which I work, it became necessary to use an indexed local database on the client side and the choice immediately fell on IndexedDB.
')
But as always there is one “BUT”, this is the “BUT” - the limitation of the size of the database on the user's machine in the amount of 5 MB, which did not suit us at all. Since this technology was planned to be used in the admin panel of our project and all users used Google Chrome as the default browser, it was decided to search for a circumvention of the restriction through the proxy extension. Shooting a lot of information, we concluded that the limit on the size of the database can be removed using special flags in the manifest of our extension:
We go further. We dealt with unlimited data storage, but now it became necessary to work with that unlimited database directly from the site itself. For this purpose, sending messages between the site and the extension was used (the extension acted as a proxy, between the site and the unlimited database). To this end, the following flags were added to the manifest of our extension:
It turned out that the following URLs are considered valid: *: //google.com/* and http: //*.chromium.org/*, and, http: // * / *, *: / / *. Com / are not.
More information about externally_connectable can be found here .
We go further.
The stage of writing the very “bridge” between the site and the extension to access the database has begun.
The main library for working with IndexedDB on the extension side was db.js, which you can find here .
In order not to reinvent the wheel, it was decided to use the access syntax on the site side, which is implemented in db.js.
And so we went, we create background.js, which we will listen to incoming messages, and respond to them. The code listing is below:
But here we were in for a surprise, namely, on the execution of the code segment:
we get an exception:
To resolve this problem, add another line to the manifest, which allows user js to run on the extension side.
We also had to implement the auxiliary function of driving an object to an array, in order to pass it as function arguments.
Full listing manifest.json
With the extension figured out, now let's start writing a client library to work with our proxy extension.
The first thing you need to do when sending a message from a client is to indicate to which extension we want to send it; for this, we specify its id:
Full listing of the client library:
At this stage, our complex for working with unlimited indexDB is ready. Below are examples of use.
At the moment, this decision is actively used on one of our projects, I will be grateful for constructive criticism and suggestions. Since this is my first article on Habré, please do not judge much.
You can familiarize with source codes on github .
For those who do not know what IndexedDB is and what it is eaten with, you can read here .
And we go further.
Unlimited
In the office in which I work, it became necessary to use an indexed local database on the client side and the choice immediately fell on IndexedDB.
')
But as always there is one “BUT”, this is the “BUT” - the limitation of the size of the database on the user's machine in the amount of 5 MB, which did not suit us at all. Since this technology was planned to be used in the admin panel of our project and all users used Google Chrome as the default browser, it was decided to search for a circumvention of the restriction through the proxy extension. Shooting a lot of information, we concluded that the limit on the size of the database can be removed using special flags in the manifest of our extension:
"permissions": [ "unlimitedStorage", "unlimited_storage" ], Posting site extension site
We go further. We dealt with unlimited data storage, but now it became necessary to work with that unlimited database directly from the site itself. For this purpose, sending messages between the site and the extension was used (the extension acted as a proxy, between the site and the unlimited database). To this end, the following flags were added to the manifest of our extension:
"externally_connectable": { "matches": [ "*://localhost/*", "____URL " ] } It turned out that the following URLs are considered valid: *: //google.com/* and http: //*.chromium.org/*, and, http: // * / *, *: / / *. Com / are not.
More information about externally_connectable can be found here .
We go further.
The stage of writing the very “bridge” between the site and the extension to access the database has begun.
The main library for working with IndexedDB on the extension side was db.js, which you can find here .
In order not to reinvent the wheel, it was decided to use the access syntax on the site side, which is implemented in db.js.
Expansion
And so we went, we create background.js, which we will listen to incoming messages, and respond to them. The code listing is below:
var server; chrome.runtime.onMessageExternal.addListener( function (request, sender, sendResponse) { var cmd = request.cmd, params = request.params; try { switch (cmd) { case "getUsageAndQuota": navigator.webkitPersistentStorage.queryUsageAndQuota(function(u,q){ sendResponse({"usage": u,"quota":q}); }); break; case "open": db.open(params).done(function (s) { server = s; var exclude = "add close get query remove update".split(" "); var tables = new Array(); for(var table in server){ if(exclude.indexOf(table)==-1){ tables.push(table); } } sendResponse(tables); }); break; case "close": server.close(); sendResponse({}); break; case "get": server[request.table].get(params).done(sendResponse) break; case "add": server[request.table].add(params).done(sendResponse); break; case "update": server[request.table].update(params).done(sendResponse); break; case "remove": server[request.table].remove(params).done(sendResponse); break; case "execute": var tmp_server = server[request.table]; var query = tmp_server.query.apply(tmp_server, obj2arr(request.query)); var flt; for (var i = 0; i < request.filters.length; i++) { flt = request.filters[i]; if (flt.type == "filter") { flt.args = new Function("item", flt.args[0]); } query = query[flt.type].apply(query, obj2arr(flt.args)); } query.execute().done(sendResponse); break; } } catch (error) { if (error.name != "TypeError") { sendResponse({RUNTIME_ERROR: error}); } } return true; }); But here we were in for a surprise, namely, on the execution of the code segment:
flt.args = new Function("item", flt.args[0]); we get an exception:
Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' chrome-extension-resource:". .To resolve this problem, add another line to the manifest, which allows user js to run on the extension side.
"content_security_policy": "script-src 'self' 'unsafe-eval'; object-src 'self'" We also had to implement the auxiliary function of driving an object to an array, in order to pass it as function arguments.
var obj2arr = function (obj) { if (typeof obj == 'object') { var tmp_args = new Array(); for (var k in obj) { tmp_args.push(obj[k]); } return tmp_args; } else { return [obj]; } } Full listing manifest.json
{ "manifest_version": 2, "name": "exDB", "description": "This extension give proxy access to indexdb from page.", "version": "1.0", "background": { "page": "background.html" }, "content_security_policy": "script-src 'self' 'unsafe-eval'; object-src 'self'", "externally_connectable": { "matches": [ "*://localhost/*" ] }, "permissions": [ "unlimitedStorage", "unlimited_storage" ], "icons": { "16": "icons/icon_016.png", "48": "icons/icon_048.png" } } Customer
With the extension figured out, now let's start writing a client library to work with our proxy extension.
The first thing you need to do when sending a message from a client is to indicate to which extension we want to send it; for this, we specify its id:
chrome.runtime.sendMessage("ID_", data, callback); Full listing of the client library:
(function (window, undefined) { "use strict"; function exDB() { var self = this; this.extensionId = arguments[0] || "eojllnbjkomphhmpcpafaipblnembfem"; this.filterList = new Array(); this._table; this._query; self.sendMessage = function sendMessage(data, callback) { chrome.runtime.sendMessage(self.extensionId, data, callback); }; self.open = function (params, callback) { self.sendMessage({"cmd": "open", "params": params}, function(r){ var tn; for(var i=0;i< r.length;i++) tn = r[i]; self.__defineGetter__(tn,function(){ self._table = tn; return this; }); callback(); }); return self; }; self.close = function (callback) { self.sendMessage({"cmd": "close", "params": {}}, callback); return self; } self.table = function (name) { self._table = name; return self; }; self.query = function () { self._query = arguments; return self; }; self.execute = function (callback) { self.sendMessage({"cmd": "execute", "table": self._table, "query": self._query, "filters": self.filterList}, function (result) { if (result && result.RUNTIME_ERROR) { console.error(result.RUNTIME_ERROR.message); result = null; } callback(result); }); self._query = null; self.filterList = []; }; self.getUsageAndQuota = function(callback){ self.sendMessage({"cmd": "getUsageAndQuota"},callback); }; "add update remove get".split(" ").forEach(function (fn) { self[fn] = function (item, callback) { self.sendMessage({"cmd": fn, "table": self._table, "params": item}, function (result) { if (result && result.RUNTIME_ERROR) { console.error(result.RUNTIME_ERROR.message); result = null; } callback(result); }); return self; } }); "all only lowerBound upperBound bound filter desc distinct keys count".split(" ").forEach(function (fn) { self[fn] = function () { self.filterList.push({type: fn, args: arguments}); return self; } }); } window.exDB = exDB; })(window, undefined); At this stage, our complex for working with unlimited indexDB is ready. Below are examples of use.
Connection
var db = new exDB(); db.open({ server: 'my-app', version: 1, schema: { people: { key: { keyPath: 'id', autoIncrement: true }, // Optionally add indexes indexes: { firstName: { }, answer: { unique: true } } } } }, function () {}); Connection closure
db.close(); Adding record
db.table("people").add({ firstName: 'Aaron', lastName: 'Powell', answer: 142},function(r){ }); Record Update
db.table("people").update({ id:1, firstName: 'Aaron', lastName: 'Powell', answer: 1242}, function (r) {}); Deleting records by ID
db.table("people").remove(1,function(key){}); Getting record by ID
db.table("people").get(1,function(r){ console.log(r); }); Sampling / Sorting
db.people.query("firstName").only("Aaron2").execute(function(r){ console.log("GETTER",r); }); db.table("people").query("answer").all().desc().execute(function(r){ console.log("all",r); }); db.table("people").query("answer").only(12642).count().execute(function(r){ console.log("only",r); }); db.table("people").query("answer").bound(20,45).execute(function(r){ console.log("bound",r); }); db.table("people").query("answer").lowerBound(50).keys().execute(function(r){ console.log("lowerBound",r); }); db.table("people").query("answer").upperBound(43).execute(function(r){ console.log("upperBound",r); }); db.table("people").query("answer").filter("return item.answer==42 && item.firstName=='Aaron'").execute(function(r){ console.log("filter",r); }); Obtaining database size (used / maximum size)
db.getUsageAndQuota(function(r){ console.log("used", r.usage); //bytes console.log("quota", r.quota); //bytes }); Conclusion
At the moment, this decision is actively used on one of our projects, I will be grateful for constructive criticism and suggestions. Since this is my first article on Habré, please do not judge much.
You can familiarize with source codes on github .
Source: https://habr.com/ru/post/198666/
All Articles