AJAX Client Storage Study - From Dojo Surgery to Fullajax Storage's own implementation
With the intensive development of web applications, storing large amounts of data on the client side is becoming increasingly popular. This does not just simplify the application — often by using client data storage, you can significantly extend the functionality of the application being created, allowing it to do something that otherwise would not be possible.
At the same time, it should not be taken as a “whim” of developers of some special and exotic web services - according to the latest data, even such a grandeur among the mass Western social services, like the MySpace network, will apply the solution from Google - Google Gears (more details - gearsblog. blogspot.com/2008/05/myspace-message-center-is-now-searching.html ). The network has a number of publications on the use of local repositories, for example, the following publications that we studied in preparing the material can be noted:
browserpersistence.ru
abrdev.com/?tag=storage
pablotron.org/software/persist-js
If you do not have a clue what a local repository is, or you are not very familiar with this, you will find comprehensive information on storages and its areas of application using the links above.
Having explored all possible options for implementing local data warehouses, the most attractive at this time is Flash-based storage. Notice that we are talking about persistent data storages, that is, saving any data there, we expect to be able to access it on the client computer after the page is closed in the browser, and updated, and even after the computer is restarted. This article will analyze the implementation of Flash-based storage.
Let's look at the clear advantages of Flash Storage:
Some developers tend to attribute the disadvantage to this method that users allegedly block Flash because of banners, but I think this problem is more contrived. If so, then they will turn off the graphics, and the scripts - which means, as they say in one maxim, “Buratino themselves” - Flash is now often an integral part of many sites and the only acceptable technology for implementing multimedia in the web environment.
It should be noted that among all the Flash storages analyzed by us, the most attractive in terms of functionality turned out to be the stack that is included in the Dojo library (dojotoolkit.org). Moreover, by and large, this is the first industrial implementation of this technology.
Distinctive advantages:
')
Of course, this solution, for all its merits, is not without flaws:
Test example download.dojotoolkit.org/release-1.1.0/dojo-release-1.1.0/dojox/storage/tests/test_storage.html?forceStorageProvider=dojox.storage.FlashStorageProvider loads 13 (!) Scripts. All of them are loaded using AJAX, and even with explicit instructions to use Flash Storage, a bunch of additional files are still being loaded that are not needed for this storage to work.
The simplest example of helloworld is download.dojotoolkit.org/release-1.1.0/dojo-release-1.1.0/dojox/storage/demos/helloworld.html loads at least a few JS scripts. Therefore, we can conclude that if your application does not use any Dojo Toolkit functionality, using it for implementation only of Storage is rather resource-intensive and inefficient.
Since the “root of the shortcomings” is the same, we decided to get rid of these shortcomings, or at least minimize their influence. After downloading the Dojo 1.1 distribution, I had to sit down to analyze its architecture in order to isolate Flash Storage only. This is where the first difficulty turned out to be. Subjectively, Dojo's internal architecture is very nontrivial and complex (well, of course, because this is the most powerful framework from all, probably implemented in JS, using several programming paradigms, for example, recently added a module to implement aspect-oriented programming). Therefore, it was not possible to quickly and easily separate Flash Storage. But the desire to eliminate these shortcomings turned out to be more of the difficulties encountered.
The first stage of work took several hours of work, and this version (without changing the internal architecture of the code) weighed more than 100 KB (the Dojo core + the Storage module). Having made an in-depth analysis, we realized that there was still a lot of excess left. The second stage, in order to shrink this super-minimized version even more, took several days and this implementation takes only 15 Kb, which was at the cost of a huge number of edits to the Dojo Toolkit source code.
Even more detailed tests of Flash Storage showed another advantage: the storage is the same for all browsers on a computer, i.e. The data saved using Firefox is easily obtained from Internet Explorer and vice versa.
An additional flaw was also identified: it is necessary to wait for Flash Storage to initialize. That is, the use of storage is possible only in asynchronous mode, after the flash movie is loaded. Accordingly, web applications using Flash Storage need to be designed with this feature in mind.
But the ability to simply store some data on the client and get it for the application to work is not the only functionality. I would very much like to be able to control the data in such a cache and automatically update it from the server.
When using local storage, a very important point is the implementation of the data update algorithm on the client side. We offer you three different options for implementation.
Global versioning site method
The algorithm of this method is similar to the implementation of SVN. The site has a global version number. Any changes to individual content or a group change of content increases the version number. When the site is initially loaded, the version number on the client side is sent to the server. The server checks the sent version on the client with the current version on the server and, if there are differences, the list of changed content identifiers-keys is generated in response to the difference between the client and server versions of the site. In other words, when content changes on the server, a list of content keys is sent to the client, which must be deleted in the repository. This list corresponds to the difference in changes between the client site version and the current version on the server. At the same time, the content itself is not sent to the client. Content is sent only when the client side requests specific data and, upon successful loading, enters the repository.
The original image of the algorithm fullajax.ru/doc/storage/Storage-Algorithm-1.jpg
The considered algorithm reduces the number of connections to the server and the total amount of information transmitted. This method is convenient to use with a large number of content units.
The method of versioning individual content units
The algorithm is based on identifying the uniqueness of each individual content item specifically. The bottom line is that during the initial loading of the site, a list of value pairs {key: novelty identifier} comes from the server. The client script checks the coincidence of the identifiers of novelty in the repository. If there are differences, the old values ​​of the novelty identifiers and the data corresponding to the updated content key are deleted from the storage. Immediately in the repository recorded new values ​​of identifiers of novelty. As in the previously discussed method, the content is sent to the client only when the client side requests specific data and, with a successful download, enters the repository.
The original image of the algorithm fullajax.ru/doc/storage/Storage-Algorithm-2.png
The considered algorithm reduces the number of connections to the server and the total amount of information transmitted. This algorithm is simple to implement and its use is effective with a small amount of content units. With a large number of content units, the list of value pairs {key: novelty identifier} may be too large, which reduces the effectiveness of this algorithm.
Standard Method for Using Etag
The method is based on sending the content identifier to the Etag server. If the sent identifier on the client matches the current server identifier, the server responds with a response of 304 Not modified and the client takes data from the storage. When identifiers are different, the server responds with new content. The client has successfully uploaded the new content, replaces the old data in the storage with new ones and writes its etag with the novelty identifier.
The considered algorithm reduces only the total amount of information transmitted. The method is simple to implement, but in terms of the number of connections to the server, it is less efficient than the previously discussed algorithms.
The choice of using one approach or another depends on the specific situation.
Application examples
Using Flash Storage is convenient not only for storing configuration data and web application objects, but also for storing scripts, styles, images, and even the content itself. An example of the use of storage is implemented on the site fullajax.ru already known to you. On this site, for the update of data on the client side, the aforementioned “Method of Versioning Separate Content” has been applied . Scripts and content are saved in the storage, i.e. once having downloaded the page of this site, the subsequent times it gets from the repository until a newer version of the page appears on the server.
AJAX + Storage is particularly effective. The Fullajax library includes a module for implementing Flash Storage. For those users who already use the Fullajax library, it is enough to connect an additional script and when requesting data using AJAX (dax or hax methods), the data will automatically get into the repository. It only remains to take care of the mechanism for updating data on the client side.
Of course, this implementation is not the only one. And in many cases, it will be very lacking of certain functionalities. I will not hide - indeed the best option now will be the built-in database from Google Gears, but this solution is quite exclusive and can be effectively used only in special cases of web applications. Well, for mass use, including on ordinary websites, it is really best to use either the solution described above or one of the universal ones, for example, Browserpersistense or PersistJS.
The author of the material: Ruslan Sinitsky (sirus, fullajax.ru ), co-author - Alexander Lozovyuk (aleks_raiden, abrdev.com ).
At the same time, it should not be taken as a “whim” of developers of some special and exotic web services - according to the latest data, even such a grandeur among the mass Western social services, like the MySpace network, will apply the solution from Google - Google Gears (more details - gearsblog. blogspot.com/2008/05/myspace-message-center-is-now-searching.html ). The network has a number of publications on the use of local repositories, for example, the following publications that we studied in preparing the material can be noted:
browserpersistence.ru
abrdev.com/?tag=storage
pablotron.org/software/persist-js
If you do not have a clue what a local repository is, or you are not very familiar with this, you will find comprehensive information on storages and its areas of application using the links above.
Having explored all possible options for implementing local data warehouses, the most attractive at this time is Flash-based storage. Notice that we are talking about persistent data storages, that is, saving any data there, we expect to be able to access it on the client computer after the page is closed in the browser, and updated, and even after the computer is restarted. This article will analyze the implementation of Flash-based storage.
Let's look at the clear advantages of Flash Storage:
- Maximum cross-browser (for obvious reasons, we do not consider exotic versions of browsers that do not support the usual web technologies).
- Maximum prevalence: Flash installed on more than ~ 95% of browsers.
- The ability to store a virtually unlimited amount of data (although with some reservations, for example, that the user must explicitly allow storage of the specified volume or establish permission to store an unlimited amount of information).
Some developers tend to attribute the disadvantage to this method that users allegedly block Flash because of banners, but I think this problem is more contrived. If so, then they will turn off the graphics, and the scripts - which means, as they say in one maxim, “Buratino themselves” - Flash is now often an integral part of many sites and the only acceptable technology for implementing multimedia in the web environment.
It should be noted that among all the Flash storages analyzed by us, the most attractive in terms of functionality turned out to be the stack that is included in the Dojo library (dojotoolkit.org). Moreover, by and large, this is the first industrial implementation of this technology.
Distinctive advantages:
- Ability to store JS-objects, automatic serialization and deserialization (of course, in JSON format).
- Some bugs of the Flash ExternalInterface operation are taken into account.
- Powerful and versatile data warehouse API.
- Built-in support for the namespace for stored data, work with multiple repositories simultaneously.
- PENDING status.
- When the current storage size limit is exceeded - automatic display of the permission request dialog to increase the size.
')
Of course, this solution, for all its merits, is not without flaws:
- Large distribution volume (meaning that if you use Dojo only for the sake of the Storage system).
- When initializing the repository, download extra code that may not be necessary when the application is running.
- Crash when running a script from file: // local file system
Test example download.dojotoolkit.org/release-1.1.0/dojo-release-1.1.0/dojox/storage/tests/test_storage.html?forceStorageProvider=dojox.storage.FlashStorageProvider loads 13 (!) Scripts. All of them are loaded using AJAX, and even with explicit instructions to use Flash Storage, a bunch of additional files are still being loaded that are not needed for this storage to work.
The simplest example of helloworld is download.dojotoolkit.org/release-1.1.0/dojo-release-1.1.0/dojox/storage/demos/helloworld.html loads at least a few JS scripts. Therefore, we can conclude that if your application does not use any Dojo Toolkit functionality, using it for implementation only of Storage is rather resource-intensive and inefficient.
Since the “root of the shortcomings” is the same, we decided to get rid of these shortcomings, or at least minimize their influence. After downloading the Dojo 1.1 distribution, I had to sit down to analyze its architecture in order to isolate Flash Storage only. This is where the first difficulty turned out to be. Subjectively, Dojo's internal architecture is very nontrivial and complex (well, of course, because this is the most powerful framework from all, probably implemented in JS, using several programming paradigms, for example, recently added a module to implement aspect-oriented programming). Therefore, it was not possible to quickly and easily separate Flash Storage. But the desire to eliminate these shortcomings turned out to be more of the difficulties encountered.
The first stage of work took several hours of work, and this version (without changing the internal architecture of the code) weighed more than 100 KB (the Dojo core + the Storage module). Having made an in-depth analysis, we realized that there was still a lot of excess left. The second stage, in order to shrink this super-minimized version even more, took several days and this implementation takes only 15 Kb, which was at the cost of a huge number of edits to the Dojo Toolkit source code.
Even more detailed tests of Flash Storage showed another advantage: the storage is the same for all browsers on a computer, i.e. The data saved using Firefox is easily obtained from Internet Explorer and vice versa.
An additional flaw was also identified: it is necessary to wait for Flash Storage to initialize. That is, the use of storage is possible only in asynchronous mode, after the flash movie is loaded. Accordingly, web applications using Flash Storage need to be designed with this feature in mind.
But the ability to simply store some data on the client and get it for the application to work is not the only functionality. I would very much like to be able to control the data in such a cache and automatically update it from the server.
When using local storage, a very important point is the implementation of the data update algorithm on the client side. We offer you three different options for implementation.
Global versioning site method
The algorithm of this method is similar to the implementation of SVN. The site has a global version number. Any changes to individual content or a group change of content increases the version number. When the site is initially loaded, the version number on the client side is sent to the server. The server checks the sent version on the client with the current version on the server and, if there are differences, the list of changed content identifiers-keys is generated in response to the difference between the client and server versions of the site. In other words, when content changes on the server, a list of content keys is sent to the client, which must be deleted in the repository. This list corresponds to the difference in changes between the client site version and the current version on the server. At the same time, the content itself is not sent to the client. Content is sent only when the client side requests specific data and, upon successful loading, enters the repository.
The original image of the algorithm fullajax.ru/doc/storage/Storage-Algorithm-1.jpg
The considered algorithm reduces the number of connections to the server and the total amount of information transmitted. This method is convenient to use with a large number of content units.
The method of versioning individual content units
The algorithm is based on identifying the uniqueness of each individual content item specifically. The bottom line is that during the initial loading of the site, a list of value pairs {key: novelty identifier} comes from the server. The client script checks the coincidence of the identifiers of novelty in the repository. If there are differences, the old values ​​of the novelty identifiers and the data corresponding to the updated content key are deleted from the storage. Immediately in the repository recorded new values ​​of identifiers of novelty. As in the previously discussed method, the content is sent to the client only when the client side requests specific data and, with a successful download, enters the repository.
The original image of the algorithm fullajax.ru/doc/storage/Storage-Algorithm-2.png
The considered algorithm reduces the number of connections to the server and the total amount of information transmitted. This algorithm is simple to implement and its use is effective with a small amount of content units. With a large number of content units, the list of value pairs {key: novelty identifier} may be too large, which reduces the effectiveness of this algorithm.
Standard Method for Using Etag
The method is based on sending the content identifier to the Etag server. If the sent identifier on the client matches the current server identifier, the server responds with a response of 304 Not modified and the client takes data from the storage. When identifiers are different, the server responds with new content. The client has successfully uploaded the new content, replaces the old data in the storage with new ones and writes its etag with the novelty identifier.
The considered algorithm reduces only the total amount of information transmitted. The method is simple to implement, but in terms of the number of connections to the server, it is less efficient than the previously discussed algorithms.
The choice of using one approach or another depends on the specific situation.
Application examples
Using Flash Storage is convenient not only for storing configuration data and web application objects, but also for storing scripts, styles, images, and even the content itself. An example of the use of storage is implemented on the site fullajax.ru already known to you. On this site, for the update of data on the client side, the aforementioned “Method of Versioning Separate Content” has been applied . Scripts and content are saved in the storage, i.e. once having downloaded the page of this site, the subsequent times it gets from the repository until a newer version of the page appears on the server.
AJAX + Storage is particularly effective. The Fullajax library includes a module for implementing Flash Storage. For those users who already use the Fullajax library, it is enough to connect an additional script and when requesting data using AJAX (dax or hax methods), the data will automatically get into the repository. It only remains to take care of the mechanism for updating data on the client side.
Of course, this implementation is not the only one. And in many cases, it will be very lacking of certain functionalities. I will not hide - indeed the best option now will be the built-in database from Google Gears, but this solution is quite exclusive and can be effectively used only in special cases of web applications. Well, for mass use, including on ordinary websites, it is really best to use either the solution described above or one of the universal ones, for example, Browserpersistense or PersistJS.
The author of the material: Ruslan Sinitsky (sirus, fullajax.ru ), co-author - Alexander Lozovyuk (aleks_raiden, abrdev.com ).
Source: https://habr.com/ru/post/26784/
All Articles