2GIS for% browser% or how we did the extension
Once we asked ourselves a question: how can we help a user choose a company outside 2gis.ru ? The idea of implementing the idea in the form of a browser extension was proposed almost immediately, and after the research and planning stages, we started developing 2GIS for browsers .
As a basic implementation, we stopped at a button on the side of the address bar, when clicked, the information window opens. Advanced - highlight phone numbers within site content.
')
For greater convenience, it was decided to determine the current coordinates of the user in order to show him reliable information about the distance to the company of interest and give a quick link to the search for directions from the current location to the company. Subsequently, the definition of coordinates became more important, since it was necessary to determine the city in which the user is located in order to show him the data from the most suitable source - 2GIS or Google.
From a technical point of view, an average browser extension is a relatively simple JavaScript application, with one or more entry points.
Regardless of which browser the extension is being written for, there are two groups of scripts inside the application, namely, background and content scripts.
Background scripts (background scripts) are executed in the sandbox (sandboxed evaluation) at the application level, and their data is the same for the running browser instance. They are launched once per session, either when the extension is installed or when the browser is started with the extension installed.
Content scripts , if any, run in the context of the content of the viewed site and have access to the DOM tree, but their data for each window and browser tabs are different. Content scripts are launched each time the onload event of a page loaded into the browser occurs. Background scripts do not have direct access to page content, but browser engines provide mechanisms for interaction between scripts in the background and scripts in the content area. In addition, each engine has its own characteristics in the organization of data exchange.
In addition to background and content scripts, the extension may include other entry points — one for each additional resource. For example, a pop-up info window and one or more pages of extension options. Scripts of additional resources are executed every time they are initialized.
We wanted the extension to work in different browsers, so we paid attention to the framework for cross-browser development. Some of them were provided as SaaS, which did not suit us at all. Others were able to work only with the content area and were more applicable to the implementation of cross-browser user scripts (userscripts). And only two frameworks of all this “diversity” met the requirements. We stopped at the Kango Extensions framework , since it has:
In the free version, which we have chosen, you can create extension packs for Mozilla Firefox, Safari, Google Chrome and its derivatives. The interface is quite limited, common to all browsers, but it is enough to create buttons in the address bar and pop-up windows.
The framework includes:
Most of the 2GIS data for browsers is stored in background scripts and, to avoid unnecessary requests, is cached there too. Among them, for example - the current location of the user and data about organizations obtained by searching by domain or by phone. These organizations are pulling in from 2GIS and Google Places through their API . The location of the user, in turn, is determined either by using the HTML5 Geolocation API, or by using a third-party open GeoIP service, if the API returns an error or is not available.
There are several difficult decisions associated with determining the current location of the user, since there are quite a few possible situations. For example, what will happen if the user closes the lid of the laptop and travels to the other end of the city, and then opens the laptop? And if he just switches between two Wi-Fi points with a short-term loss of connection? Solve the problem in the forehead - through the built-in method navigator.geolocation.watchPosition - failed. The positionChanged event occurred much less frequently than we would like, and besides, it did not work in the situation with closing and opening the lid of the notebook. The navigator.onLine flag came to the rescue. Watching him, we could trigger an event at the time of the appearance of the Internet connection. Unfortunately, the flag only works reliably in webkit browsers, but it’s better than nothing.
If the organization is located in the same city as the user, or in one of the 2GIS cities, we will show the information from 2GIS, as we consider it to be more reliable and verified. To obtain the coordinates of the area about which we have no information, use Google Places, but using filters. Information is displayed only if the domain in the search results matches the domain on which the page is viewed. Similarly, we do and search by phone. The only difference is we filter by phone, not by domain. This is useful if the phone of the organization is placed, for example, on a third-party ad site.
In other words, even if the user is at the other end of the globe relative to the organization on whose site he has entered, we will still try to find and provide correct information about this organization.
The extension works as a regular javascript application: it receives the initial data necessary for the operation, adds handlers to certain events and processes them with their help. Thus, the behavior is realized when switching windows and tabs, for example, changing the indicator, depending on the availability of information on the site opened in this tab.
In addition to switching tabs, we also need to respond to the URL change by the user. To do this, we handle the onBeforeNavigate event, which the browser calls before starting to load the page. A minor difficulty arose with Safari - it causes this event not every time the address bar changes, only if the page is not in the cache or the cache has expired. I had to compromise - to add a handler to the onLoad event, which is called after the page is loaded.
We also faced the problem of possible redirects through response headers, via a meta-tag or via a js-call — software redirects of this kind do not generate additional onBeforeNavigate events. On the other hand, it is also not at all desirable to be attached to onLoad for all browsers, since it would visually look annoyingly slow. As a solution, we chose to periodically poll the status of the URL in the current tab for 30 seconds and update the data if necessary.
In the content area of the page, the extension does a single task — extracting from the content sequences of numbers and symbols that can be considered telephones. This allows you to get information on these phones and call by sending a call to your smartphone.
The task of recognizing phones is in itself non-trivial due to the wide variety of formats, so we did not harbor illusions about a completely reliable solution. Examples of formats that we had to encounter: 34.76.35.05.39 - France, +1 234 345 6789 - USA, 67 2354 9548 - Brazil.
The best solution was a two-stage filter based on regular expressions and simple conditions. At the first stage , sequences of characters that theoretically resemble telephones are distinguished from the text. Relatively non-strict regular expressions and conditions are used for this, besides, the context that can provide additional information is checked for the occurrences. In addition, at the first stage, the long sequences can be divided into short ones, and parts of the sequence with inappropriate context can also be excluded.
The second stage accepts a set of found sequences and, based on the already more stringent rules, decides whether to accept this expression or not. The rules include checks for well-known masks and simple conditions. For example, is the transmitted sequence a string representation of a floating point number.
Because of some strict rules, real numbers are not recognized, but this is corrected by exceptions or an additional context view. Today, a set of 385 test lines, including both positive and negative cases, is used to check the phone's parser. Readiness for production is estimated at 95% threshold of successfully passed tests from the set.
In the course of a simple iterative traversal of the tree width, the phones are searched in the contents of each text DOM node of the document. Step by step it looks like this: the phone found in the text is highlighted → when you hover over it with external services, the company’s request for information about the phone number goes away → the received data is displayed in the pop-up window → if nothing similar is found, the “Call” link is displayed.
Data is received in the background, which does not interfere with the operation of the page, and data caching excludes additional requests on the same phone from other tabs and windows.
If everything is more or less clear with the analysis of the page code at the stage of its loading, what should be done with phones that can be obtained dynamically via ajax or simply formed by client scripts?
In this case, we have provided for the processing of new textual data in a page based on the MutationObserver interface, which is supported by all modern browsers (with the exception of Safari 5 for Windows). To subscribe to events that occur when the DOM changes, we used the mutation-summary library. It allows you to easily and easily get added DOM nodes. To reduce the amount of data processed, we subscribe only to receive new text nodes and try to parse their contents. It is interesting to observe the behavior of the extension in the case of entering the phone in the field while simultaneously displaying it in another text node :
Currently, the extension is available in Russian and English, the choice of language depends on the current locale of the browser and its settings. When translating, we are faced with the fact that from the content and additional scripts, the i18n component can only be reached through an asynchronous background call. This did not fit the idea of the internationalization interface at all. Therefore, a small module was implemented, whose tasks included:
Due to the potential asynchrony, it was necessary to ensure that the locales could be initialized earlier than other objects, since their constructors could contain calls for obtaining localized texts. At the same time, we did not deviate from the locale format adopted by kango, because besides kango.i18n, the locale files are partially used in assembling extension packages.
As for the build itself, the framework provides an easy way to build an extension using the python builder. To build, simply call the kango.py script with the build parameter and specify the directory where the extension files are located. After that, the builder will create the necessary unpacked assemblies and will even package those related to Chrome and Firefox. Unfortunately, due to the complicated mechanism for signing extensions, the out-of-box builder does not know how to package a package for Safari. You can get by assembling the extension through the Safari extension builder by going through the whole procedure of obtaining certificates in the Safari developer center. And you can use the patched xar archiver and your certificates to be able to build the extension automatically and on any * nix-system.
In addition to the kango collector and a simple bash script to automate the build under safari, we also use gulp to check the code style, run tests and a number of auxiliary actions. First, you need to substitute the version number of the current build into several files. Secondly, it would be nice to make sure that the entire debug output is disabled. Finally, we have a need to build special package options. For example, addons.mozilla.org requires the absence of an URL extension for automatic updating, citing the fact that the update is automatically from their site. On the other hand, we publish the extension for Firefox through our own source, so a variant is also required where the URL for the update will be present. Also, a number of changes related to localizations are required to place an extension in the Opera store.
The world does not stand still and 2GIS for browsers is also growing and developing, and here is some of what we are going to do in the near future:
Thanks for attention. Make browser extensions - not only fun, but also useful!
As a basic implementation, we stopped at a button on the side of the address bar, when clicked, the information window opens. Advanced - highlight phone numbers within site content.
')
For greater convenience, it was decided to determine the current coordinates of the user in order to show him reliable information about the distance to the company of interest and give a quick link to the search for directions from the current location to the company. Subsequently, the definition of coordinates became more important, since it was necessary to determine the city in which the user is located in order to show him the data from the most suitable source - 2GIS or Google.
Background and Content Scripts
From a technical point of view, an average browser extension is a relatively simple JavaScript application, with one or more entry points.
Regardless of which browser the extension is being written for, there are two groups of scripts inside the application, namely, background and content scripts.
Background scripts (background scripts) are executed in the sandbox (sandboxed evaluation) at the application level, and their data is the same for the running browser instance. They are launched once per session, either when the extension is installed or when the browser is started with the extension installed.
Content scripts , if any, run in the context of the content of the viewed site and have access to the DOM tree, but their data for each window and browser tabs are different. Content scripts are launched each time the onload event of a page loaded into the browser occurs. Background scripts do not have direct access to page content, but browser engines provide mechanisms for interaction between scripts in the background and scripts in the content area. In addition, each engine has its own characteristics in the organization of data exchange.
In addition to background and content scripts, the extension may include other entry points — one for each additional resource. For example, a pop-up info window and one or more pages of extension options. Scripts of additional resources are executed every time they are initialized.
Framework for creating extensions. Kango Extensions
We wanted the extension to work in different browsers, so we paid attention to the framework for cross-browser development. Some of them were provided as SaaS, which did not suit us at all. Others were able to work only with the content area and were more applicable to the implementation of cross-browser user scripts (userscripts). And only two frameworks of all this “diversity” met the requirements. We stopped at the Kango Extensions framework , since it has:
- lower threshold of entry;
- cross-platform build;
- Russian developer as part of their team :).
In the free version, which we have chosen, you can create extension packs for Mozilla Firefox, Safari, Google Chrome and its derivatives. The interface is quite limited, common to all browsers, but it is enough to create buttons in the address bar and pop-up windows.
The framework includes:
- kango.storage persistent data store, accessible only from background scripts. Interestingly, on different browsers, the storage behaves a little differently. For example, in firefox, data is saved when the extension is removed and reinstalled, and in webkit browsers it is erased when the extension is removed;
- an interface for controlling the kango.browser browser, from which you can not only get the current state of windows and tabs, but also manage them;
- a wrapper for XMLHttpRequest called kango.xhr that solves the problem of creating an instance of XMLHttpRequest in Firefox background scripts;
- kango.console debugging interface, with a single log method for outputting debug messages from scripts in the background sandbox;
- single interface for messaging between background and content scripts. Methods of background scripts are available through kango.invokeAsync, there is also a mechanism for dispatching messages to the content area from the background through the methods KangoBrowserTab.dispatchMessage and kango.addMessageListener. The above methods obtain data asynchronously, so event-based programming will have to be fully applied and monitor potential race conditions.
Used data and external services
Most of the 2GIS data for browsers is stored in background scripts and, to avoid unnecessary requests, is cached there too. Among them, for example - the current location of the user and data about organizations obtained by searching by domain or by phone. These organizations are pulling in from 2GIS and Google Places through their API . The location of the user, in turn, is determined either by using the HTML5 Geolocation API, or by using a third-party open GeoIP service, if the API returns an error or is not available.
There are several difficult decisions associated with determining the current location of the user, since there are quite a few possible situations. For example, what will happen if the user closes the lid of the laptop and travels to the other end of the city, and then opens the laptop? And if he just switches between two Wi-Fi points with a short-term loss of connection? Solve the problem in the forehead - through the built-in method navigator.geolocation.watchPosition - failed. The positionChanged event occurred much less frequently than we would like, and besides, it did not work in the situation with closing and opening the lid of the notebook. The navigator.onLine flag came to the rescue. Watching him, we could trigger an event at the time of the appearance of the Internet connection. Unfortunately, the flag only works reliably in webkit browsers, but it’s better than nothing.
If the organization is located in the same city as the user, or in one of the 2GIS cities, we will show the information from 2GIS, as we consider it to be more reliable and verified. To obtain the coordinates of the area about which we have no information, use Google Places, but using filters. Information is displayed only if the domain in the search results matches the domain on which the page is viewed. Similarly, we do and search by phone. The only difference is we filter by phone, not by domain. This is useful if the phone of the organization is placed, for example, on a third-party ad site.
In other words, even if the user is at the other end of the globe relative to the organization on whose site he has entered, we will still try to find and provide correct information about this organization.
How does it work
The extension works as a regular javascript application: it receives the initial data necessary for the operation, adds handlers to certain events and processes them with their help. Thus, the behavior is realized when switching windows and tabs, for example, changing the indicator, depending on the availability of information on the site opened in this tab.
In addition to switching tabs, we also need to respond to the URL change by the user. To do this, we handle the onBeforeNavigate event, which the browser calls before starting to load the page. A minor difficulty arose with Safari - it causes this event not every time the address bar changes, only if the page is not in the cache or the cache has expired. I had to compromise - to add a handler to the onLoad event, which is called after the page is loaded.
We also faced the problem of possible redirects through response headers, via a meta-tag or via a js-call — software redirects of this kind do not generate additional onBeforeNavigate events. On the other hand, it is also not at all desirable to be attached to onLoad for all browsers, since it would visually look annoyingly slow. As a solution, we chose to periodically poll the status of the URL in the current tab for 30 seconds and update the data if necessary.
Phone Recognition in Web Pages
In the content area of the page, the extension does a single task — extracting from the content sequences of numbers and symbols that can be considered telephones. This allows you to get information on these phones and call by sending a call to your smartphone.
The task of recognizing phones is in itself non-trivial due to the wide variety of formats, so we did not harbor illusions about a completely reliable solution. Examples of formats that we had to encounter: 34.76.35.05.39 - France, +1 234 345 6789 - USA, 67 2354 9548 - Brazil.
The best solution was a two-stage filter based on regular expressions and simple conditions. At the first stage , sequences of characters that theoretically resemble telephones are distinguished from the text. Relatively non-strict regular expressions and conditions are used for this, besides, the context that can provide additional information is checked for the occurrences. In addition, at the first stage, the long sequences can be divided into short ones, and parts of the sequence with inappropriate context can also be excluded.
The second stage accepts a set of found sequences and, based on the already more stringent rules, decides whether to accept this expression or not. The rules include checks for well-known masks and simple conditions. For example, is the transmitted sequence a string representation of a floating point number.
Because of some strict rules, real numbers are not recognized, but this is corrected by exceptions or an additional context view. Today, a set of 385 test lines, including both positive and negative cases, is used to check the phone's parser. Readiness for production is estimated at 95% threshold of successfully passed tests from the set.
In the course of a simple iterative traversal of the tree width, the phones are searched in the contents of each text DOM node of the document. Step by step it looks like this: the phone found in the text is highlighted → when you hover over it with external services, the company’s request for information about the phone number goes away → the received data is displayed in the pop-up window → if nothing similar is found, the “Call” link is displayed.
Data is received in the background, which does not interfere with the operation of the page, and data caching excludes additional requests on the same phone from other tabs and windows.
And if the data is obtained by Ajax?
If everything is more or less clear with the analysis of the page code at the stage of its loading, what should be done with phones that can be obtained dynamically via ajax or simply formed by client scripts?
In this case, we have provided for the processing of new textual data in a page based on the MutationObserver interface, which is supported by all modern browsers (with the exception of Safari 5 for Windows). To subscribe to events that occur when the DOM changes, we used the mutation-summary library. It allows you to easily and easily get added DOM nodes. To reduce the amount of data processed, we subscribe only to receive new text nodes and try to parse their contents. It is interesting to observe the behavior of the extension in the case of entering the phone in the field while simultaneously displaying it in another text node :
Internationalization. We speak English
Currently, the extension is available in Russian and English, the choice of language depends on the current locale of the browser and its settings. When translating, we are faced with the fact that from the content and additional scripts, the i18n component can only be reached through an asynchronous background call. This did not fit the idea of the internationalization interface at all. Therefore, a small module was implemented, whose tasks included:
- definition of the current locale;
- loading of localization lists depending on the locale, either through a direct call to kango.io.getExtensionFileContents, or asynchronous if the module is instantiated in content or in additional scripts;
- providing a simple interface for receiving localization messages with the simplest line interpolation.
Due to the potential asynchrony, it was necessary to ensure that the locales could be initialized earlier than other objects, since their constructors could contain calls for obtaining localized texts. At the same time, we did not deviate from the locale format adopted by kango, because besides kango.i18n, the locale files are partially used in assembling extension packages.
Build Packages
As for the build itself, the framework provides an easy way to build an extension using the python builder. To build, simply call the kango.py script with the build parameter and specify the directory where the extension files are located. After that, the builder will create the necessary unpacked assemblies and will even package those related to Chrome and Firefox. Unfortunately, due to the complicated mechanism for signing extensions, the out-of-box builder does not know how to package a package for Safari. You can get by assembling the extension through the Safari extension builder by going through the whole procedure of obtaining certificates in the Safari developer center. And you can use the patched xar archiver and your certificates to be able to build the extension automatically and on any * nix-system.
In addition to the kango collector and a simple bash script to automate the build under safari, we also use gulp to check the code style, run tests and a number of auxiliary actions. First, you need to substitute the version number of the current build into several files. Secondly, it would be nice to make sure that the entire debug output is disabled. Finally, we have a need to build special package options. For example, addons.mozilla.org requires the absence of an URL extension for automatic updating, citing the fact that the update is automatically from their site. On the other hand, we publish the extension for Firefox through our own source, so a variant is also required where the URL for the update will be present. Also, a number of changes related to localizations are required to place an extension in the Opera store.
What's next
The world does not stand still and 2GIS for browsers is also growing and developing, and here is some of what we are going to do in the near future:
- iterative improvement of the phone recognizer. Add more test cases, fix the parser - life becomes better and more fun;
- introducing an interactive 2GIS map into a pop-up window instead of a static picture;
- cosmetic edits that allow our elements to look great even on very poorly laid out sites;
- and of course a variety of fixes and improvements, suggestions for which we are waiting for mail extension@2gis.ru
Thanks for attention. Make browser extensions - not only fun, but also useful!
Source: https://habr.com/ru/post/225807/
All Articles