In this article, I complete a cycle of publications in which I wanted to tell about my experience of writing a web browser extension. I already had the experience of creating a web extension, which was installed by about 100,000 Chrome users who worked autonomously, but in this series of articles I decided to delve into the process of developing a web extension by tightly integrating it with the server part.







Part 1 , Part 2 , Part 3

Scheduled Tasks

The web extension allows you to configure a custom script to automatically execute its code after the page has finished loading. This is convenient, for example, when you have a list of similar pages to crawl and retrieve data from the execution of the script. Or, if it is necessary to remove annoying ads on any page on the Internet, code to track your actions on a web resource, change the page background to dark, etc.

')

At the same time, it is often necessary to receive data from the page in the daily schedule mode. For example, if currency exchange rates are placed on the site, and a direct request by URL protects the hashed argument. As such a copy can be considered the site of the Central Bank of Europe . In this case, to get current exchange rates, you must know the hashed URL to get the correct XML data.



Using the web extension, you can set the required frequency for receiving data through the execution of a script to request exchange rates. The web extension accepts a line in cron format for input, therefore, to receive currency rates on a daily basis, * * / 1 * * * must be specified.



From a technical point of view, the server side launches Puppeteer with the addition of a custom script to the Chromium browser page. In this case, we are faced with the problem of deliberately shutting down the Chromium process over time, as an increase in the number of processes will adversely affect the overall performance of the server side. In the case of planning to run a user script in periodic mode, to solve the above problem, you must send a request from the user script to Puppeteer to complete the Chromium process.



After long tests of the task scheduler implementation process, it was decided to track the script console output. Thus, in order to stop the Chromium process, the user script must have a command to stop the process in the required code section:

console.log('script is ended'); 

Using this command, a custom script can close the Chromium server process spawned by the Puppeteer. This is implemented by tracking the Puppeteer “console” event:

 //initialize browser and page page.on('console', async msg => { for (let i = 0; i < msg.args().length; ++i) { if(await msg.args()[i].jsonValue() == 'script is ended') { await browser.close(); } } }); 

If the user script does not have such a command, it is necessary to force the completion of such a script by timer to complete the process in the task scheduler mode. This is implemented in a simple way with setTimeout:

 const timeLimitCron = 30; const timeout = setTimeout(async () => { if(browser) { await browser.close(); } clearTimeout(timeout); }, timeLimitCron * 1000 ); //time limit for cron, then forced closing, default 30 seconds 

Since the task scheduler works on the server side, and the user often needs to use a specific IP address, the option to use a proxy server has been added.







The user can also download the latest message log from the Puppeteer console. For example, it is convenient to check the correct operation of the proxy server.

Hot Keys for Custom Scripts

During testing and field trials, it turned out that for some types of scripts it is useful to have shortcuts for their execution on web resources. An example of such a script is Redability.js. This script creates a “clean view” for reading the contents of the site. That is, the js library analyzes the structure of the page with the content and allows you to get, with a high degree of probability, the title, author and content of the page. After that, the user script can overwrite the html of the web resource and allow the user to read in “pure form”, without unnecessary markup, advertising, etc.



Initially, launching a custom script was only possible from the pop-up web extension by clicking on the “Execute” button. This interface logic is often justified by security considerations. But in the example above, it does not allow you to easily bring the content of a web resource to a “pure mind”.



As stated above, the web extension allows you to work with the DOM through content.js, but the window object cannot be used, for example, to store parameters, since it is not the window object of the open tab. This condition limits the operation of the web extension as an object for tracking keystrokes.



In the problem being solved, it is necessary to transfer “hot keys” from the web extension interface to the server side for storage. Then, each time the web resource page loads, get a list of “hot keys” and load the user script after pressing the right combination. Hotkeys.js is used as a library for working with hot keys.



After receiving the list of “hot keys” from the server part, the following code is executed:

 var hotKeysMap = {keys: [], id: []}; for(var i in data.response.data) { hotKeysMap.keys.push(data.response.data[i]['hotkeys']); hotKeysMap.id.push(data.response.data[i]["_id"]); } if(hotKeysMap.keys) { getScript("hotkeys.js", function() { var script = document.createElement("script"); script.setAttribute("class", "gCore-hotKeys"); script.setAttribute("data-endpoint", event.data.endPoint); script.setAttribute("data-token", event.data.token); script.setAttribute("data-userid", event.data.userId); script.innerHTML = "GChotKeys = JSON.parse(\"" + JSON.stringify(hotKeysMap).replace(/"/g, "\\\"") + "\");\n" + "hotkeys(GChotKeys.keys.join(','), function(event, handler) {" + " event.preventDefault();" + " localStorage.setItem('runHotKeysScript', GChotKeys.id[GChotKeys.keys.indexOf(handler.key)]);" + "});"; document.body.appendChild(script); }); } 

The getScript function generates html code for the script tag and writes it to the web resource page. Thus, we have on each page the ability to track keystrokes. We also need to pass an array of hot key matching with the id of the script to execute. This is implemented by adding html code for the script tag, the content of which initiates a global variable for storing the matching array in JSON format.



It has already been mentioned above that there is a message problem between the open page of the web resource and the web content extension.js script that is used to manipulate the DOM. In this case, you can resort to a simple technique of checking the value in localStorage, which is a common object for the two interaction points mentioned above.



In content.js, you can just once a second check the localStorage value and perform the same DOM manipulations as when you click on the “Execute” button in the pop-up web extension.

 setInterval(function() { if(localStorage.getItem('runHotKeysScript')) { // run user script localStorage.removeItem('runHotKeysScript'); } }, 1000); 

Thus, the technique of “hot keys” is implemented, which allows you to quickly run custom scripts, using the power of the internal library for the needs of solving practical problems.

Conclusion

In the process of implementing this project, practical tasks of writing integration between the server part based on meteor.js and the cross-browser web extension were solved. The key stumbling blocks were the SCP and the interaction processes between the three components of the client side — the browser open page, the content.js and background.js scripts.



I hope that my experience will simplify the writing of more specialized cross-browser web extensions.



In the future we plan to localize the web extension and write useful scripts for the community. If the reader has an idea or desire to help in writing such custom scripts, then please write in private messages.