Simple site parsing with SlimerJS

Due to the lack of good material on parsing using SlimerJS scripting browser and the availability of free time, I decided to write a small article.

Beginning of work

In order to start working with SlimerJs, you need to download the latest version of the script browser (I usually download the full version, which includes XulRunner, which allows you to run SlimerJS in the absence of Firefox) for your OS.

I will not describe the installation of SlimerJS, as it is described in some detail here , so we will immediately proceed to the analysis of the situations that can most often be encountered when parsing a site with dynamic content.
')

Using

Suppose that site content is loaded by an Ajax after the page itself is already loaded. There are several ways to get dynamic content, the main one is to write a function that will check the availability of data after an interval of time. I used this option everywhere, but if the site needed to perform many sequential actions, then the code became larger and the more difficult it was to maintain it. As a result, I began to use more pauses at run time, which simplified the task:

var webpage = require('webpage').create(); webpage .open('http://example.com') .then(function(){ slimer.wait(3000); // ,    var someContent = webpage.evaluate(function () { //      ,    return document.querySelector("#aDiv").textContent; }); }); 

Another useful feature is event creation. It is very easy to simulate a character set in a text field, which is often required when authorizing or registering, for example:

 var webpage = require('webpage').create(); webpage .open('http://example.com') .then(function(){ webpage.evaluate(function () { document.getElementById("input").focus(); }); webpage.sendEvent('keypress', "hello World"); }); 

In general, the function senEvent () is very flexible, you can read more about it here .

Quite often there is a situation when captcha appears on the site when parsing. In this case, you will need to take a screenshot of the captcha and send it to one of the services that provide captcha solving services.

 var webpage = require('webpage').create(); webpage .open('http://example.com') .then(function(){ webpage.clipRect = { top: 14, left: 3, width: 400, height: 300 }; //           var screen = webpage.renderBase64('png'); //      base64,       var webpage2 = require('webpage').create(); //    webpage2 .open('http://example2.com') .then(function(){ //...      }); 

SlimerJS is very similar to PhantomJS, but the most important difference that I would like to note is the browser window during the execution of SlimerJS. In other words, debugging and development are easier than with a phantom.

Total

SlimerJS allows you to quickly and simply parse complex dynamic sites that can not be parsed with the usual cUrl.