Selenium and Node.js: we write reliable browser tests

There are many good articles on how to start writing automated browser tests using the Node.js version of Selenium.



Some materials talk about how to wrap tests in Mocha or Jasmine, some automate everything using npm, Grunt or Gulp. In all such publications, you can find information on how to install and configure everything you need. There you can see simple examples of working code. All this is very useful, since, for a beginner, it may not be so easy to assemble a working test environment consisting of many components.
')
However, as far as the Selenium pitfalls are concerned, with regard to the analysis of the best practical techniques for developing tests, these articles usually do not justify expectations.

Today we will start with what other materials on automation of browser tests usually end with using Selenium for Node.js. Namely, we will talk about how to increase the reliability of tests and "untie" them from the unpredictable phenomena that browsers and web applications are full of.

Sleep is evil

The Selenium driver.sleep method is the worst enemy of the test developer. However, despite this, it is used everywhere. Perhaps this is due to the brevity of the documentation for the Node-version of Selenium , and because it covers only the syntax of the API. She lacks real life examples.

Perhaps the reason is that this method is used in a variety of code examples on blogs and question and answer sites like StackOverflow.

In order to understand the features of the driver.sleep method, consider an example. Suppose we have an animated panel that, during the appearance on the screen, changes its size and position. Take a look at her.


Animation panel

This happens so quickly that you may not notice that the buttons and controls inside the panel also change in size and change position.
Here is a slow version of the same process. Notice how the green Close button changes with the panel:


Slow Motion Panel Animation

It is unlikely that this behavior of the panel can interfere with the normal operation of real users, since the animation happens very quickly. If it is slow enough, as in the second example, and you try to click the Close button during the animation process, you may very well just not get to it.

Typically, these animations occur so quickly that the user does not have the desire to "catch" changing buttons. People are just waiting for the completion of the animation. However, this does not apply to Selenium. It is so fast that it can try to click on an element that is still animated. As a result, you may encounter something like this error message:

 System.InvalidOperationException : Element is not clickable at point (326, 792.5) 

In this situation, many programmers will say: “Yeah, I need to wait for the animation to complete, so I just use the driver.sleep(1000) in order for the panel to driver.sleep(1000) to its normal state.” It seems that the problem is solved? However, not all so simple.

Driver.sleep problems

The driver.sleep(1000) command does exactly what you can expect from it. It stops the test for 1000 milliseconds and allows the browser to continue to work: load pages, place fragments of documents on them, animate or smoothly display elements, or do anything else.

Returning to our example, assuming that the panel reaches the normal state in 800 milliseconds, the driver.sleep(1000) command usually helps achieve what it is called for. So why not use it?

The main reason is that this behavior is non-deterministic . This means that sometimes such code will work, and sometimes it will not. Since this does not always work, we come to unreliable tests that, under certain conditions, fail. Hence the bad reputation of automated browser tests.

Why constructions with driver.sleep not always efficient? In other words, why is this a non-deterministic mechanism?

A web page is much more than what you can see. And the animation of the elements is a great example. However, while everything is working as it should, nothing special is needed to be seen.

It is worth saying that web pages are designed in the expectation that people will work with them. During testing using Selenium, a program that interacts much faster with humans will interact with the pages. For example, if you tell Selenium to first find an element and then click on it, it can take only a few milliseconds between these operations.

When a person is working with a website, he waits for the element to appear fully before clicking on it. And when the appearance of an element takes less than a second, we probably won't even notice this “wait”. Selenium is not only faster and more demanding than a regular user. Tests, in the course of working with pages, have to face various unpredictable factors. Consider some of them:

1. A designer can change the animation time from 800 milliseconds to 1200 milliseconds. As a result, a test with driver.sleep(1000) will fail.
   
2. Browsers do not always do exactly what is required of them. Because of the load on the system, the animation can slow down and take more than 800 milliseconds. Perhaps even longer than the wait time set to 1000 milliseconds. As a result, the test failed again.
   
3. Different browsers have different data visualization mechanisms, assign different priorities to the operations of placing elements on the screen. Add a new browser to the testing suite and the test will crash again with an error.
   
4. Browsers that control pages, JavaScript calls that change their content, are asynchronous in nature. If the animation in our example is applied to a block that needs information from the server, then before starting the animation you will have to wait for something like the result of an AJAX call. Now, among other things, we are dealing with network delays. As a result, it is impossible to accurately estimate the time required to display the panel on the screen. The test will not work properly again.
   
5. Of course, there are other reasons for test failures that I don’t know about. Even the browsers themselves, without taking into account external factors, are complex systems in which, besides, there are errors. Different errors in different browsers. As a result, trying to write reliable tests, we strive to ensure that they work in different browsers of different versions and in several operating systems of different releases. Nondeterministic tests in such conditions sooner or later fail. If you take into account all this, it becomes clear why programmers refuse automated tests and complain about how unreliable they are.
   

What will the programmer do to fix one of the above problems? He will start looking for the source of the failure, find out that the whole thing is in the time it takes the animation and will come to the obvious solution - to increase the waiting time in the driver.sleep call. Then, relying on luck, the programmer will hope that this improvement will work in all possible test scenarios, that it will help to cope with various system loads, smooth out differences in the visualization systems of various browsers, and so on. But we still have a non-deterministic approach. Therefore, it is impossible to do so.

If you have not yet verified that driver.sleep is a harmful command in many situations, think about this. Without driver.sleep tests will run much faster. For example, we hope that the animation from our example takes only 800 milliseconds. In a real test suite, such an assumption would lead to using something like driver.sleep(2000) , again, in the hope that 2 seconds would be enough for the animation to complete successfully, whatever the additional factors affecting on the browser and page.

This is more than a second lost by just one step of the automated test. If there are many such steps, a lot of such “spare” seconds will come very quickly. For example, a recently reworked test for just one of our web pages, which took a few minutes due to excessive use of driver.sleep , now takes less than fifteen seconds.

I offer you specific examples of getting rid of driver.sleep and converting tests to robust, fully deterministic constructs.

A couple of words about promises

JavaScript-Selenium intensively uses promises. At the same time, the details are hidden from the programmer due to the use of the integrated promis manager. It is expected that this functionality will be removed , so in the future you will either have to figure out how to independently combine promises in chains, or how to use the new JavaScript async / await mechanism.

In this material, the examples still use the traditional embedded manager of Selenium promises and the possibility of combining promises into chains. If you understand how promises work, this will be a big plus when analyzing the following code examples. However, you will get benefit from this material even if you haven’t properly dealt with the promises.

We write tests

Let's continue the example with the button located on the animated panel. We want to click on this button. Let's look at a few specific features that our tests may break.

What about an item that is dynamically added to the page and does not exist immediately after the page completes the download?

Waiting for an item to appear in the DOM

The following code will not work if an element with a CSS id my-button was added to the DOM after the page loads:

 //   Selenium    //  . driver.get('https:/foobar.baz'); //  . const button = driver.findElement(By.id('my-button')); button.click(); 

The driver.findElement method expects the element to already be present in the DOM. It will give an error if the item cannot be found immediately. In this case, “immediately,” due to a call to driver.get , means: “after the page has finished loading.”

Remember that the current version of Selenium for JavaScript independently manages promises. Therefore, each expression will be fully completed before proceeding to the next expression.

Please note that the above scenario is not always desirable. The driver.findElement call driver.findElement can be handy if you are sure that the element is already in the DOM.

To begin, take a look at how you should not correct this error. Suppose we know that adding an element to the DOM can take several seconds:

 driver.get('https:/foobar.baz'); //  ,     driver.sleep(3000); // ,      ,            . const button = driver.findElement(By.id('my-button')); button.click(); 

For the reasons mentioned above, such a construction can lead to failure, and is likely to result. We need to figure out how to wait for an element to appear in the DOM. It's pretty simple, like this is often found in the examples that can be found on the Internet. Let's use the well-documented driver.wait method to wait no more than twenty seconds for the element to appear in the DOM.

 const button = driver.wait( until.elementLocated(By.id('my-button')), 20000 ); button.click(); 

This approach will immediately give us a bunch of advantages. For example, if an item is added to the DOM within one second, the driver.wait method will exit in one second. He will not wait all the twenty seconds that are allotted to him.

Because of this behavior, we can set timeouts with a large margin, without worrying that they will slow down the tests. This model of behavior compares favorably with the driver.sleep , which will always wait for the entire specified time.

This works in many situations. But the only case in which such an approach does not help us is to try to click on an element that is present in the DOM but is not yet visible on the screen.

Selenium is intelligent enough to not try to click on an invisible element. This is good, because the user cannot click on such an element, but this complicates the work of creating reliable automated tests.

Waiting for item to appear on screen

We will build on the previous example, since, before we wait for the element to become visible, it makes sense to wait until it is added to the DOM. In addition, in the code below, you can see the first example of using a chain of promises:

 const button = driver.wait( until.elementLocated(By.id('my-button')), 20000 ) .then(element => {  return driver.wait(    until.elementIsVisible(element),    20000  ); }); button.click(); 

On this, in general, it would be possible to stop, as we have already considered enough to significantly improve the tests. Using this code, you can avoid a lot of situations that would otherwise cause the test to fail because the element is not in the DOM immediately after the page has finished loading. Or due to the fact that it is invisible immediately after loading the page because of something like an animation. Or even for both of these reasons.

If you master the above approach, you should not have any reason to write nondeterministic code for Selenium. However, writing such code is simply not always the case.

When the complexity of a task grows, developers often give up their positions and again turn to driver.sleep . Consider a few more examples that will help to do without driver.sleep in more difficult circumstances.

Description of own conditions

Thanks to the until method, the JavaScript API for Selenium already has a number of helper methods that can be used with driver.wait . In addition, you can arrange to wait until the element no longer exists, wait for an element containing specific text to appear, wait for a notification to show, or use many other conditions.

If you cannot find what you need, among the standard methods, you will need to write your own conditions. It is, in fact, quite simple. The main problem here is that it is difficult to find examples of such conditions. Here is another pitfall that we need to deal with.

According to the documentation , you can provide the driver.wait method with a function that returns true or false .

Let's say we need to wait for the opacity property of a certain element to become equal to one:

 //  . const element = driver.wait( until.elementLocated(By.id('some-id')), 20000 ); // driver.wait    ,   true  false. driver.wait(() => { return element.getCssValue('opacity')        .then(opacity => opacity === '1'); }); 

Such a construction seems useful and suitable for reuse, so we put it in a function:

 const waitForOpacity = function(element) { return driver.wait(element => element.getCssValue('opacity')        .then(opacity => opacity === '1'); ); }; 

Now use this function:

 driver.wait( until.elementLocated(By.id('some-id')), 20000 ) .then(waitForOpacity); 

And here we are faced with a problem. What if you want to click on an element when it becomes completely opaque? If we try to use the value returned by the above code, nothing good will come of it:

 const element = driver.wait( until.elementLocated(By.id('some-id')), 20000 ) .then(waitForOpacity); //  .  element   true  false,   ,     click(). element.click(); 

For the same reason, we cannot use the combination of promises in chains in such a construction.

 const element = driver.wait( until.elementLocated(By.id('some-id')), 20000 ) .then(waitForOpacity) .then(element => { //    , element     . element.click(); }); 

This is all, however, easy to fix. Here is the improved method:

 const waitForOpacity = function(element) { return driver.wait(element => element.getCssValue('opacity')        .then(opacity => {     if (opacity === '1') {       return element;     } else {       return false;   }); ); }; 

This piece of code returns an element if the condition is true, otherwise returns false . This template is suitable for reuse, it can be used when writing your own conditions.

Here's how to apply this together with the combination of promises in chains:

 driver.wait( until.elementLocated(By.id('some-id')), 20000 ) .then(waitForOpacity) .then(element => element.click()); 

Or even like this:

 const element = driver.wait( until.elementLocated(By.id('some-id')), 20000 ) .then(waitForOpacity); element.click(); 

Creating your own conditions allows you to expand the capabilities of tests and make them deterministic. However, this is not always enough.

Leaving in the minus

That's right, sometimes you need to go into the minus, and not to strive to get a plus. I am referring here to checking something that no longer exists, or something that is no longer visible on the screen.

Suppose an element is already present in the DOM, but you do not need to interact with it before some data is loaded via AJAX. The item can be covered with a panel that says "Loading ...".

If you paid close attention to the conditions that the until method suggests, you might have noticed methods like the elementIsNotVisible or elementIsDisabled or the not-so-obvious stalenessOf method.

Thanks to one of these methods, you can organize a check to hide the panel with the load indicator:

 //     DOM,     . const desiredElement = driver.wait( until.elementLocated(By.id('some-id')), 20000 ); //        ,      //  . driver.wait( until.elementIsNotVisible(By.id('loading-panel')), 20000 ); //        ,     ,   . desiredElement.click(); 

Exploring the above methods, I found that the stalenessOf method stalenessOf particularly useful. It waits until the item is removed from the DOM, which, among other reasons, may occur due to a page refresh.

Here is an example of waiting for an update of the iframe content to continue:

 let iframeElem = driver.wait( until.elementLocated(By.className('result-iframe')), 20000 ); //   ,     iframe. someElement.click(); //    iframe  : driver.wait( until.stalenessOf(iframeElem), 20000 ); //    iframe. driver.wait( until.ableToSwitchToFrame(By.className('result-iframe')), 20000 ); // ,    ,     iframe. 

Results

The main recommendation that can be given to those who seek to write reliable tests on Selenium is that one should always strive for determinism and abandon the sleep method. Relying on the sleep method is based on arbitrary assumptions. And this, sooner or later, leads to failures.

I hope that the examples given here will help you to move towards the creation of high-quality tests on Selenium.

Dear readers! Do you use Selenium to automate tests?