TestCafe: A brief history of finding harmony in functional testing (with pictures)

TestCafe is a cross-platform framework for functional testing of web applications, recently released by DevExpress.

We at DevExpress are developing a wide range of components for web developers ( ASP.NET WebForms , ASP.NET MVC , as well as JavaScript components and frameworks ) and, of course, we test them. If everything is clear with unit tests and there is no space left for some incredible revelations, the situation with the functional tests is not so simple because of the complexity of their implementation. Initially, the TestCafe framework was an internal development aimed at solving immediate problems with the functional testing of our components and sites. And it so happened that this internal development eventually grew into an independent product.
')
In this article, I will tell you what the prerequisites were for the creation of this framework, what problems (besides the fatal flaw ) we faced when testing existing solutions and how we tried to cope with them.

About daily

Problem:

In the existing test solutions, the problem is the need to install browser plug-ins. Setting up the test environment becomes very nontrivial - you need to install plugins on browsers, sometimes configure the browsers themselves, the plug-in for the new version of the browser may simply not work, and you need to test here and now.

It often happens that in a large development team, not all developers simply have a test environment installed or an old version of the test framework installed. And there is such a situation that in the system of continuous integration a test falls, an urgent need to correct the error, and at the same time a significant part of the time is spent only on the fact that the test "took off" from you locally. Updating plug-ins, framework in the system of continuous integration is a separate fun topic.

Also here stand out mobile devices. For most configurations, there are simply no plug-ins, for some - the installation of the plug-in is far from trivial.

The main "chip" of TestCafe is that the framework does not require browser plug-ins and any browser pre-settings to work. TestCafe uses the plug'n'play approach for browsers. You can verify this by running a demo test in the browser in which you are currently viewing this article. If you are working on a local machine, then TestCafe will automatically detect installed browsers and create a worker for you:



Connecting a mobile device is just as easy, by pressing a single button from the browser of a mobile device or by scanning a QR code:



Here it is worth mentioning separately that the entire control system has a web-based interface. Those. You can run tests and analyze their results on your colleague's machine (assuming that you are on the same network) simply by connecting its browser to the framework. In this case, your colleague will not have to install anything. Moreover, this flexibility makes it possible to run tests in any remote environment, for example, on virtual machines Browser Stack .

Also TestCafe does not block the browser and keyboard / mouse input. Those. You can run tests in parallel in all browsers you have and at the same time continue to work in another tab of your favorite browser.

Problem:

There is one significant difference between test code and production code: as a rule, tests are code written once, but we often return to this code to analyze violations in the work of the application being developed. Those. The main purpose of the test code is to identify problems, so there are two requirements for it:

• determinism of the test scenario - the test should carry out exactly one scenario of actions and this scenario should be easily understood from the test code;
• the ability to quickly localize the stage of the scenario where the problem occurred.

These considerations are strongly imprinted in my head, when one day my colleague and I debugged a very complex functional test. It was a wall of code, from which it was difficult to trite to understand what generally happens in the test script. Therefore, we step by step debugged the test and compared the visible actions with the code. It took us just half an hour to figure out which scenario of action we are dealing with at all!

Yes, here the problem is in many respects the human factor - the test was written clumsily and the person who created the test did not take care of those who would analyze its results. But what to do - “if a person can shoot himself in the foot, then sooner or later he will shoot himself in the foot . ”

But since correct all people is a task, to put it mildly, a little feasible, then at TestCafe we ​​did not use a universal behavior algorithm in difficult situations ^[1] . We went a little different way, donating a little time to the developer of tests (recall that the test is usually written once) in favor of those who later analyze the results of this test.

TestCafe tests are divided into steps:

'@fixture Example'; '@page http://testcafe.devexpress.com/Example'; '@test'['Drag slider'] = { '1.Click label "I have tried TestCafe"': function () { act.click(':containsExcludeChildren(I have tried TestCafe)'); }, '2.Check initial slider value': function() { eq($('#Developer_Rating').val(), 0); }, '3.Drag slider handle': function () { act.drag('.ui-slider-handle', 360, 0); }, '4.Check resulting slider value': function() { eq($('#Developer_Rating').val(), 7); } }; 

There are two requirements for steps:

• steps must be named, i.e. the developer must describe in natural language what happens;
• Each step must contain no more than one user action.

I have to say that this approach was most ambiguously met by our developers and testers. They wanted to write code, and had to write text. However, soon the indignation faded away, the developers got used to this style. But now we have well-documented and structured tests, and the developers create a more meaningful test code, because they first formulate their actions in natural language. If problems arise, detailed analysis of the code is no longer required, the essence of the test can be simply read, without going into implementation details.

It is also worth noting that TestCafe analyzes network traffic and JavaScript errors. And if the test suddenly fell (did not pass assertion or some other error occurred), the framework will point out possible causes of the crash - unloaded resources or errors in the code, which sometimes is very useful in locating the problem.

Problem:

It would seem that a functional test is simple: we perform the emulation of the actions of the end user in a certain scenario and evaluate the results. But there are a number of hidden mechanisms that occur outside the user's control, but affect the execution of the script. And test developers have to take them into account.

For example, you press a button, an asynchronous XHR request is sent, some data is sent in response, from which some block of the page is built, which is displayed to the user. For the user, this is just a slight visual delay, and for the machine, it is an action with side effects and a non-deterministic completion time. And it turns out that without deepening the meaning of what is happening, it is difficult to write a good test (after all, in the test we need to explicitly indicate that it is necessary to wait until the XHR request has been loaded and then proceed to the next step). There are other options when internal mechanisms have a strong influence on passing tests. For example, in many frameworks you have to manually clear cookies before running tests. It is worth forgetting to do this cleaning and you can get a very unstable and difficult to debug test.

Remember the test steps mechanism I mentioned earlier? In addition to improving the structure of the code, it solves another problem - it allows to level the transitions between the application states. Those. in TestCafe, you do not need to explicitly prescribe things like waitForPageLoad () or waitForXhrComplete () . Since each step contains exactly one action, then TestCafe automatically waits for the completion of the transition between pages and the end of XHR requests caused by this action. Thus, you do not need to describe the transition states in the test. The test contains absolutely nothing that should contain - emulation of user actions and analysis of the final states to which they lead.

We also took care of clearing the cookie . Each test is essentially performed in a certain test container, which always contains empty cookies when the test starts. You do not need to worry about resetting the state and setting up a browser test environment. This happens automatically.

Problem:

Often a test page requires Basic or Windows authentication. And often this problem has no solution "out of the box."

TestCafe has built-in support for these authentication protocols — just enter your login information and TestCafe will automatically authenticate, just like your browser does:



You can also specify login details in the test code:

 '@auth guest:MostlyHarmless'; '@test'['Click Input'] = { '1. Type name': function() { act.type(getInput(), 'Peter Parker'); }, '2. Move caret position': function() { act.click(getInput(), { caretPos: 5 }); }, '3. Erase a character': function() { act.press('backspace'); }, '4. Check result': function() { eq(getInput().val(), 'Pete Parker'); } }; 

Problem:

The API of functional testing frameworks is a thing that deserves special attention. These monstrous add-ins on DOM, which sometimes do not keep pace with the innovations in the DOM API browsers - not the easiest thing to learn.

“But testers who are not familiar with DOM and JavaScript are easier!” You say. But, this statement contains an internal contradiction: this API, anyway, is a wrapper for the native DOM-tree. Those. in any case, you need to understand the structure of the DOM and, moreover, keep in mind the additional API and understand how one is projected into another.

As you may have noticed, the test creation language in TestCafe is JavaScript. Or rather, to be precise, this is DSL , which is a subset of JavaScript. This means that any existing JavaScript editor can easily open test files with all the accompanying "buns", such as syntax highlighting, auto-completion and refactoring.

Moreover, there is no more natural language for the web than JavaScript. There is no middleware API in TestCafe, you can work with DOM directly (this is if you are a big fan of Vanilla JS) or you can use the automatically included version of jQuery.

Separate attention deserve available in TestCafe emulation of end-user actions. We tried to keep the API as minimalistic as possible, while keeping it as flexible as possible. Only 11 methods , but they cover almost all scenarios we encountered in practice.

API TestCafe also includes 4 types of assertions , able to recognize the types of objects and make a "deep" comparison. To edit tests, you can use the editor built into the framework:



You can also use your favorite JavaScript editor. TestCafe compiles tests on the fly, so if you make a syntax error or form a test structure incorrectly, you will immediately be notified of this:



Problem:

Functional tests may be unstable. As a rule, the cause is an incorrectly set waiting time for effects to complete from a user action. Changes in time intervals can be caused by many factors: network delays, slow computers, slowness of the application being tested. TestCafe, as mentioned earlier, has built-in mechanisms for waiting for page transitions and ending XHR requests. However, heuristics for animation expectations are still at the research stage and are not included in the release branch of the framework. In addition, the instability of the test can be caused by the instability of the application being tested.

In order to catch such tests, TestCafe introduced a special mode - Quarantine Mode , which in general is the technical implementation of Martin Fowler's idea of ​​quarantine for unstable tests . Its essence is quite simple: if the test falls, then it is not immediately added to the list of dropped tests, but enters the "quarantine". This means that the test will be run 2 more times and if the results of the passage vary, then such a test will be added to the list of unstable tests in the report.

This rather simple mechanism allows to effectively capture tests with a non-deterministic result. It gives a particularly good result when working in a system of continuous integration, where the volume of statistical sampling is naturally much larger. By the way, TestCafe has a built-in API for continuous integration . The framework is available as a regular node-library via npm .

About the test recorder

Of course, in addition to everything described earlier, TestCafe has a built-in test recorder with a convenient live editor of selectors and the ability to roll back the actions performed:

As a conclusion

When developing TestCafe, we used a huge amount of open-source development, so from our side it would be a little unfair not to pay tribute to the open source community. So, in the context of the development of TestCafe, parse5 was created - the HTML parser ^[2] , which is the fastest of the parters that are fully compatible with the HTML5 specification for node.js. And whacko is a fork of the cheerio library, using parse5 as a platform.

We still have a huge number of plans for the future, which we are trying to implement as quickly as possible (the major release of TestCafe is released approximately every 2 months). In the future, in addition to the obvious improvements in usability and speed of work, many unique and interesting features are waiting for us. Stay in touch!

Notes