TextTest is a cross-platform python framework for GUI testing and more. Part 2

Continuation of the story about the wonderful cross-platform framework for functional testing TextTest. The first part of the article.

GUI Testing Methods

Now the vast majority of GUI testing tools work in one of two ways:

1. Take screenshots after each screen change and then compare the changes.
2. Provide functions that allow you to read widget parameters through the operating system API

Problems of the first - the instability of tests, it is worth changing the relative position of the buttons on the form, even a couple of pixels, a lot of tests will fall, the screen resolution will change - all tests will fall, switched to a new version of the GUI framework, for example, improved font rendering - again all tests will fall. For each operating system, you need to do your own set of tests, you may have to redo it even after installing the next service pack, if it affected the graphics subsystem.
I do not argue about the appearance of testing useful, but it should be tests only for the design, to test through them the logic of the GUI is a mockery of a person.

The second approach has other problems - tests are written long and difficult, and in addition they are often completely incomprehensible to others, especially to non-programmers. Footcloths code like this:

ActivateWindow("Unsaved Document 1 - gedit") SetEdit(5, "5") ClickButton("Save") Wait(10) VerifyLabel(2, 10) 

They do not give an idea of ​​what is happening and what is actually being tested. The programmer needs to explicitly write assertions, to the fact that after letting this button be pressed, this label has changed its value to this, but this one has not changed. Also, the operating system API may not provide full access to the internals of widgets, and if it is a non-standard widget, then it becomes almost impossible to consider its properties.

But then it is easier to test the logic of the GUI, we are not tied to a graphical representation of the elements, and with a good tool you can even try to write cross-platform tests.
')
Finally, both approaches have a common problem with testing non-synchronous events. Suppose you enter a URL into your browser, and when to check that the page has loaded?

You can make a timeout, but here the problem is that if the page loads on average for 5 seconds, then you need to “for faithfulness” set a timeout of 5 times more, and it’s not a fact that everything will work. This leads to a slowdown of tests and to the probabilistic nature of their implementation.

Option two - “sharpen” on some secret widgets and as soon as they accept the desired state, assume that the action has been completed. This adds work to the programmer and makes the tests even more unreadable and incomprehensible.

What alternatives does TextTest offer?

He suggests using the StoryText library to solve the problems described above. The latter, before launching your application, “wraps” the GUI library reference interfaces, pushing its interface options to the program. And this is done completely transparent. In most cases, you do not have to change a single line of code in the application under test. This makes it possible to save the user's actions and the program's response to these actions to the log when recording a test. And when testing - to repeat the actions of the user and compare the reaction of the program with the reference.

So what does this give us:

• We abstract from the arrangement of elements on the form and their appearance, it does not matter to us that in different OS the buttons have a different appearance, we test only the logic. If we need to check that after clicking on the button, the label values ​​have changed, then no design changes to the form will break this test.
  
  
• We get full access to the widget, no WinApi will allow to get as much information about the widget as the native library that creates this widget.
  
  
• For asynchronous events, such a concept is introduced as Application Events, which, although it causes a little modification of the code, is just one line for each asynchronous event. In short, the essence of the latter is that upon completion of an asynchronous event, we call the code notifying StoryText that the operation has completed. StoryText remembers this fact and during playback it will be honest to wait for this event from the application and only then continue the test. Everything! No timeouts, no hidden widgets and other perversions, everything is simple and as fast as possible.
  
  
• The ease of writing tests, allows them to create even a little familiar with programming person. You don’t have to search for the necessary button with the code and click on it, then you don’t need to sort the values ​​of the changed elements, StoryText itself records all the events that the user has made and saves all the changes to the GUI, and, as you will see, does readable form.
  
  
• Easy to understand tests. After recording the first test, StoryText offers to enter aliases for all events that have occurred, i.e. automatically created test will not look like this:
  

   entry_in = FindEditByName("entry_in") SetValueForEdit(entry_in, 5) calc_async = FindButtonByName("calc_async") SendEvent(calc_async, <Enter>) SendEvent(calc_async, <Button-1>) SendEvent(calc_async, <ButtonRelease-1>) WaitEvent("data to be loaded") exit = FindButtonByName("exit") SendEvent(exit, <Enter>) SendEvent(exit, <Button-1>) SendEvent(exit, <ButtonRelease-1>) 

  
  that agree a little nice, and here it is:
  

   enter_data 5 run_calc_async wait for data to be loaded exit_from_form 

  
  Where enter_data, run_calc_async and exit_from_form are user names that he entered after recording a test (the values ​​of aliases - you can always see the special settings file)

Agree ideology - a wonderful, tests are created quickly, conveniently, readable. There are of course problems, not with ideology, but with implementation. The author seems to have the main library for the GUI gtk, the rest are added in a certain experimental mode and not fully implemented, however, I will show below how easy it is to add new functionality.

Fourth example. Test synchronous GUI on Tkinter

Test class will TestGUI. The class creates a very simple form on which it is proposed to enter a value, and press one of the OnCalcSync or OnCalcAsync buttons, the first will display the value multiplied by two at once, the second will wait 10 seconds and will output the same. Yes, and I apologize in advance for the terrible look of the form, but this is only an example.

I want to note that when you run the test, the tkinter_ex module is imported (you can download it from here and put it next to test.py). He is needed because tkinter support in the library is still “Experimental and rather basic support for Tkinter” as the author himself writes. In particular, the standard module for tkinter is completely unable to track text changes in the Label class, but fortunately it is not difficult to fix it. To do this, we replace the standard Tkinter.Label with our own while preserving the original name and functionality, and in functions that could change the text of the label - “configure” and "__setitem__", we add the following type of logging: “Updated Text for label '% s' ( set to% s) ". Thanks to the dynamism of Python, the code is not complicated.

So, let's go: Add a new test-suite “Suite_GUI” and test it with “Test_GUI_Sync” with the parameter “gui”. in simpletests \ config.cfg we add settings indicating that we will be testing the GUI based on tkinter

 # Mode for Use-case recording (GUI, console or disabled) use_case_record_mode:GUI # How long in seconds to wait between each GUI action slow_motion_replay_speed:3.0 # Which Use-case recorder is being used use_case_recorder:storytext # Single program to use as interpreter for the SUT interpreter:storytext -i tkinter virtual_display_count:0 

I will not decipher all the settings, you can read about them here . I’ll draw attention to only one: “virtual_display_count”, it should only make sense on UNIX systems and allows you to run tests on virtual displays via Xvfb. But due to an implementation error, if this parameter is not set, StoryText attempts to create virtual displays on Windows, where Xvfb is missing. Therefore, the setting must be explicitly added and set to 0.

Do not forget to restart the IDE after making the settings. After that, in the “Test_GUI_Sync” menu, the “Record Use-Case” item will become available, we launch it, we do not change anything in the appeared window. Our test-free form appears:

in it, you need to perform the actions you want to test, enter a value in the "Input" field, click "OnCalcSync" and then "Exit". After this, StoryText will ask us to give human names for the actions performed, fill them in, for example like this:

As usual, we save and try to run the test again, everything works fine.
To understand what happened, you can look at what appeared on the disc: simpletests \ storytext_files \ ui_map.conf, where the aliases we just entered are described. It is also worth looking in the folder with the test (Test_GUI_Sync) on the file usecase.cfg, in which the resulting actions are described in completely human language:

 enter_data 5 run_calc_sync exit_from_form 

in stdout.cfg, you can see all the changes in the form state that occurred after user actions. In the same place, we will see the line generated by our class in response to a change in the Label value “Updated Text for label 'label_out' (set to 10)”. So it works. Excellent move on to the next example.

Fifth example. Test the asynchronous GUI on Tkinter

Add a new test “Test_GUI_Async” to “Suite_GUI” and write it as last time, only instead of “OnCalcSync” press the button “OnCalcAsync” and the results of the calculation will have to wait for this time for 10 seconds. Upon completion, you will need to somehow call the action of clicking on a new button, for example, for example, the path will be “run_calc_async”. All other actions we called last time and StoryText remembered them.

This test is interesting because the actions on the form now occur asynchronously and after clicking on the button it takes ten seconds before the “TestGUI._operation_finish” is called from the test.py module. In this function, in addition to calculating the result, the line is added:

 storytext.applicationEvent('data to be loaded') 

which tells StoryText that some kind of asynchronous operation has completed its work and it needs to be reflected in the test, and when automatically executed, you should stop at the same place and wait for the event to occur.
If you look at usecase.cfg for a new test, you will see:

 enter_data 5 run_calc_async wait for data to be loaded exit_from_form 

those. the difference from the previous example is only that after pressing the button we expect the “data to be loaded” event and only after it close the form. Save the results, run again, we see that StoryText patiently waits for the required 10 seconds before completing the test.
As you can see, testing asynchronous events is not at all difficult.

Batch execution

Everyone knows that if the launch of tests requires a lot of time and effort, no one will do it, so tests should be run with one button or even run on a separate server in a fully automatic mode. TextTest for this function provides batch execution with the subsequent formation of the report and either uploading it to a format (html, JUnit, Jenkins), or sending it by e-mail.

We will upload everything in html. You only need to configure the folder path to store the results of the tests and the path to store the html reports. To do this, add in the main config.cfg these lines to the end of the file:

 [batch_result_repository] default:batchresult [historical_report_location] default:historicalreport 

There you can specify quite a few additional parameters, I will not dwell on them, here is a link to the documentation on the batch mode.
To run the tests automatically, you need to start the texttest.py module with the "-b nightjob" parameters, like this:

 python c:\TextTest\texttest-3.24\source\bin\texttest.py -b nightjob 

Again, for the command line, there are also a bunch of parameters, here you can read more.
After the execution, a new batchresult with the results will appear next to the simpletests folder, of which it is already possible to generate a report in the form of html by adding the "-coll web" parameter:

 python c:\TextTest\texttest-3.24\source\bin\texttest.py -b nightjob -coll web 

now there will be another historicalreport folder in which lies the html page with the results. After several days of work, it may look like this:

Well, or you can look at the test results of TextTest itself. So, by the way, quite interesting statistics - about 4,000 tests are run daily, and the test coverage, almost all modules approach 100% (although I certainly understand that the percentage of coverage in itself means little).

Let's sum up

pros

Let's try to sum up the positive aspects of the framework, mainly from the point of view of GUI testing:

• Cross-platform, cross-library (meaning GUI libraries PyGTK, Tkinter, etc.), even partially cross-language support (the author often mentions working with Java, though I don’t know how convenient it is)
  
  
• Free and open source.
  
  
• The activity of the author in the development of the framework, which is surprising for a 10-year project.
  
  
• After initial setup - easy to use.
  
  
• A lot of work has been done to make it possible for non-programmers to use the product, and to make the tests themselves human-readable.
  
  
• To test the GUI, not the operating system API is used, but the toolkit API itself (PyGTK, Tkinter, etc.), which makes integration as complete as possible and at the same time simple.
  
  
• Allows you to test the logic of work, abstracting from the appearance.
  
  
• Quite flexible, when faced with a problem, you can almost always climb through the documentation and find how to fix everything by setting it up.
  
  
• Allows you to integrate with popular bugtrackers (Bugzilla, Jira, Trac), version control systems (CVS, Bazaar, Mercurial), and upload work results in formats (html, JUnit, Jenkins) or send by mail. And add integration with other systems, it seems not very difficult. Although I confess, I have not tried yet how well he knows how to work with all these systems.

Minuses

It would be wrong not to mention the cons, especially since they are:

• The first is a GUI, it is quite a bit functional, the settings for the parameters are mainly made not through it, but through the documentation + manual editing of configs. For the sake of fairness, it’s necessary to say that the adjustment is performed once, and then we only rivet the type tests. But at the initial stage of the study there is very little opportunity to quickly arrange the checkboxes and start working to understand what's what.
  
  
• The stability of the GUI, sometimes it falls, I don’t even know where it belongs, either to the work of PyGTK, or to the weak testing under Windows, but the fact remains. But again, the GUI is used only when creating a test, and then it is run many times and I did not observe any problems with the latter.
  
  
• Sometimes there are errors and you have to go into the code to understand what happened (I gave an example with the virtual_display_count parameter.) However, everything that I came across in one way or another is solved, especially since the source is open and you can always see what needs to be corrected. Here again, I sin on Windows, it seems the author is more interested in working under UNIX systems, I hope there are no such problems.
  
  
• Documentation. Maybe it's me, but I have to read the documentation very, very thoughtfully, several times to understand how the next thing works. Some understandable things are painted very well, and in complex ones nuances are missed. The available documentation is quite convenient to use as a reference book, but only after you understand how this or that thing works.
  
  
• Community Here things are really bad, on the Russian Internet in general, everyone is silent about the framework, the maximum mention that it exists. In English, all I found was stories about ideology and how good it is. Any sensible descriptions, manuals, articles, etc. I did not find everything on the author’s site. I can’t explain this fact in any way, the framework looks very mature, interesting and “tasty”, there are practically no alternatives, I don’t understand what’s wrong with it.
  
  
• It is hosted on a launchpad, and not on the popular GitHub today, I am afraid that this fact will alienate many who have the desire and ability to help the author in writing the code.

Alternatives

The description would not be complete if you did not specify alternatives for testing the GUI in python, this is what was found:

robotframework say something definite about him is difficult, it is very extensive and you need to read the documentation and try for a long time and thoughtfully. Apparently, this is a cross-platform framework for testing the TextTest level and it is definitely worth seeing for everyone who chooses a tool for himself.

ldtp is the next GUI testing tool. It has three implementations.
LDTP / Cobra / PyATOM for Linux / Windows / OS X respectively. It supports a bunch of languages ​​(Java / Ruby / C #, etc.) including the Python we need. I liked the documentation. Actively developing.
Not only liked the principle, judging by the description of the tests are written something like this:

 selectmenuitem('frmUnsavedDocument1-gedit', 'mnuFile;mnuOpen') settextvalue('frmUnsavedDocument1-gedit', 'txt0', 'Testing editing') setcontext('Unsaved Document 1 - gedit', '*Unsaved Document 1 - gedit') verifytoggled('dlgOpenFile...', 'tbtnTypeafilename') 

those. the element with the necessary heading is searched for and the necessary action is performed on it or the state is read. That after TextTest seems to be a step backwards, but if something does not work out with the latter, then ldtp will be one of the first candidates for the transition.

pywinauto about him and in Habré you can find a few words . The principle is the same as in ldtp - we find the element we need and do something with it. Yes, and it works only under Windows.

Dogtail judging by the description is very tied to Unix, there are even separate versions for “GNOME 3, KDE4.8” and for “Gnome 2”, so it looks like it uses the graphical shells for testing the API, which means it is not cross-platform. However, it is still evolving, so if something needs to be tested under Unix, it is probably worth looking in its direction.

The guitest library for testing GUI applications in python applications, mainly for pyGTK. Last release date is 13/11/2005. I'm afraid that after spending so many years without development, she will not be able to test the applications announced by pyGTK, not to mention the required Tkinter. Did not even look.

pyAA is another abandoned project, abandoned since 2005, and it’s scary to get involved in it, and it works only with Windows ...

pyGUIUnit by reference states that the library can test PyQt applications. If you go to the documentation, we see that the whole framework consists of one class and two functions, skimming through which - you can understand that nothing good can be expected.

Plus, there is still a fairly large number of common testing frameworks that allow you to test any application through screenshots, I described above what they are uncomfortable and therefore did not even consider, besides most of them work only on Windows.

Total: cross-platform frameworks for testing GUI logic in python with Tkinter support are surprisingly few, most of them are abandoned and forgotten. Those 2-3, which are worth it to look at them, not a fact that fit. TextTest so far, in my opinion - the perfect choice, let's see what happens next.