Functional testing of Qt programs

Foreword

The only way to check that after your last correction made to the version control system, important application usage scenarios still work correctly (or at least work somehow) is, of course, take and run these scripts through the test system. Doing it manually is long, tedious and fraught with mistakes.

Considering all of the above, as well as the "insignificant" fact that the customer in TK prescribed the need for automatic testing of the functional requirements specified in the same TZ, at the start of the next project, the issue of choosing a tool to automate the GUI testing became relevant. The project was on Qt, and required cross-platform (Windows, Linux).

What finally opensource tool appeared, look on a cat.

GUI Testing Tools

What off-the-shelf GUI testing solutions were available at that time (a few years ago)?

To summarize, there were two classes of possible utilities:

1. Programs created initially to automate user actions, and not to test.
   I think everyone can call a couple for their favorite OS. For example, for Linux / X11 - see the post on Habré .

   
   

   None of these utilities was suitable for us, since it did not satisfy at least one of the requirements formulated:

   
   

   1. Cross platform
      Most of them are not cross-platform, i.e. work either on Windows only or on Linux only.

      
      
   2. Excessive attachment to implementation details.
      Even if you solve the problem with cross-platform (for example, running the program on a Linux machine, and the X server for it on Windows), then over-binding to the implementation details leads to the following problems.
   
   

   The simplest utilities recorded and reproduced mouse clicks in the coordinate system (CK) of the screen (i.e., another screen resolution - and the test drops), smarterly used the ICs of the window (another Qt style, another DPI - and the test drops). The best knew how to recognize native OS widgets, but unfortunately, Qt very little used the native elements of a specific OS (Qt masks the appearance of its widgets for a specific OS, in this case Linux or Windows, but inside it is the same QWidget) .

   
   

   Bottom line: none of the utility of this class did not suit us. It would be necessary to create two variants of tests for the same user experience scenario - for Linux and for Windows. Plus, it would be too difficult to support the tests created with the help of them (any change in the interface breaks them).

   
   
2. Programs created specifically for testing. Considering experience (1), from this class of utilities we considered only utilities that have Qt support. This was exactly one - squish ( description on Habré ) with the appropriate approach to pricing - "contact us, we will evaluate you and set the price." So it was a few years ago, perhaps now something has changed. I sent them a request, but did not wait for an answer.

The result is natural - it was decided to make such a tool on their own.

Qt Monkey

Alpha version

Initially, the task seemed pretty simple. There is a Qt subsystem with the characteristic name QTest (I already used it in the project to write unit tests for my own widgets). Using it is quite easy to record a sequence of keystrokes and mouse clicks ( QTest::mouseClick , QTest::keyCick ). You can generate the test code using the QEvent -> QTest::something transformation, first asking Qt with qApp->installEventFilter to report all events in the application under test. As a result, the preliminary version was quickly ready.

However, loading tests with the help of the plug-in mechanism and writing tests in C ++ for some reason did not cause understanding among QA engineers. Fortunately, Qt has an easy way to embed JavaScript in an application — QtScript . This subsystem makes it very easy, almost a couple of lines of code, to ensure the interaction of QObject and JavaScript heirs by translating calls in both directions:

 QScriptEngine engine; QScriptValue global = engine.globalObject(); QScriptValue val = engine.newQObject(qobject); global.setProperty(QLatin1String("myobject"), val); 

and in javascript:

 myobject.slot1(); var v = myobject.property1; 

It remains only to figure out how to identify widgets, because to create an object ( QObject ), the properties ( Q_PROPERTY ) of which - pointers to all graphic elements of the program that have ever been created, are quite difficult.

After some period of reflection, we stopped at the following naming scheme for the widget:

"Parent identifier (if any)" point "identifier of a specific object" ,
where the "parent ID (if any)" again breaks into a pair
"Parent ID of the parent (if any)" point "ID of a specific parent . " And so while QObject::parent returns non- nullptr .

The identifier of a specific object can be either the name of the object — the simplest case (if Qt Designer is used, the name of the object will be present), if the object is nameless, then we identify it through the class name and the sequence number.

Example:

MainWindow.centralwidget.tabWidget.qt_tabwidget_stackedwidget.tab.pushButton_ModalDialog

Beta version

It would seem that everything is great, tests are easily created, work on both platforms without any changes (because they are not tied to the pixel-by-pixel arrangement of the elements). But it turned out that everything is not so simple. The modal dialogs came into effect (then they were file open dialogs, but a banal QMessageBox with Yes / No would have caused the problem).

The error was as follows:

• QTest::__ created the necessary instance of the QEvent descendant and using QApplication::notify delivered it to the necessary object;
• QApplication::notify works synchronously, i.e. until the event is processed, he will not return control;
• in the case of a dialog, a new QEventLoop is created, and it starts processing events until the dialog is closed.

Thus, control from QTest::-_ not return until the dialog is closed. But how can the script close it, if the management does not return to it, until the dialogue is closed?

The situation was aggravated by the fact that blocking the behavior of QTest::-_ is just what we need, except, of course, the call of dialogues. It is much easier to insert all sorts of checks into a test if you know what exactly after the line
Test.activateItem('MainWindow.centralwidget.tabWidget.qt_tabwidget_tabbar', 'Tab 5'); the tab 'Tab 5' tab will be activated, rather than, say, five lines after it.

Of course, if we have a blocking call, which we cannot avoid, the obvious solution is to create another thread, but, unfortunately, events related to the GUI should be processed only in the main thread. Therefore, the first solution was the following (pseudocode):

 //addition thread qApp->postEvent(objectInGuiThread, customEventObject); semaphore->tryAcquire(timeout); //gui thread void ClassInGuiThread::customEvent() { QTest::somthing(); semaphore->release(); } 

Those. using QEvent we notify an object that is in the main (GUI) thread that we need to do something, but because objectInGuiThread is in a GUI thread, its ::customEvent method will be called in the context of the thread's GUI, and the threads are synchronized using the semaphore.

Actually, the signal / slots mechanism works in a similar way when calling QObject::connect with the Qt::QueuedConnection , in the case when the signal is sent from one stream and the object that owns the slot is in another stream.

The obvious disadvantage is that you need to correctly select the timeout in the semaphore->tryAcquire(timeout); call semaphore->tryAcquire(timeout); . If, for example, clicking on a widget causes a long operation, and we fall off on timeout, and then continue working, say, from trying to click on a widget that appears only after the completion of a long operation, the result may be unexpected for the author of the test.

The sadness is also caused by the fact that QCoreApplication::loopLevel() made obsolete during the transition from Qt 3 to Qt 4, and it is only available when building Qt 4.x with the qt3support option, and returning its analog QThread::loopLevel() only in Qt 5.5. Those. It is difficult to distinguish the case of "pressing -> long operation" from the case of "pressing -> modal dialogue."

The unobvious disadvantage of this code is how two consecutive events can be processed, the first of which closes the modal window, and the second, for example, simulates pressing on the keyboard. In this case, because closing a QEventLoop created inside QDialog::exec not instantaneous (at least in a QAbstractEventDispatcher implementation using glib ), the QKeyEvent can fall into a QEventLoop class, and then it will not call, for example , triggering the corresponding QAction in the main window.

Therefore, the final version for dealing with modal dialogues was rather complicated:

1. Get the current qApp->activeModalWidget() modal widget, and you need to take into account that this method is not marked as thread-safe, so the code is called in another thread;

   
   

2. Send a message asking to perform the desired click / click in the GUI thread;

   
   

3. Verify with qApp->sendPostedEvents that the message of 2) reached
   and began to be processed;

   
   

4. Repeat 1) and compare the results;

   
   
5. If the new model widget does not appear, then we are waiting for completion 2) without a timeout, otherwise only a user-defined timeout (by default 5 seconds).

In this algorithm, there are also problems (if the new dialogue "appears" or "disappears" for more than 5 seconds), but in this case, using JavaScript, you can set a longer timeout, or even somehow synchronize. Also, the user’s use QEventLoop without creating a modal dialog is not processed. After defeating the dialogues, qt monkey earned quite stable, and the project was handed over.

Github version

Since I still haven’t seen solutions with open source code for this kind of problem; then, being on forced leave, I decided to release the project for free floating. Although the previous work did not object to the publication of qt monkey, I, just in case, rewriting from its scratch, laid out on github.

The current version consists of three components:

1. A library to link with your project, then create a class qt_monkey_agent::Agent somewhere in the main thread;

   
   

2. qtmonkey_app is a console program for communication with 1), using it, for example, you can run your tests in the continuous integration system;

   
   
3. qtmonkey_gui is an elementary GUI for qtmonkey_app.

2 and 3 communicate using stdout / stdin , data streams are structured using JSON . It is assumed that qtmonkey_gui can be easily replaced by a plugin for your favorite IDE.

If there are people who are interested in this project, then it is easy to find by the words "qt monkey" on github. Pull requests are welcome.