How does the testing of the developers KOMPAS-3D

Recently, a new version of CAD KOMPAS-3D v17 was released, but until the very final release, changes were made to the system, testing continued. KOMPAS-3D team from the ASCON development center in Kolomna tells about what tests passed the new KOMPAS-3D before reaching the users.

Each new version has to overcome many stages of testing, in which almost everyone takes part: programmers create unit tests, analysts conduct usability testing, an automated testing team develops scripts using the user interface, API and special test functions. The same team is responsible for testing the performance of converters and an extensive database of user documents.


"Valley Dali" by Dmitry Verba

When testing such a complex system as KOMPAS-3D, it is impossible to do without manual testing. All testers who perform manual testing have design experience in manufacturing and know firsthand how and why users use this or that KOMPAS-3D functionality.
')

Manual testing

Evgeny Filimonov, 3D testing engineer:

We look at KOMPAS-3D from the user's side, that is, we actually design it ourselves. When testing any operation, for example, Extrusion, we check all possible ways to perform it; We consider the use cases in combination with other objects: separately, extrusion can work correctly, but no longer in arrays or versions. In general, we come up with a variety of scenarios - testers have a well-developed imagination. At the same time we try to keep the scripts close to user ones. Although sometimes it is useful to consider and exotic cases.


Incorrect construction of rounding (clickable image).

In v17, there have been such dramatic changes (relative to previous versions) that the work on coding and on testing was much more than usual. Special attention from our side was required, for example, by the operation By sections, to which significant improvements were made, including those related to new mathematical algorithms. Testing of this operation goes on throughout the development version.

How it works: test plans.

When the tester starts to work, he draws up a test plan. Consider a simplified example - a segment. It can be built by two points, by length and angle; a segment can be built between objects, in different layers, defined by the style of the curve. When building a segment, editing or deleting a tester, all these nuances are described in a test plan. The plan contains several parts: the functionality itself (for example, construction by length and angle), protection (no license — construction is not performed), printing (when printing, the segment should be displayed in accordance with the specified curve style), and so on. The test plan is made out in the form of a “tree of thoughts”, and the tester is guided exactly by it when checking the system operation. However, the test plan covers only the main scenarios, it is impossible to foresee 100% of the use cases in it, since there may be an infinite number of them.

User Interface Testing

Ekaterina Rodina, interface testing engineer:

The interface is the “face” of the program. Unlike 2D or 3D functionality, where specific operations are tested, the interface has to be checked throughout the system, since KOMPAS-3D must save its face when performing any operation (command), in any of the components. A good interface provides a comfortable perception and work in the system. It should look neat so that all texts are clearly displayed on the screen, the icons are well scaled. Convenience is checked by the location of the buttons, the number of clicks that need to go to the desired command.

Each operation is formed using a specific set of interface elements - controls. There are about 50 such elements: a field for entering text, a field with a drop-down list, a field for tracking the cursor, etc. Each control is tested separately. We check what the control looks like, whether it is tracking the cursor, whether it gives messages about entering incorrect values, whether it works with text labels and expressions.


Error: the text disappeared (the picture is clickable).

In addition to individual controls, all panels and dialogs are checked: appearance, size, behavior when moving, usability.


The color of the icon became black by mistake (the image is clickable).

During the development process, the interface changes very actively. Sometimes it happens that because of the innovations, what was done earlier “breaks down”. In order not to miss such moments, regression tests are necessary. For this purpose, the tape recorder tool mentioned at the beginning of the article was developed. First, each interface element is checked manually, the errors found are fixed, their solution is discussed. When the work on the element is almost completed, you can write autotests for regression testing.

With the help of the “tape recorder”, not only the work is monitored, but also the position of the interface elements displayed at the current time. In COMPASS there are a lot of teams. In order not to look for what element has changed its position, we have automated this process. Tests go through all the teams in turn, checking that the elements are in their places.


Video of the "tape recorder"

How it works: "tape recorder."

The program records and reproduces the actions of the tester, signaling the appearance of errors. The “tape recorder” sees everything in KOMPAS: interface elements and document objects, understands what state they are in, can manage them. Originally intended to test the user interface, it is successfully used to test functionality. It is used by most of the development staff: testers, programmers, analysts. For v17, this is generally the main test automation tool. In the Magnitfon development department, by complexity and by the resources allocated to it, is the second product after COMPASS - without it, debugging v17 would be very difficult and time consuming.

Automated Testing

Alexey Chirva, head of automated testing group:

One of the areas of automated testing KOMPAS-3D is to ensure the correctness of user documents created in previous versions. We need to make sure that all user experience will behave as expected in the new version: for example, the assembly will not scatter, its color, equipment, specification will not change. The database of documents used in the tests, has more than 700,000 works. Kits are formed from them according to certain criteria.

The main scenarios for checking the correctness of user documents are aimed at identifying changes and errors in geometry.


Errors in the geometry of models (The image is clickable)

Special attention is required for testing the import and export of all supported document formats. We check the correctness of the conversion results (geometry, recording format, file size), perform performance measurements.

When testing the performance of KOMPAS-3D, the execution time of operations, the response time of the interface, the expenditure of system resources, the speed of displaying models and drawings, etc. are monitored.

Mathematics testing

Sergey Biryukov, a mathematician and programmer at C3D Labs:

Not only a new version of KOMPAS-3D has been released, but also a new version of the C3D geometric kernel. Usually, the achievements of the nucleus almost immediately fall into KOMPAS-3D, the bridge between them is very short. The functionality that we are adding now can be immediately tested in COMPASS. In addition, certain kernel methods cannot be fully tested without the use of KOMPAS, for example, testing the conjugation of geometric elements. Kernel testing inside the C3D Labs unit occurs exclusively in automatic mode. The first stage of testing is unit tests that are run during the day with each new kernel build at each revision. In addition, tests from a small set of models are tested three times a day, checking the construction of associative views, model rebuilding and conversion from various data formats. The detected errors are most often corrected in the afternoon, so that by the evening they can deliver the most clean, without breakage, revision (code change) of the kernel. At night, a large regression test is run on the internal server on the basis of 400,000 models in the c3d core format. And the next morning, all programmers at C3D Labs are sent a message with the results of the tests.


AST (automatic testing system) core C3D performance report

The convenience of our testing system is that in case of unsuccessful results, all the changed models are automatically detected, and for each of them a revision is detected, on which the changes took place. Therefore, in the generated morning letter already contains a complete list of these models, indicating the number and author of the revision that caused the changes. If a large test passed without errors, then the revision on which it was performed is considered a high-quality working version of the kernel. We collect KOMPAS-3D with a new kernel and transfer it to the KOMPAS developers. They test this assembly on a basic test, and if everything is good, this version of mathematics shifts in KOMPAS-3D.

Application Testing

Larisa Ivanova, Application Testing Engineer:

The main task in the v17 version was to “make friends” of the application with the new KOMPAS-3D interface. To make this acquaintance go smoothly, we joined the testing process long before the developers started to adapt the applications. First you had to make sure that the applications are connected. Then, testing of each application began separately: connecting, launching each command, displaying icons, starting processes, appearing dialogs, testing performance using standard scripts -


Application Equipment: Welded connections. All parameters are selected, but there is no “Create Object” button (The image is clickable).

Application functionality should not have been worse than in previous versions.


Appendix Dimension chains. Command icons are not displayed (Image is clickable).

We systematized all the errors - we separated the errors “induced” by the modifications in the interface from the errors of the applications themselves - and sent them for revision. And only when the applications became operational, their developers joined the work on the new interface.

After the developers completed their part, functional testing of applications began - in essence, the development of real projects, including the design and receipt of design documentation. We manually launched all application commands, checked the operation of each button, compared it with the work in previous versions. Upon detection of problems, as well as the appearance of new features and the identification of features, they notified application developers through the Release Management System, which we developed for previous versions of KOMPAS, but was fully evaluated in v17.


Application Equipment: Metal structures. Team Special cutting. The result of the operation does not match the specified parameters (The image is clickable).

Yes, the methods of work and the interface in KOMPAS-3D v17 have changed dramatically. But thanks to the ASCON testing team, which has tried it on itself more than once, users can rest easy - KOMPAS-3D v17 will solve any of their tasks.