Geekly Articles each Day
📜 ⬆️ ⬇️

How to evaluate firmware testing tools

Introduction from the author of the post


Having experience in software development for critical systems for more than 8 years, I want to acquaint the community with some materials related to software development and verification for critical systems (aerospace, medicine, transport, and industry). Having consented to the translation and adaptation of a number of interesting articles from foreign colleagues, I decided to use this resource. I would be glad if the article will interest our community. The article uses materials from the company Vector Software, Inc.
I will answer questions in comments or in lichku

What is your testing tool?


Over the past few years, the market for automated testing tools has been filled with tools claiming to perform the same function — automated testing. Wikipedia lists 38 test environment assessment tools for C / C ++ programming languages ​​only. Unfortunately, potential users, studying the description of these products, as well as their simplified demos, can conclude that most of the tools are almost the same.

The purpose of this document is to provide engineers with information that needs to be taken into account when evaluating automated software testing tools, especially in terms of automated dynamic testing tools.
')

You cannot evaluate the testing tool by reading its specification.


All specifications look quite the same. The keywords are the same: “industry leader”, “unique technology”, “automated testing”, “best practices”. Screenshots are alike: histograms, structural diagrams, HTML reports and percentages. All this casts boredom.



What is software testing?


Everyone who has ever been involved in software testing knows that it consists of many components. For simplicity, we will use three terms:

Each one conducts system testing to varying degrees, during which it performs some of the actions that the end user will perform. Notice, we are talking about some actions, not about "all possible." One of the most common causes of errors in an application is the appearance of unforeseen and, therefore, unexplored combinations of input data.

Not many people do integration testing, and very few do unit testing. If you performed integration or unit testing, you know how many test codes you need to create in order to isolate a single file or group of files from everyone else in the application. At the most demanding levels of testing, there is a situation where the amount of written code for testing exceeds the amount of code being tested. As a result, such testing levels are applied, as a rule, to applications that are critical for functionality and safety in such areas as aviation, medical devices, and the railway industry.

What does automated testing mean?


It is well known that the process of integration and unit testing by hand is extremely expensive and time consuming; as a result, every instrument appearing on the market is “trumped” as if using automated testing. But what does this mean - “automated testing”? Different people understand the word “automation” in their own way. For many developers, automated testing means the ability to click a button and get a result - a “green tick”, meaning that the code is correct or a “red cross”, meaning an error.

Unfortunately, this tool does not exist. And if it existed, would you want to use it? Think about it. What would it mean if the tool showed that your code is “in order”? Would it mean that the code is flawlessly formatted? Maybe. Would it mean that the code conforms to your coding standards? Maybe. Would it mean that the code is correct? Definitely not!

Fully automated testing is unattainable and, in principle, undesirable. Automation should be focused on the algorithmic and time-consuming parts of the testing process. This will free software developers from constantly creating increasingly sophisticated and complex tests.

When evaluating testing tools, a logical question arises: what degree of automation does this testing tool provide? This is the main area of ​​uncertainty when an organization attempts to calculate the return on investment in testing tools.

Analysis of testing tools


Testing tools generally provide a variety of functionality. Since different tools are provided by different companies, the functionality may differ when using another tool. In accordance with the common criteria system, we chose the following names for components that may exist in the evaluated testing tools:

The following sections will describe in more detail how the above components should be evaluated in the testing tool in question.

Types of test tools / automation levels


Due to the fact that none of the testing tools includes all the functionality of the testing components described above, and due to significant differences between the tools according to the degree of automation, we have created the following broad classification of testing tools. The estimated instrument can be attributed to one of the following groups:


Implicit testing tool differences


In addition to comparing tool features and automation levels, it is important to evaluate and compare the testing approaches used. For example, when you create a test project, the tool simply loads the files into its integrated development environment, but does not create a test program or test script until you take certain actions.

As a result, there may be hidden flaws in the testing tool. Therefore, it is important not only to load your code into the testing tool, but also to try to build some simple test scripts for each method in the class that you are testing. Does the tool build a complete test program? Are stubs created automatically? Can you use a graphical user interface to define parameters and general data for test scripts, or do you have to write code manually, as if you were testing manually?

Similarly, support for the target platform differs in the tools used. Be careful if the provider says: "We support all compilers and targets." This means only one thing: “You have to do all the work so that our tool works in your work environment.”

How to evaluate testing tools


The next few sections will describe in detail what information you should carefully consider when evaluating a software testing tool. Ideally, you should confirm the information obtained in practice by testing each of the proposed tools.

Due to the fact that the subsequent material in this documentation will be of a technical nature, we would like to introduce some symbols. Each section has a title that describes the question to be considered, a further explanation of why this issue is important, and, finally, “key points” as a summary of the material reviewed.

Also, speaking of the conventions, we must pay attention to the terminology. The term “function” refers to both the function of the programming language C and the class method of the programming language C ++, the term “module” refers to both the file of the programming language C and the class of the programming language C ++. Finally, it must be remembered that almost all testing tools in some way support the points mentioned in the “key points”, your task is to evaluate the level of automation, ease of use and degree of support.

Analyzer and code generator


It is relatively simple to build an analyzer for the C programming language; much harder to build a full-featured analyzer for the programming language C ++. One of the questions that needs to be answered in the assessment process is how reliable and well thought out analyzer technology. Some suppliers of testing tools acquire and resell licensed analyzer technology, others have own-made analyzers. The reliability of the analyzer and the code generator can be checked with the help of complex code constructions - typical code samples that you will use in your project.

Test driver


The test driver is the main testing control program. Let's give a simple example of a driver that will test the sine mathematical function from the standard library of the C programming language:

#include <math.h> #include <stdio.h> int main () { float local; local = sin (90.0); if (local == 1.0) printf ("My Test Passed!\n"); else printf ("My Test Failed!\n"); return 0; }

Although this is a fairly simple example, a “manual” testing tool may require you to type (and debug) this small piece of code manually, a semi-automated tool will produce some type of scripting language or a simple GUI to enter the sine argument. The automated tool will contain a full-fledged GUI for building test scenarios, an integrated code coverage analysis, an integrated debugger program and an integrated software deployment capability on the target platform.

I wonder if you noticed that there is an error in this driver. It lies in the fact that the function of calculating the sine actually uses the angle as the input of the radians, not degrees.

Using stubs for dependent functions


Building substitutions for dependent functions is necessary when you want to control the values ​​returned by the dependent function during the test. The use of stubs is an important part of integration and unit testing, as it allows you to isolate the code being tested from other parts of the application and it is easier to test the subsystem of interest.

Many tools require manual creation of a test code for stub to perform something more complicated except for returning a static scalar value (return0;)

Test data


There are two main approaches used by semi-automated and automated tools in the implementation of the test scenario. The first is the data-driven architecture, the second is the architecture of the unit test.

For a data-driven architecture, a test program is created for all modules under test and supports all the functions defined in these modules. When a test is run, the tool supplies data to the information flow, such as a file descriptor or a physical interface, such as a UART.

For a single test architecture, each time a test is run, the tool creates a test driver and compiles it into an executable program. It is important to note a couple of points; first, the creation of additional code required for a single test, as well as subsequent compilation will take more time during the test; secondly, as a result, you create separate test programs for each test case.

This means that the evaluated tool can work normally for certain nominal scenarios and does not work correctly for more complex tests.

Automated test data generation


Various automated tools provide a degree of automation in the creation of a test script. Different approaches are used for this, some of them are shown in the table below:

When thinking about the automated creation of test scripts, it is important to remember what purpose they serve. Automated tests are good for assessing the robustness (reliability) of an application code, but not for correctness (even if they provide a high level of code coverage). For correctness, you must create tests based on what is expected of the application (requirements), and not what it does (code).

Compiler integration


Compiler integration has a double meaning. On the one hand, integration allows you to automatically build test program components without the need for the user to enter compiler parameters. On the other hand, integration allows the test tool to accept for processing any language extensions specific to the compiler used. A common practice is that cross-compilers support extensions that are not part of the C / C ++ programming language standards. Some tools define such extensions as the empty string. This is a very rough approach and its main disadvantage is that it changes the object code generated by the compiler. For example, consider the following global external variable with the GCC attribute:
 extern int MyGlobal __attribute__ ((aligned (16)));

If the tool in question does not support the attribute when defining a global MyGlobal object, then the code will behave differently in the testing process and in working condition, since the alignment will not be the same.

Support for testing on the target platform


In this section, we will use the term “toolkit” as applied to the development environment, including cross-compiler, debugging interface (emulator), target platform, and real-time operating system (RTOS). It is important to find out if the tool in question has excellent integration with your toolbox and understand what needs to be changed in the tool for migration to another toolbox.

It is also important to find out the level of automation and reliability of integration with the target platform. As mentioned earlier, if the provider says: “We support all compilers and target platforms,” this means only one thing: “You have to do all the work for our tool to work in your production environment.”

Ideally, the tool you have chosen should perform “one button click” testing, and all the difficulties of downloading software to the target platform and collecting test results on the working machine are hidden behind the message “Running test”, so that the user does not need any special actions .

An additional challenge to testing on a target platform is the issue of hardware availability. Often, the hardware is developed in parallel with the software or there is a limitation of access to the hardware. An important point is the ability to start testing in your own environment and later make the transition to a real target platform. Ideally, the test tool artifacts should be hardware independent.

Test Script Editor


Obviously, it is in the test script editor that you will spend most of your time when using the testing tool. If the tool truly supports the automation of the listed items mentioned above, then the amount of time spent on setting up the test environment and communication from the target platform will be minimal. Remember what we said at the beginning — our goal is for engineers to use their time to create more sophisticated and more complete tests.

An important question that needs to be answered when evaluating is how difficult it is to set test input data and expected values ​​for non-trivial constructions. The existing tools on the market provide a fairly simple way to set scalar values. For example, is the evaluated tool equipped with a simple and intuitive way to construct classes? What about setting up an STL container like vector or map. Such nuances should be evaluated in the test script editor.

Code coverage


Most semi-automated and all automated testing tools contain an embedded code coverage tool that allows you to see metrics that show the amount of software code that is executed when you run your test scripts. Some tools provide this information in tabular form. Others give a flow graph, and some give a commentary listing of the source code. Although the table is a good representation of a summary, but if you are trying to achieve 100% code coverage, a listing with comments is the best way. This listing shows the source file with staining for the coated, partially coated and uncoated structures. This makes it easy to see which additional test scenarios are needed for 100% coverage.

It is also important to evaluate the impact of the toolkit, since additional source code is added to your software. There are two points that need to be taken into account: one is an increase in the size of the object code, the second is additional costs during program execution. It is important to understand whether your software has limitations in terms of memory or program runtime (or both). This will help focus attention on what is more important for your software.

Regression testing


When choosing a testing tool, we must remember two main goals. The first goal is to save testing time. If you have read these words, then you agree with us. The second goal is to ensure the effective use of the tests created during the software life cycle. This means that time and money invested in the creation of tests should result in the possibility of multiple use of tests (when software changes over time) and should ensure the ease of configuration management of these tests. The main thing is to evaluate in the instrument you are interested in, which individual entities should be saved for passing the same tests in the future, and how the restart is controlled.

Making report


Most tools provide a reporting function in a similar way. At a minimum, they should produce an easy-to-understand report showing input data, expected output, actual output, and a comparison of expected and actual values.

Integration with other tools


Regardless of the quality or practical use of any individual instrument, all instruments must operate in an environment that unites systems from different manufacturers. Large companies have spent a huge amount of time and money buying small companies that have developed a tool that performs “everything for everyone”. For these mega tool sets, it is characteristic that "the total amount is less than the sum of the parts." Companies often take 4-5 small but efficient tools and integrate them into one bulky and unusable tool.

In addition to integration with the toolkit, which we have already considered, the most useful types of integration for testing tools are integration with static analysis, configuration management, and requirements management tools. Everyone wants to keep the testing tool artifacts in the configuration management system (to reuse them), and most people would like to trace the requirements to the test scripts.

Additional desirable characteristics of the testing tool


We have completed the review of the “Analysis of Testing Tools” section. The previous sections described the functional content that is found in any tool considered automated. In the following sections, we will list some desired (and less common) characteristics, while at the same time arguing the importance of these characteristics. These characteristics may have different levels of applicability to your project.

True Integration Testing / Multi-Module Testing


Integration testing is a continuation of unit testing. It is used to test the interfaces between modules and requires you to combine modules that perform a specific functional process. Many tools are declared as supporting integration testing by linking object code for real modules using a test program. This method constructs many files within the test program being executed, but does not provide the ability to activate functions within these additional modules. Ideally, you should be able to activate any function in any module in any order within a single test scenario. Testing interfaces between modules as a whole will reveal many hidden assumptions and bugs in software. In fact, integration testing can be the first step for projects that do not have a unit testing history.

Using dynamic stubs


Creating dynamic stubs means the ability to dynamically enable and disable their own stub functions. This allows you to create a test for an individual function, silencing all other functions (even if they exist in the same module as the function being tested). For very complex code, this is the most important feature that greatly simplifies the execution of testing.

Testing at the library and application level


One of the difficult problems with system testing is that test impact directed to fully integrated software may require the user to press buttons and switches or print on the console. If the application is embedded, the input will be even harder to control. Suppose you can submit external influences on a functional level, similar to the integration testing way. This will allow you to build a set of test scenarios, based only on the software API.

Some of the most advanced tools allow you to test this way. An additional advantage of this test mode is the absence of the need for source code to test the application. You just need an API definition (usually header files). This methodology provides the tester with an automated and scripted way of performing system testing.

Agile testing and development through testing (TDD)


Development through testing implies the following: instead of first writing the application code and, secondly, testing the module, you first create the tests before writing the application code. This is a new popular approach to development - to develop a test first. Your automated tool should support this testing method if you plan on using a flexible development methodology.

Bidirectional integration with requirements management tools


If you are concerned with linking requirements with test scenarios, then integrating a testing tool with a requirements management tool is desirable. If you are interested in this feature, it is important to note the bi-directionality of the interface; when requirements are marked in test scenarios, test scenario information, such as the test name and the pass / fail status, is transferred to the requirements database. This allows you to get a sense of completeness testing requirements.

Tool qualification


If you work in a regulated environment, such as civil aviation or the manufacture of class III medical devices, then you must qualify the development tools used to build and test your application.

Qualification involves documenting the purpose of the instrument (its operational requirements) and the results of tests proving that the instrument works in accordance with these requirements. Ideally, the supplier should have these materials in finished form, as well as the history of customers who used the qualification data in their industry.

Conclusion


Remember that virtually every firmware testing tool supports the points mentioned at key points in one way or another. It is important to understand how automated it is, easy to use, and evaluate the usefulness of support.

Source: https://habr.com/ru/post/182544/

More articles:


All Articles