Use of automated verification tools for software testing in accordance with the standards DO-178B / C (КТ-178В), EN 50126, IEC 61508, ISO 26262, FDA, IEC 62304

Introduction from the author of the post

Having experience in software development for critical systems for more than 8 years, I want to acquaint the community with some materials related to software development and verification for critical systems (aerospace, medicine, transport, and industry). Having consented to the translation and adaptation of a number of interesting articles from foreign colleagues, I decided to use this resource. I would be glad if the article will interest our community. The article uses materials from the company Vector Software, Inc. and an example of using the VectorCAST automated verification toolkit is considered.
In this article, we focus on the standards DO-178B / C (-178), but this article is also relevant for use in accordance with the standards: EN 50126, IEC 61508, ISO 26262, FDA, IEC 62304.
I will answer questions in comments or in a pm.

Introduction

In this article, we will look at how the VectorCAST firmware testing platform can be used to achieve the objectives of the software verification process defined in Section 6.0 of the DO-178B and DO-178C (KT-178B) standards. equipment and systems for the certification of aviation equipment ". The document also highlights the differences between the two standards in part 12 of the Process of qualifying instrumental software.
')

What are the standards DO-178B and DO-178C

DO-178B was first published in December 1992 by RTCA, Inc. The document outlined the basic principles applied by organizations that develop onboard equipment and certification bodies, such as the FAA, EASA, TransportCanada. The development of DO-178B was the result of the joint efforts of RTCA and EUROCAE, the latter published a document called ED-12B. Software development teams at certification authorities (CAST) developed clarifying documentation after initial publication.

DO-178C will replace DO-178B as the main document, according to which certification authorities approve all commercial aerospace systems that have software in their composition. DO-178C revises DO-178B in the light of experience and information gathered in the light of avionics software development. A new document called DO-178C (ED-12C) was prepared in November 2011, approved by RTCA in December 2011, and became available for use in January 2012.

Both the DO-178B and DO-178C (DO-178B / C) both govern the process to be followed when developing onboard systems. One of the key requirements in the software verification process in accordance with the DO-178B / C standard is to achieve a structural code coverage in conjunction with testing high and low level software requirements.

Based on the assessment of the safety of the system, categories of failure states were established. These categories determine the level of software integrity required for safe operation in the avionics industry. Document DO-178B / C classifies software into 5 levels of criticality depending on its impact on the functioning of the entire system. The table below demonstrates the relationship between the category of the abandoned state and the goal of structural coverage in accordance with DO-178B / C.

Level	Determination of the failure state	Type of structure coating
A	Software can lead to a catastrophic failure condition of the system and the aircraft	Modified coverage of conditions and branches; Coating of branches; Cover instructions;
B	The software may lead to an emergency failure condition of the system and the aircraft	Coating of branches; Cover instructions;
C	The software can lead to a complex failure condition of the system and the aircraft	Cover instructions;
D	The software can lead to a failure condition of the type “complication of flight conditions” of the aircraft	Not required
E	The software may lead to a failure condition without affecting the operational capabilities of the aircraft.	Not required

What is VectorCAST?

The VectorCAST toolbar supports information gathering and reporting on structural code coverage at all levels specified in DO-178B / C, including level A, and the creation of all the test artifacts required for testing using DO-178B / C. The VectorCAST product line consists of five complementary technologies:

• VectorCAST / C ++ / Ada - These tools automate the process of testing source modules written in C,
  C ++ or Ada - for both unit and integration testing. Coverage data can be combined with VectorCAST / Cover data to achieve 100% coverage.
• VectorCAST / Cover - Designed to create code coverage artifacts during the execution of a functional and / or system test for C, C ++ and Ada.
• VectorCAST / RGW - Provides a link between requirements and VectorCAST test scenarios.
• VectorCAST / RSP - Expands VectorCAST / C ++ or VectorCAST / Ada, providing the ability to test real-time applications on target platforms and / or in a simulation environment.
• VectorCAST / Manage - Used to automate activities in regression testing for unit and integration testing.

The tools also support the creation and management of test scripts to confirm the fact of testing low-level software requirements.

How does VectorCAST support DO-178B / C

The objectives of the software verification process are defined in Section 6.0 of the DO-178B / C standard.

Section 6.4 - Software Testing

This section establishes three types of testing:

Hardware / Software Integration Testing
This type of testing is used to meet high-level requirements and is performed on the target hardware using a full-fledged image of the executable program. In this type of testing, VectorCAST / Cover is used, which registers code coverage during the execution of system and / or functional test procedures.

Software Integration Testing
Integration software testing confirms interconnectedness of elements. Testing at this level is performed using VectorCAST / C ++ or VectorCAST / Ada to test multiple software components at a time. A fully-fledged test program is automatically generated to support this type of testing, and software requirements can be flagged in separate test scenarios to ensure that all requirements are tested.

Low level testing
It is used to test low-level requirements and is usually performed by a series of unit tests that allow you to isolate a single module of source code. During this phase of testing, VectorCAST / C ++ and VectorCAST / Ada are used.

Section 6.4.1 - “Test Environment”

This section states that more than one test environment may be required to achieve software testing goals. While testing the application on the target platform will be considered an “ideal” environment, it may not be feasible to build and collect demand-oriented coverage and structural coverage in a fully integrated environment. You may need to perform testing on small isolated components in a simulated environment.

VectorCAST fully supports testing on the target platform or using its simulator, usually provided by the compiler vendor. The structural coverage of the isolated components tested can be combined with the coating collected during the full integration testing to demonstrate the cumulative presentation of the coverage metrics.

VectorCAST test scripts are supported regardless of the source code for the DDT testing approach. This technique allows you to perform tests on the developer's machine, in the simulator, or directly on the target platform in a fully automatic mode.

Section 6.4.2 - “Selection of test cases based on requirements”

DO-178B / indicates that software verification should be based on requirements, as opposed to source-based verification. Requirements-based tests require testers or developers to create input data to execute code that first take into account requirements. This type of test takes two forms: test cases in the allowable input range and robust tests.

Section 6.4.2.1 - “Test Examples in the Allowable Range”

The purpose of testing the normal range of values ​​is to demonstrate the ability of the software to respond to normal input data and operating modes. More precisely, the real and integer values ​​of the input data should be checked. VectorCAST provides the ability to set these values ​​using the GUI or using a script, as shown in the figure below:

For functions related to time, repeated iterations of the code should be performed to ensure the correctness of the characteristics of the function being tested.

VectorCAST provides a simple and convenient way to iterate tests over time. This feature is called "compound testing." It allows the tester to perform an individual test many times or multiple tests over a period of time, so that the data remains constant during execution.

For requirements expressed by a logical equation, you may need to perform MC / DC. VectorCAST provides for MC / DC-level coverage and equivalence matrices.

Section 6.4.2.2 - “Robust Test Examples”

The goal of robust test scenarios is to demonstrate the ability of the software to respond to values ​​and conditions outside a specified range. VectorCAST supports testing "out of range" values ​​or abnormal values ​​of any type. To do this, use the option to disable the normal range checking. The following figure shows an example of setting the value “out of range”.

Section 6.4.3 - “Test Methods Based on Requirements”

Test methods (tests) based on requirements consist of:

• Integration testing of hardware / software
• Integration Software Testing
• Low level testing

VectorCAST supports all three levels of testing. For integration testing of hardware / software, VectorCAST provides structural coverage capabilities for projects developed to levels A, B and C. This testing can be performed on a target platform or using a target platform simulator.

For software integration and low-level testing, VectorCAST allows you to construct a test program that is executed automatically, providing testing of individual components or a combination of components. During these testing phases, a structural coating is assembled, which is combined into one unit to assess the completeness of testing.

Section 6.4.4 - “Test Coverage Analysis”

Test coverage analysis includes verification of structural coverage and coverage analysis based on requirements. The first step is to analyze the test scenario to confirm that all requirements are related to a specific / specified test scenario. The next step is to check the execution of tests based on the requirements.

VectorCAST code coverage analysis determines which source code lines (instructions), which source code branches (branch), or which equivalent pairs (MC / DC) were executed by a series of test scenario data. Reports show you the completeness of your test suite. After analyzing the untested code, you can easily go back and develop scripts to test these parts of the code.

Section 6.4.4.1 - “Analysis of Test Coatings Based on Requirements”

VectorCAST / RGW (RequirementsGateway) allows you to track the association of software requirements with test scenarios to meet the requirements of coverage analysis.

VectorCAST / RGW (RequirementsGateway) allows you to create a data flow between requirements management tools and the VectorCAST testing tool. Using a simple and intuitive interface, developers can quickly link requirements to VectorCAST test scripts.

If test scripts were run at least once, traceability metrics can be viewed from VectorCAST or the requirements management tool. The user has full control over which attributes VectorCAST will give back to the requirements database. Data such as “test name”, “test result” (success / failure / no result) can be associated with the database attribute of the requirements selected by the user.

Section 6.4.4.2 - “Structural Coverage Analysis”

The purpose of this analysis is to determine which part of the code was not executed during the tests based on the requirements.

VectorCAST allows you to calculate the effectiveness of your testing efforts and determine which parts of the application were performed during testing based on testing. This is a handy tool for analyzing the completeness of your tests, excluding the possibility of releasing an application with code that has not been tested. VectorCAST allows you to analyze any part of your application or the application as a whole. For each analyzed file, VectorCAST creates a source code viewer that includes the following information:

• A coverage summary is a color-coded view of your source code, which determines whether the code is fully, partially covered, or uncovered.
• The metrics summary provides a list of source code complexity (McCabe) and the current source code coverage status for each subroutine.
• The conditional coverage analysis shows all control paths for each subroutine.
• MC / DC is designed for the RTCA DO-178B / C standard for software level A.

The MC / DC analysis states that each sub-condition can independently act on the result of the whole condition. To confirm this, you must be able to record the values ​​of the result of the condition, as well as the values ​​of each of the sub-conditions. VectorCAST / Cover collects this information in two formats: a commentary-listed listing of source code and pairwise comparison matrices for each logical condition.

Differences in structural coverage for level A

There is a significant difference between the standards DO-178B and DO-178C, which refers to the structural coverage of software at level A. In all cases not related to level A, the analysis of structural coverage is performed on the source code. For level A, if the compiler creates an object code that is not traced directly to source code operators, additional verification should be performed to ensure the correctness of such code.

The meaning of the phrase “additional verification must be performed” differs in DO-178B and DO-178C. DO-178B states that additional verification should be performed on the object code. DO-178C removed this requirement and simply states that additional verification must be performed.

Section 12.1.3 - “Changing the Terms of Use or Development Environment”

The use and modification of a previously developed application may include a new development environment, a compiler, a target processor, or integration with software different from the one used in the original.

When using different compilers or different sets of compiler options, as a result we will get different object code, and the results of the verification activities of the previous software may be unsuitable for the new application.

Target processor or compiler changes

It is important to note that all tests developed with VectorCAST should be considered as independent of the target platform. VectorCAST tests correspond to the data used by the application, but not by the target processor itself. When an application is transferred to a new target processor, VectorCAST tests can simply be re-executed in a new target environment.

The same approach applies when a new compiler is used. The mechanism of the regression test in VectorCAST will rebuild all running test programs using the new cross-compiler, and all existing tests will be re-executed to verify the test results in the new environment. For new target processors or a compiler environment, it may be necessary to re-qualify the VectorCAST tools for the new environment.

Tooling qualification

Although the overall tool qualification process has remained unchanged, there are several differences between DO-178B and DO-178C in describing the tool qualification process.

Both standards establish that the software verification tool qualification is necessary when the verification process defined in the documents is ignored, truncated or automated using a software tool with unconfirmed output.

VectorCAST refers to automating the verification process as defined in DO-178B and DO-178C. The qualification of the software tool is approved by section 12.2 of the DO-178B and DO-178C standards.

Section 12.2.1 of DO-178C states: “The tool is qualified only for use in specific systems where the intentions to use the tool are approved in the Program Aspect Certification Plan (PSAC) supporting the system. If a tool that previously qualified for one system is intended to be used on another system, it must be re-qualified within the context of another system. ”

Difference between standards in tool qualification

The tool qualification guide for verification software in accordance with DO-178B is quite simple. The supplier of the instrument is required to have two documents: operational requirements for the instrument (TOR) and data on the qualification of the instrument (TQD). These documents define software requirements in normal operating conditions and software-related tests and test results.

For DO-178C, the tool qualification level (TQL) must first be determined to assess the impact of the tool in the software life cycle. Section 12.2.2 establishes “criteria levels” defining the effect of the instrument. Three levels of criteria are defined:

• Criterion 1: A tool whose output is part of the on-board software and, therefore, it may introduce an error;
• Criterion 2: A tool that automates the verification process (s) that may not detect errors and whose output is used to confirm the exception or truncation:
  
  • Verification process (s) with the exception of automated software tools, or
  • Development process (s) that may affect on-board software.
• Criterion 3: A tool that, within its intended purpose, may not reveal an error.

Based on these criteria levels and software levels (from A to D), you can set the skill level (TQL) that is defined in the table below:

Responsibility for qualifying software verification tools (such as VectorCAST) is usually divided between the main counterparty and the tool supplier. Based on the assessment of the safety of the system, categories of failure states were identified. These categories establish the level of software integrity necessary for the safe operation of on-board electronic equipment.

To further guide the process of qualification of instruments, an RTCA DO-330 document, “Recommendations on the qualification of a software tool,” was developed, interrelated with DO-178C.

VectorCAST tool qualifications

To qualify VectorCAST as a software verification tool, Vector Software publishes the following information for each project that uses the tool:

Performance requirements for the tool

The Instrument of Operational Requirements for a Tool (TOR) includes:

• Description of the functionality of the tool in verifiable requirements
• Project operating conditions
• Configuration management process
• The method of achieving verification, according to which VectorCAST has satisfied testing for the requirements

Instrument qualification data

Instrument qualification (TQD) data includes:

• Test data and tool test results

The tool qualification process usually involves interaction with both the qualifying user and the appropriate certification authority and / or DER.

VectorCAST tool qualification process

VectorCAST delivers a preliminary version of qualification materials to the customer. Additionally, a test suite is transferred so that a qualified user can execute them in the same environment as their project. Vector Software includes changes to documentation and tests, if appropriate. Vector Software sends version 1.0 TQD for final approval.