1. Automating Software Module Testing for FAA Certification Usha Santhanam The Boeing Company

2. Automating Software Module Testing for FAA Certification Challenges in code based software testing FAA certification testing Test Set Editor (TSE) tool set and process Regression testing Benefits of TSE Limitations of TSE Summary

3. Challenges in Code Based Software Testing Software complexity and accelerated development schedules make avoiding defects difficult Untested code in user environment Difficulty of accurately capturing and analyzing the sequence of events that occur during program execution

4. FAA Certification Testing Structural coverage is often not achieved through requirements based testing alone Testing each module in isolation provides the desired coverage The three levels of structural coverage to deal with are statement coverage, decision coverage, and modified condition and decision coverage (MCDC)

5. Structural Coverage Statement coverage: execute each source level statement at least once Decision coverage: take each execution branch at least once MCDC: designed to demonstrate the independence of individual conditions that constitute a complex decision

6. TSE Tool Set and Process TSE is designed to achieve shorter turnaround time and cut cost during regression testing TSE is domain-independent and removes the language-dependent test driver chores TSE requires the test engineer to construct the test cases, but the rest of the testing process is automated

7. TSE Tool Set and Process (cont.) Tasks of testing a module for structural coverage are –Identifying inputs and outputs of a module (or collection of modules) –Selecting test cases and tagging them with the requirements they address –Constructing a test driver and stubs for subprograms called by the module under test

8. TSE Tool Set and Process (cont.) –Compiling and running the tests –Analyzing coverage TSE assists the tester with all these tasks except the selection of the test cases.

9. Example: Procedure Get_End in Package Io package Io is Debugging : Boolean := False; procedure Get_End; end Io; with Text_Io; package body Io is K : constant := 1024; Max_Line_Length : constant := 16*K; Line : String (1.. Max_Line_Length); procedure Error (Msg : String) is begin null; end Error; procedure Get_Line is begin null; end Get_Line; procedure Get_End is begin if Line (1.. 4) = ".end" then Get_Line; else Error ("Missing.END, assume here."); end if; end Get_End; end Io;

11. TSE_Tools.tcl

16. Regression Testing All test cases are self-checking A batch file is used to run the test files through the TSE testing process A batch log file is used to check for failures Failed tests are examined and justified or updated

17. Benefits of TSE Tests can be developed by engineers who are not necessarily expert programmers in Ada TSE includes a tool to analyze the source code and identify inputs and outputs of a given module Provides coverage information

18. Benefits of TSE (cont.) TSE tests are easy to maintain due to uniformity of test definition files TSE tests are repeatable with no manual interaction necessary TSE tests can be executed on host or target platform using different support packages

19. Productivity Improvements Based on our experience we have seen a range of 1:5 to 1:10 productivity advantage. – Depends on the complexity of the module

20. Estimate of TSE Based Testing We have developed a tool that estimates the number of hours needed to test a module with TSE Based on Statement count, number of Ifs, Cases, Loops, Exits, Logical operators, short circuit control forms, and stub calls The Get_End procedure estimated testing time is 7 hours

21. Estimate of Manual Testing Estimated size of the get_end test driver is 90 SLOC –Including main driver and stubs Programmer productivity for the test code development is 2 SLOC per hour (Hihm) This yields the productivity improvement of 1:6.5 for get_end procedure

22. Limitations of TSE It is not possible to construct one generic module test to verify all instantiations –Handled on an instantiation by instantiation basis Tasking is difficult to test because tasks run in independent parallel threads –Each subprogram can be tested independently regardless of the number of tasks sharing it

23. Limitations of TSE (cont.) For modules using private type parameters or returning private type, construction of external test driver is awkward –Embed the test driver in the body of the package that defines the limited private type Nested subprogram testing not available –Requires the test driver to be nested –Must be tested in the context of enclosing subprogram

24. Summary of TSE Benefits Minimizes the need for programming knowledge Cuts testing effort several fold over manual approach Minimizes cost of retesting Allows us to meet FAA/DO-178B requirements in cost effective manner