Toward Replacing Structure-Based Testing with Analysis Vdot Santhanam Boeing

Structure-Based Testing DO-178B measures efficacy of software verification through structural coverage measures DO-178B measures efficacy of software verification through structural coverage measures Level C – Statement CoverageLevel C – Statement Coverage Level B – Decision CoverageLevel B – Decision Coverage Level A – Modified Condition/Decision CoverageLevel A – Modified Condition/Decision Coverage Often this leads to significant amount of time spent achieving structural coverage for the sake of coverage Often this leads to significant amount of time spent achieving structural coverage for the sake of coverage

Certification Contrary to EASA and MoD practices, the FAA has not embraced analysis as a means of software verification Contrary to EASA and MoD practices, the FAA has not embraced analysis as a means of software verification There are no requirements to perform even simple forms of analysisThere are no requirements to perform even simple forms of analysis We’d like to see that culture change because it willWe’d like to see that culture change because it will Reduce the cost of verification Reduce the cost of verification Enhance the reliability of the process Enhance the reliability of the process Improve our confidence in the software Improve our confidence in the software With DO-178C planning starting, the time is rightWith DO-178C planning starting, the time is right

Where Do We Want To Go? Would like to see Would like to see Varying degrees of analysis by assurance level, e.g.,Varying degrees of analysis by assurance level, e.g., Set-use analysis for level C Set-use analysis for level C Range-constraint analysis for level B Range-constraint analysis for level B Exception and memory leak analysis; RMA for level A Exception and memory leak analysis; RMA for level A Alternate form of credit for verification- by-analysis (in lieu of structural coverage testing)Alternate form of credit for verification- by-analysis (in lieu of structural coverage testing)

Research Needed Devise analysis techniques that achieve the same/similar objectives as testing Devise analysis techniques that achieve the same/similar objectives as testing Devise new, more powerful techniques that can surpass testing in cost- effectiveness Devise new, more powerful techniques that can surpass testing in cost- effectiveness Stream-line research toward more pragmatic applications for existing formal techniques (rather than pursue the elusive goal of proving programs correct) Stream-line research toward more pragmatic applications for existing formal techniques (rather than pursue the elusive goal of proving programs correct)