INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 1 Experience Report: Error Distribution in Safety-Critical Software & Software Risk Analysis Based on Unit Tests Stephan Ramberger Thomas Gruber ARC Seibersdorf research GmbH

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 2 3 Classes of Impairments (J.-C. Laprie)  Faults = detected or assumed reason of an error  Errors = portion of a system state which necessarily leads to a subsequent failure: The error is an indicator for a failure that occors or has occurred.  Failures = a deviation between the delivered system function and the required one

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 3 Error Cost – Example Ariane 5 Flight 501  June 4, 1996: Maiden flight of Ariane 5  37 secods after ignition horizontal speed = ,0 internal units  Exception in „idle“ subsystem by 16 bit – integer conversion (overflow)  Subsystem software had been tested only for the smaller Ariane 4 model.  Redundant computer had the same problem 72 ms earlier – switched off  Main controller interprets diagnostic data as trajectory data  Unreasonable control commands to the engines  Purposeful blowing-up after 40 Seconds at m altitude  Damage:  € launch cost  € cluster satellites  € subsequent improvements

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 4 Testing: Cost and Benefit Total cost test effort commercial software safety-critical software

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 5 Term Definitions  Test ►Testing is the process of exercising or evaluating a system or a system component by manual or automated means to verify that it satisfies specified requirements or to identify differences between expected and actual results. [IEEE "IEEE Standard Glossary of SW Engineering Terminology"]  Verification ►Verification means confirmation by examination and provision of objective evidence that specified requirements have been fulfilled. [ISO 8402:1994]  Validation ►Validation means confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use are fulfilled. [ECSS-P-001A, Rev. 1] Have we built the system correctly ? Does the system meet our expectation ?

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 6 SUT software under test SUT software under test V&V report test- class test case test results requirements + design docs dynamic test static analysis document verification Test & Verification

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 7 System test Integration test Acceptance tests Unit test bugfix re-test ? 1st test ? white box test ? ? ? existing code regression test black box test ? Classification of Tests

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 8 precompile ctp test object C++ classes C++ frames with test commands main make (compile cpp & link) Cantata++ wrapper classes instrumented code C++ coverage, white & black box test cases C++ reports Cantata++ libraries reports C/C++ libraries C++ Unit Test with IPL Cantata++

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 9 Test Metrics  Residual error rate  Value e.g. 20%  Ascending over time (Yet many errors occurring?)  Convergence (Do we discover less errors by time?)  Test coverage (dynamic)  „Our“ test tool Cantata++ implements 11 different coverage metrics  Ideal: 100%, often <100% due to „dead code“, „defensive programming“...  Complexity (static)  Statements per unit (static)  LOCs per unit (static)

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 10 Test Coverage – Examples Statement Decision Path coverage ? ? ? test cases

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 11 Cyclomatic Complexity Cyclomatic complexityrisk 1-10simple, low-risk program 11-20more complex, medium risk 21-50complex, high risk >50unstable program, very high risk c = e – n + 2 c = cyclomatic c omplexity e = number of e dges n = number of n odes

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 12 Experience Report

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 13 Error Types  Documentation Errors  Coverage Errors  Coding Errors  Other

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 14 Distribution of Error Types

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 15 Documentation Errors

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 16 Coding Errors

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 17 Incomplete Coverage

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 18 From Errors to Complexity

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 19 Defining Risk Areas

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 20 Categorization Example

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 21 Errors by Cyclomatic Complexity

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 22 Mean Error Count

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 23 Quality Management of Safety-Critical Systems  QA throughout the whole software life-cycle  Professional requirements engineering  Maintainability through quality of design  Verification und validation is essential  Code/document inspections  Often hazard analysis required  Usually safety standards (IEC 61508, DO178B,...)  Tests are in any case of highest importance

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept St. Ramberger / Th.Gruber 24 Conclusion  detailed design  code review  thorough verification