1. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 1 Experience Report: Error Distribution in Safety-Critical Software & Software Risk Analysis Based on Unit Tests Stephan Ramberger (stephan.ramberger@arcs.ac.at)stephan.ramberger@arcs.ac.at Thomas Gruber (thomas.gruber@arcs.ac.at)thomas.gruber@arcs.ac.at ARC Seibersdorf research GmbH

2. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 2 3 Classes of Impairments (J.-C. Laprie)  Faults = detected or assumed reason of an error  Errors = portion of a system state which necessarily leads to a subsequent failure: The error is an indicator for a failure that occors or has occurred.  Failures = a deviation between the delivered system function and the required one

3. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 3 Error Cost – Example Ariane 5 Flight 501  June 4, 1996: Maiden flight of Ariane 5  37 secods after ignition horizontal speed = 32.768,0 internal units  Exception in „idle“ subsystem by 16 bit – integer conversion (overflow)  Subsystem software had been tested only for the smaller Ariane 4 model.  Redundant computer had the same problem 72 ms earlier – switched off  Main controller interprets diagnostic data as trajectory data  Unreasonable control commands to the engines  Purposeful blowing-up after 40 Seconds at 3.700 m altitude  Damage:  € 130.000.000 launch cost  € 435.000.000 cluster satellites  € 310.000.000 subsequent improvements http://uprhmate01.upr.clu.edu/~pnm/notas4061/ariane5.htm

4. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 4 Testing: Cost and Benefit Total cost test effort commercial software safety-critical software

5. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 5 Term Definitions  Test ►Testing is the process of exercising or evaluating a system or a system component by manual or automated means to verify that it satisfies specified requirements or to identify differences between expected and actual results. [IEEE 729-1983 "IEEE Standard Glossary of SW Engineering Terminology"]  Verification ►Verification means confirmation by examination and provision of objective evidence that specified requirements have been fulfilled. [ISO 8402:1994]  Validation ►Validation means confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use are fulfilled. [ECSS-P-001A, Rev. 1] Have we built the system correctly ? Does the system meet our expectation ?

6. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 6 SUT software under test SUT software under test V&V report test- class test case test results requirements + design docs dynamic test static analysis document verification Test & Verification

7. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 7 System test Integration test Acceptance tests Unit test bugfix re-test ? 1st test ? white box test ? ? ? existing code regression test black box test ? Classification of Tests

8. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 8 precompile ctp test object C++ classes C++ frames with test commands main make (compile cpp & link) Cantata++ wrapper classes instrumented code C++ coverage, white & black box test cases C++ reports Cantata++ libraries reports C/C++ libraries C++ Unit Test with IPL Cantata++

9. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 9 Test Metrics  Residual error rate  Value e.g. 20%  Ascending over time (Yet many errors occurring?)  Convergence (Do we discover less errors by time?)  Test coverage (dynamic)  „Our“ test tool Cantata++ implements 11 different coverage metrics  Ideal: 100%, often <100% due to „dead code“, „defensive programming“...  Complexity (static)  Statements per unit (static)  LOCs per unit (static)

10. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 10 Test Coverage – Examples Statement Decision Path coverage ? ? ? 1 2 4 test cases

11. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 11 Cyclomatic Complexity Cyclomatic complexityrisk 1-10simple, low-risk program 11-20more complex, medium risk 21-50complex, high risk >50unstable program, very high risk c = e – n + 2 c = cyclomatic c omplexity e = number of e dges n = number of n odes

12. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 12 Experience Report

13. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 13 Error Types  Documentation Errors  Coverage Errors  Coding Errors  Other

14. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 14 Distribution of Error Types

15. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 15 Documentation Errors

16. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 16 Coding Errors

17. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 17 Incomplete Coverage

18. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 18 From Errors to Complexity

19. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 19 Defining Risk Areas

20. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 20 Categorization Example

21. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 21 Errors by Cyclomatic Complexity

22. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 22 Mean Error Count

23. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 23 Quality Management of Safety-Critical Systems  QA throughout the whole software life-cycle  Professional requirements engineering  Maintainability through quality of design  Verification und validation is essential  Code/document inspections  Often hazard analysis required  Usually safety standards (IEC 61508, DO178B,...)  Tests are in any case of highest importance

24. INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th.Gruber 24 Conclusion  detailed design  code review  thorough verification