SOUND METHODS and EFFECTIVE TOOLS for ENGINEERING MODELING and ANALYSIS _________________ by David Coppit, College of William and Mary, and Kevin J. Sullivan, University of Virginia Proceedings of the 25th International Conference on Software Engineering Portland, Oregon - May 3-10, 2003 Presentation by Bryan E. Bloss - University of Central Florida, Nov. 2003

1. INTRODUCTION Designing Software Tools for Formal Design ________________________________________________________________________________________ Engineering modeling & analysis methods are based on modeling languages for describing systems, with semantics for mapping expressions (models) to estimates of system properties (results). To be safe and effective, a modeling method requires a language with a validated semantics; feature-rich, easy-to-use, dependable tools; and low engineering costs. Hampered by shortcomings in software engineering & languages, today we lack adequate means to develop such methods.

Two Sub-Problems Addressed in this Paper: - Finding a cost-effective way to ensure semantic soundness of a complex method - Using Package-Oriented Programming (POP) to produce easy-to-use, functionally-rich tools from available software packages (such as MS Office) Results: A package-based tool “Galileo” is evaluated favorably by NASA engineers, and development of “Nova”, a similar tool based on a formal semantics, proves the cost-effectiveness of a combined approach

2. FORMAL SEMANTICS & DEPENDABILITY Why are they important? _____________________________________________________________________________________________________ Specification is more fundamental than implementation. Without a formal specification: - Validation is difficult - No basis for a definitive user reference document - Programmers are left to make uninformed semantic decisions; unable to thoroughly test correct functioning. Tools used in the design of safety critical systems should be treated as critical engineering components. Our inability to develop low-cost, easy-to-use tools can thus be seen as a positive safety mechanism, but far from ideal.

Safety Example: 1996 alert from U.S. Nuclear Regulatory Commission warned of significant errors in several tools which had been adopted for use in nuclear reactor design & analysis Another Example (not in paper): Crater analysis tool, used inappropriately during flight STS-107 to analyze foam damage

3. APPROACH & BASIC RESULTS Developing the Galileo Tool for DFT Analysis ___________________________________________________________________________________________ Observation: Most applications devote less than 10% of their code to the core function of the system! 90% is devoted to superstructure-- support functions such as text & graphical editing, data validation, etc. Package-Oriented Programming (POP) is intended to save time in creating superstructure; frees more resources for the critical design activity: applying formal methods to define & validate the syntax and semantics of the modeling language

The Application: Dynamic Fault Trees (DFT) Graphcal representation of every conceivable sequence of events that could cause a system to fail. Each leaf is a basic event; internal gates define relationships leading to system failures at upper levels. Static trees model how event combinations lead to failures; Dynamic trees are order-sensitive. (Illustrations from CAIB Report)

3./4. CASE STUDY: RELIABILITY ENGINEERING The Problems With Current DFT Languages ___________________________________________________________________________________________ During development of Galileo tool, a non-trivial error was found in the underlying DFT language, DIFtree, where probability of a masked (hidden) failure wasn’t correctly computed Also, DIFtree’s informal specification had left ambiguities on how to handle special cases; prior software implementations answered these questions inconsistently in different parts of the program. And formal validation was time-consuming, due to lack of automation in available syntax & theorem-prover tools, and slow run-time perfomance Worse, the theorem-prover tool required too much user expertise; guidance often needed from the tool’s author

5. The NOVA DFT Tool Like Galileo, uses POP components from MS Office for fault tree editing: - Word for text editing - Visio graphical editor (enhanced for DFT modeling constructs) - Excel for computational results Will allow even more emphasis on formalization & validation than in Galileo

6. END-USER EVALUATION OF GALILEO New version of Galileo tool funded by NASA Langley Research Center, to support new modeling & analysis constructs and be usable in practice Featured in three workshops: 1st- Managers & engineers from several NASA divisions 2nd- Space Station engineers only 3rd- Space Shuttle engineers only A short survey (34 questions) and an in-depth survey (77 questions) were offered to engineers, to evaluate user perceptions of usability & features Feedback indicated that usability was same or better than other tools! Also confirmed that dependability is crucial; a formal specificaltion of the modeling language was second only to a comprehensive test suite as a means for increasing trust.

____Evaluation of this Article ____ Strengths: A well-informed overview of the authors’ experience with developing a new software toolset for practical engineering, while emphasizing formal validation methods. Many implications for the design of other reliability-critical software applications. Weaknesses: Little information on costs saved by POP method; little detail on formal proving methods. Also, the more advanced NOVA tool had not been user-tested at time of publication, so final results aren’t known.

____Questions? ____