Experiences Using Lightweight Formal Methods for Requirements Modeling Easterbrook, Lutz, Covington, Kelly, Ampo, Hamilton

Introduction Cheaper to catch errors early Formal Methods, in general, used as last step Paper argues formal methods should be used during early stages 11/24/2008 Joshua Priestley

About this paper 3 case studies Expert team used formal methods on existing problems during early development phase Lightweight, selective application Only applied to components of full specification Used to increase (but not guarantee) the confidence in the requirements model provided by informal methods 11/24/2008 Joshua Priestley

The Need for Formal Methods in NASA Current model assurance methods based on inspection Fault protection subsystem good application for formal methods Sensitive to change during primary development Methods of formal challenging Theorem proving, state exploration, model checking 11/24/2008 Joshua Priestley

Case Study 1 – High Level FDIR Requirements for Space Station FDIR - Fault Detection, Isolation and Recovery/Reconfiguration System 18 pages of Functional Concept Diagram (FCD) Total energy invested: two person-months 11/24/2008 Joshua Priestley

Formalization Process Reorganized FCD into more organized FCD Originally 53 algorithmic steps 3 procedural categories, 6 condition classes Created PVS model from new FCD Prototype Verification System Software 11/24/2008 Joshua Priestley

Approach Informally check new FCD model Typecheck PVS Model Reasonableness Traceability Study of how high level requirements are transformed into low-level implementation Typecheck PVS Model Syntax errors, consistency errors, etc. PVS Proof assistant Proved logical claims regarding behavior of PVS model E.g. “if a failure occurs, then it will always be recovered at some domain level” 14 claims proven to verify behavior 11/24/2008 Joshua Priestley

PVS Results 15 issues documented Ambiguity, inconsistent terms, missing assumptions FDIR well thought out, but missing assumptions Reduced understanding by developers 11/24/2008 Joshua Priestley

PVS Results (cont’d.) 3 major issues discovered FDIR processes report status before, during, after each procedure (restructuring) Missing assumptions/requirements Sequencing of FDIR processing (proof) Order of events not specified but required Consistency check between procedural parameters (formalization) Not mentioned in the spec, though implied by designers 11/24/2008 Joshua Priestley

11/24/2008 Joshua Priestley

Dilbert on Software Testing Dilbert, 7-28-2007 Copyright © Scott Adams, via Dilbert.com 11/24/2008 Joshua Priestley

Case Study 2 – Detailed bus FDIR Requirements Concrete implementation of FDIR requirements outlined in previous study Original specification written in natural language Total energy invested: 1.5 person-months 11/24/2008 Joshua Priestley

Formalization Process Rewrote the spec in tabular form Difficult due to ambiguity of natural language Combined tables into single SCR state machine 11/24/2008 Joshua Priestley

Examples 11/24/2008 Joshua Priestley

11/24/2008 Joshua Priestley

Approach Type-checked SCR model using SCR toolset Analyze State Model Statically: Simple Model Analysis without context “One and only one FDIR Response specified for all combination of error inputs” Dynamically: Model Analysis over time with multiple failures Translated into PROMELA “If error persists after all actions tried, the bus FDIR will report error to higher FDIR Domain” 11/24/2008 Joshua Priestley

Findings Issues with inconsistent terminology Ambiguities due to long sentences in spec Table reformulation Bus switch inhibit flag Ordering of inference rules SCR test for disjointness Timing Constraints Mutually exclusive events PROMELA Model 11/24/2008 Joshua Priestley

Dilbert’s Automated Testing System 11/24/2008 Dilbert, 7-30-2007 Copyright © Scott Adams, via Dilbert.com Joshua Priestley

Case Study 3 – Fault Protection on Cassini 85 pages of requirements Total energy invested: 1 person-year 11/24/2008 Joshua Priestley

Formalization Process OMT model PVS specifications from OMT model 11/24/2008 Joshua Priestley

11/24/2008 Joshua Priestley

Approach PVS checked for internal consistency and traceability by theorem prover Ensure model matched documented requirements PVS checked for safety and liveliness conditions 11/24/2008 Joshua Priestley

Findings 11 Undocumented assumptions Formalization 10 inadequate requirements for off-nominal or boundary cases Multiple errors in units of same priority 9 Traceability/inconsistency problems Inconsistency between different levels of requirements 6 Imprecise/Synonymous Terminology 1 error Process starvation spotted during initial close reading, verified by theorem prover 11/24/2008 Joshua Priestley

Discussion Majority of formal methods case studies apply FM during post-development These case studies were performed during the development phase Feedback helped to improve specifications in high level design phase 11/24/2008 Joshua Priestley

Discussion (cont’d.) Tool Impoverishment All analysis was done by software tools which are still research prototypes Lightweight formal analysis can be achieved successful without complicated professional tools 11/24/2008 Joshua Priestley

Discussion (cont’d.) Economics Timescale of FM consistent with other V&V tasks Formalization the most time consuming step Also helps to discover errors Consistency checking relatively simple Type-checking with model 11/24/2008 Joshua Priestley

Who Should Apply Formal Methods Case studies were performed by a contracted expert Must understand specs Benefits Different set of assumptions than spec designers, no bias Drawbacks Some “errors” were simply misunderstandings of contractor 11/24/2008 Joshua Priestley

Are formal methods of volatile requirements worthwhile? During early design stages, specs are updated often formal models must match the specs Minimize updating overhead Keep models broad, model only aspects needed for verification purposes Ultimately worthwhile Formal model can be discarded if specs change, but lessons learned are retained 11/24/2008 Joshua Priestley

Immediate Models Spec -> Intermediate -> Formal Flowcharts, Tables, OMT diagrams Helped in understanding the requirements before attempting to create a formal model of spec Can also help structure formal model Highlighted correspondence between spec/model 11/24/2008 Joshua Priestley

Conclusions All analyses performed by experts Formal model was not primary model Results of formal testing fed back into the development phase Formal methods shown to complement existing V&V practices 11/24/2008 Joshua Priestley

Further Research Team is investigating ways to use formal methods on rapidly changing data, such that useful and timely information is extracted 11/24/2008 Joshua Priestley