1. Experiences Using Lightweight Formal Methods for Requirements Modeling
Easterbrook, Lutz, Covington, Kelly, Ampo, Hamilton

2. Introduction Cheaper to catch errors early
Formal Methods, in general, used as last step
Paper argues formal methods should be used during early stages
11/24/2008
Joshua Priestley

3. About this paper 3 case studies
Expert team used formal methods on existing problems during early development phase
Lightweight, selective application
Only applied to components of full specification
Used to increase (but not guarantee) the confidence in the requirements model provided by informal methods
11/24/2008
Joshua Priestley

4. The Need for Formal Methods in NASA
Current model assurance methods based on inspection
Fault protection subsystem good application for formal methods
Sensitive to change during primary development
Methods of formal challenging
Theorem proving, state exploration, model checking
11/24/2008
Joshua Priestley

5. Case Study 1 – High Level FDIR Requirements for Space Station
FDIR - Fault Detection, Isolation and Recovery/Reconfiguration System
18 pages of Functional Concept Diagram (FCD)
Total energy invested: two person-months
11/24/2008
Joshua Priestley

6. Formalization Process
Reorganized FCD into more organized FCD
Originally 53 algorithmic steps
3 procedural categories, 6 condition classes
Created PVS model from new FCD
Prototype Verification System Software
11/24/2008
Joshua Priestley

7. Approach Informally check new FCD model Typecheck PVS Model
Reasonableness
Traceability
Study of how high level requirements are transformed into low-level implementation
Typecheck PVS Model
Syntax errors, consistency errors, etc.
PVS Proof assistant
Proved logical claims regarding behavior of PVS model
E.g. “if a failure occurs, then it will always be recovered at some domain level”
14 claims proven to verify behavior
11/24/2008
Joshua Priestley

8. PVS Results 15 issues documented
Ambiguity, inconsistent terms, missing assumptions
FDIR well thought out, but missing assumptions
Reduced understanding by developers
11/24/2008
Joshua Priestley

9. PVS Results (cont’d.) 3 major issues discovered
FDIR processes report status before, during, after each procedure (restructuring)
Missing assumptions/requirements
Sequencing of FDIR processing (proof)
Order of events not specified but required
Consistency check between procedural parameters (formalization)
Not mentioned in the spec, though implied by designers
11/24/2008
Joshua Priestley

10. 11/24/2008
Joshua Priestley

11. Dilbert on Software Testing
Dilbert, Copyright © Scott Adams, via Dilbert.com
11/24/2008
Joshua Priestley

12. Case Study 2 – Detailed bus FDIR Requirements
Concrete implementation of FDIR requirements outlined in previous study
Original specification written in natural language
Total energy invested: 1.5 person-months
11/24/2008
Joshua Priestley

13. Formalization Process
Rewrote the spec in tabular form
Difficult due to ambiguity of natural language
Combined tables into single SCR state machine
11/24/2008
Joshua Priestley

14. Examples
11/24/2008
Joshua Priestley

15. 11/24/2008
Joshua Priestley

16. Approach Type-checked SCR model using SCR toolset Analyze State Model
Statically:
Simple Model Analysis without context
“One and only one FDIR Response specified for all combination of error inputs”
Dynamically:
Model Analysis over time with multiple failures
Translated into PROMELA
“If error persists after all actions tried, the bus FDIR will report error to higher FDIR Domain”
11/24/2008
Joshua Priestley

17. Findings Issues with inconsistent terminology
Ambiguities due to long sentences in spec
Table reformulation
Bus switch inhibit flag
Ordering of inference rules
SCR test for disjointness
Timing Constraints
Mutually exclusive events
PROMELA Model
11/24/2008
Joshua Priestley

18. Dilbert’s Automated Testing System
11/24/2008
Dilbert, Copyright © Scott Adams, via Dilbert.com
Joshua Priestley

19. Case Study 3 – Fault Protection on Cassini
85 pages of requirements
Total energy invested: 1 person-year
11/24/2008
Joshua Priestley

20. Formalization Process
OMT model
PVS specifications from OMT model
11/24/2008
Joshua Priestley

21. 11/24/2008
Joshua Priestley

22. Approach
PVS checked for internal consistency and traceability by theorem prover
Ensure model matched documented requirements
PVS checked for safety and liveliness conditions
11/24/2008
Joshua Priestley

23. Findings 11 Undocumented assumptions
Formalization
10 inadequate requirements for off-nominal or boundary cases
Multiple errors in units of same priority
9 Traceability/inconsistency problems
Inconsistency between different levels of requirements
6 Imprecise/Synonymous Terminology
1 error
Process starvation
spotted during initial close reading, verified by theorem prover
11/24/2008
Joshua Priestley

24. Discussion
Majority of formal methods case studies apply FM during post-development
These case studies were performed during the development phase
Feedback helped to improve specifications in high level design phase
11/24/2008
Joshua Priestley

25. Discussion (cont’d.) Tool Impoverishment
All analysis was done by software tools which are still research prototypes
Lightweight formal analysis can be achieved successful without complicated professional tools
11/24/2008
Joshua Priestley

26. Discussion (cont’d.) Economics
Timescale of FM consistent with other V&V tasks
Formalization the most time consuming step
Also helps to discover errors
Consistency checking relatively simple
Type-checking with model
11/24/2008
Joshua Priestley

27. Who Should Apply Formal Methods
Case studies were performed by a contracted expert
Must understand specs
Benefits
Different set of assumptions than spec designers, no bias
Drawbacks
Some “errors” were simply misunderstandings of contractor
11/24/2008
Joshua Priestley

28. Are formal methods of volatile requirements worthwhile?
During early design stages, specs are updated often
formal models must match the specs
Minimize updating overhead
Keep models broad, model only aspects needed for verification purposes
Ultimately worthwhile
Formal model can be discarded if specs change, but lessons learned are retained
11/24/2008
Joshua Priestley

29. Immediate Models Spec -> Intermediate -> Formal
Flowcharts, Tables, OMT diagrams
Helped in understanding the requirements before attempting to create a formal model of spec
Can also help structure formal model
Highlighted correspondence between spec/model
11/24/2008
Joshua Priestley

30. Conclusions All analyses performed by experts
Formal model was not primary model
Results of formal testing fed back into the development phase
Formal methods shown to complement existing V&V practices
11/24/2008
Joshua Priestley

31. Further Research
Team is investigating ways to use formal methods on rapidly changing data, such that useful and timely information is extracted
11/24/2008
Joshua Priestley