1. SAS_08_Model_Val_Tech_Heimdahl MAC-T IVV-08-152 Model-Validation in Model-Based Development Kurt Woodham L-3 Communications Ajitha Rajan, Mats Heimdahl University of Minnesota OSMA SAS ’08September 8-12

2. 2 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Problem: Model Validation Model-Based Development (MBD) is here to stay  Use of MBD is accelerating Estimate 50% of NASA development projects using some form of MBD Many advantages: model-checking, code generation, desktop testing, closed-loop simulation  Enhances early detection of requirement, design, or implementation defects “Executable Specifications” enable evaluation of behavior that might otherwise be relegated to Inspections and Testing How do we know the models are “right”?  Manually develop black-box tests When have we validated enough?  Measure test coverage on an implementation/model

3. 3 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Problem : Current Practice Measure black-box test coverage over the model  Indirect measure Defects of omission in model not exposed.  Executable artifact is necessary Adequacy can only be determined late in the development process Incomplete Model Weak Black-Box Test set

4. 4 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Goals of Project Define metrics for objective, implementation-independent measure of adequacy of a black-box test suite Develop tools to measure validation adequacy based on the defined metrics Provide capability for autogeneration of black-box test suites

5. 5 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Testing – What does it mean? Specification Implementation Does it implement? Assertions Model Does it implement? Assertion Based Testing (ABT) to Validate Model Model Source Code Does it implement? Model-Based Testing (MBT) to Verify Code Our contribution is in providing novel ABT capabilities In General

6. 6 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl What are Assertions? Assertions Properties/ Formal Assertions Defined over System in 1 in k out 1 out m Can also be over components, interfaces,...

7. 7 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Contributions - ABT Assertions Black-Box Tests Measure Adequacy Auto-generate 2 Model Validate Assess Model and Assertion Completeness 3 1 We provide the following contributions in the Assertion-Based testing domain (indicated by in the above figure): 1.Objective, implementation-independent measure of adequacy of a black- box test suite 2.Auto-generation of black-box validation tests directly from assertions 3.Objective assessment of completeness of model as well as assertions

8. 8 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl The Idea …then define structural coverage metrics to directly and objectively describe coverage of assertions Write assertions in a formal notation… G (FD_On -> Cues_On); G((¬ Onside_FD_On Λ ¬ Is_AP_Engaged) → X(Is_AP_Engaged → Onside_FD_On)) Temporal Logic Synchronous Observers

9. 9 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl A LTL Temporal Operators OperatorNotationMeaning Globally AG(A)Formula A is true in all states Future AF(A)Formula A is true in some future state A until BA U BFormula A is true in every state until B becomes true. B must eventually become true for the property to be true. Next AX(A)Formula A is true in the next state S0S0 S1S1 S2S2 S3S3 SiSi AA A B A

10. 10 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Formalizing Assertions “If the onside FD cues are off, the onside FD cues shall be displayed when the AP is engaged” G((¬ Onside_FD_On  ¬ Is_AP_Engaged) → X(Is_AP_Engaged → Onside_FD_On)) Possible Coverage Metrics  Assertion coverage: single test case that demonstrates that assertion is satisfied Prone to “dumb” tests, e.g., execution in which AP is never engaged.  More rigorous metric is necessary

11. 11 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Task - 1 Define a collection of assertion coverage criteria Formalize the assertion coverage obligations

12. 12 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Antecedent Coverage Many of the assertions in the FGS are of the form : Globally if ‘A’ occurs then ‘B’ will occur G (A → B) Two ways of satisfying (A → B) –A is false –A is true and B is true Antecedent Coverage – test cases will exercise the antecedent. S0S0 S1S1 SnSn Not A A, B What if: A  C  D → B

13. 13 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Modified Condition/Decision Coverage (MC/DC) To satisfy MC/DC  Every point of entry and exit in the model should be invoked at least once,  Every basic condition in a decision in the model should take on all possible outcomes at least once, and  Each basic condition should be shown to independently affect the decision’s outcome ABA or B FFF TFT FTT Independent effect of A Independent effect of B Basic Conditions

14. 14 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Unique First Cause (UFC) Coverage “ System shall eventually generate an Ack (A) or a Time Out (B) ” Req. LTL property - F(A  B). Formal UFC obligation for A : ¬ (A  B) U (A  ¬ B) for B : ¬ (A  B) U (B  ¬ A) S0S0 S1S1 S2S2 S3S3 SiSi A, ¬ B ¬A, ¬B ¬A, B ¬A, ¬B Path satisfies UFC obligation for A but not B. S0S0 S1S1 SiSi To show independence of B, ¬A, B ¬A, ¬B

15. 15 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl UFC Coverage G(A) + = {A U (a  G(A)) | a є A + } G(A) - = {A U a | a є A - } F(A) + = {¬A U a | a є A + } F(A) - = {¬A U (a  G(¬A))| a є A - } (A U B) + = {(A  ¬B) U ((a  ¬B)  (A U B)) | a є A + }  {(A  ¬B) U b | b є B + } (A U B) - = {(A  ¬B) U (a  ¬B) | a є A - }  {(A  ¬B) U (b  ¬(A U B)) | b є B - } X(A) + = {X(a) | a є A + } X(A) - = { X(a) | a є A - } Michael Whalen, Ajitha Rajan, Mats Heimdahl and Steven Miller. Coverage Metrics for Requirements-Based Testing. In Proceedings of ISSTA 2006.

16. 16 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Task 2 – Validation Adequacy Measurement Tool We currently support the following coverage metrics: Assertion Coverage Assertion Antecedent Coverage Assertion UFC Coverage

17. 17 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Task - 3 Automatically generate requirements-based tests from …  Formal assertions  Abstract model called Assertion Model created using assertions and environmental constraints (specified as invariants) … to provide the defined assertion coverage.

18. 18 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Automatically Generating Requirements-Based Tests Common with the Adequacy Measurement Tool Assertions and environmental constraints specified as invariants

19. 19 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl What Are Model Checkers? Breakthrough technology of the 1990’s Widely used in hardware verification  (Intel, Motorola, IBM, …) Several different types of model checkers  Explicit, Symbolic, Bounded, Infinite Bounded, … Exhaustive search of the global state space  Consider all combinations of inputs and states  Equivalent to exhaustive testing of the model  Produces a counter example if a property is not true Easy to use  “Push button” formal methods  Very little human effort unless you’re at the tool’s limits Limitations  State space explosion

20. 20 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Preliminary Evaluation Interested in determining: Feasibility of generating assertion-based tests from a set of assertions  Generated assertion-based tests to provide UFC coverage over the assertions Effectiveness of these test sets in validating the system model  Measured MC/DC achieved by the test sets over the system model Used three realistic sized examples: Flight Guidance System (FGS), and two models related to the Display Window Manager system (DWM1 and DWM2)

21. http://www.umsec.umn.edu 21 Results Ajitha Rajan, Michael Whalen, and Mats Heimdahl. Model Validation using Automatically Generated Requirements- Based Tests. In Proceedings of 10th IEEE High Assurance Systems Engineering Symposium, Nov 2007.

22. 22 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Results and Analysis UFC test suites achieved high MC/DC coverage over DWM models – well defined set of assertions Test-suite generated for UFC achieved very low MC/DC over the FGS model “When the FGS is in independent mode, it shall be active”. G(m_Independent_Mode_Condition.result → X(Is_This_Side_Active = 1)) RSML –e Macro Structure of Independent_Mode_Condition is not captured in the property Independent_Mode_Condition = ((Is_LAPPR_Active & Is_VAPPR_Active & IS_Offside_LAPPR_Active & Is_Offside_VAPPR_Active) | ( Is_VGA_Active & Is_Offside_VGA_Active))

23. 23 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Benefits of ABT Saves time and effort in generating validation test suites from assertions Effective method for generating model validation tests when the assertions are well defined Helps in identifying missing assertions and over constrained models

24. 24 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Bonus Task – Adequacy of Conformance Testing Model Conformance Tests Code Run Measure Adequacy Assertions Measure Adequacy 1.Direct assessment of how well tests exercise the assertions 2.Will expose defects of omission 3.Assertion coverage could necessitate longer test cases than for model coverage Useful ??

25. 25 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Assertion Coverage as an Adequacy Measure for Conformance Testing Hypothesis 1(H 1 ): Conformance tests providing assertion UFC coverage are more effective than conformance tests providing MC/DC over the model Hypothesis 2(H 2 ): Conformance tests providing assertion UFC coverage in addition to MC/DC over the model are more effective than conformance tests providing only MC/DC over the model

26. 26 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Experiment Used four industrial systems :  Two models from the display window manager  Two models representing the mode logic of a flight guidance system Assessed effectiveness of test suites in terms of their fault finding ability Ajitha Rajan, Michael Whalen, Matt Staats, and Mats Heimdahl. Requirements Coverage as an Adequacy Measure for Conformance Testing. To Appear in Proceedings of 10th International Conference on Formal Engineering Methods, Oct 2008.

27. 27 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Results – Hypothesis 1 Hypothesis 1 rejected at 5% statistical significance on all but the Latctl system

28. 28 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Analysis – Hypothesis 1 Model coverage better than assertion coverage for measuring adequacy of conformance test suites Assertion UFC coverage is heavily dependent on the nature and completeness of the assertions Rigor and robustness of assertion coverage metric used is important  UFC metric gets cheated when assertions are structured to hide the complexity of conditions

29. 29 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Results – Hypothesis 2 Hypothesis 2 accepted at 5% statistical significance on all but the DWM2 system

30. 30 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Analysis – Hypothesis 2 UFC Achieved by MC/DC suites Achievable UFCRel. Diff DWM128.3%96.9%70.8% DWM259.7%64%6.7% Latctl94.7%99.5%4.8% Vertmax97.4%99%1.6% Does UFC really help in revealing additional faults?

31. 31 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Summary – Bonus Task UFC > MC/DC FALSE  3 of the 4 case examples at 5% statistical significance UFC + MC/DC > MC/DC TRUE  3 of the 4 case examples at 5% statistical significance Combine rigorous metrics for assertion coverage and model coverage to measure adequacy of conformance test suites UFC metric is sensitive to structure of assertions  Need assertion coverage metrics that are robust to structure of assertions

32. 32 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Technology Readiness Level “Requirements-Based Test Generation Tool” TRL = 6  System/subsystem model or prototype demonstration in a relevant environment “Validation Adequacy Measurement Tool” TRL = 6  System/subsystem model or prototype demonstration in a relevant environment

33. 33 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Relevance to NASA MBD is here - estimate one-half of all NASA missions in development or on the books will use model-based subsystem development  Extensive use in avionics industry How do we know the models are right?  Model validation problem We provide the capability to  Objectively measure the “quality” of assertion-based black-box validation tests  Objectively assess the completeness of a model Does the model address all assertions?  Objectively assess the adequacy of a set of assertions Are there enough assertions to adequately describe the model?  Automatically generate truly assertion-based tests

34. 34 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Achievements to Date Formal assertion notation identified  Most work with LTL  Extended to work with Live Sequence Charts (LSC) Objective validation metrics defined  Requirements, Antecedent, Unique First Cause, and Unique Cause Test case generation tool developed  Developed tool generating tests from LTL Capable of generating tests to all metrics defined  Prototype tool working on LSC developed Developed test-adequacy measurement tool for the defined validation metrics Evaluation of metrics and tool 12 papers and one PhD dissertation (Ajitha Rajan)

35. 35 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Next Steps Investigate alternative requirements notations to LTL Complete empirical evaluation of the effectiveness in model validation  Flight Guidance Sensor (FGS) evaluation complete  Display Manager (DM) evaluation in work  Coordinate evaluation on NASA IV&V project Coordinate technology transfer

36. 36 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl Discussion