1. IV&V Facility Assertion–Based Requirements Validation Patrick Theeke (PMP, CSQE) John Ryan cmg@ivv.nasa.gov September 8, 2008

2. IV&V Facility 1/18/2016 2 Assertion–Based Requirements Validation Objectives Describe the process of using executable models for requirements validation Discuss experiences and characterize outcomes from a pilot project employing the technique Identify lessons learned to date and possible next steps

3. IV&V Facility 1/18/2016 3 Assertion–Based Requirements Validation Agenda Briefly describe Model-Based Requirements Validation Define the process for Assertion-Based Requirements Validation Present experiences of pilot project Present observations Discuss additional details via panel

4. IV&V Facility 1/18/2016 4 Requirements Validation Goals of Requirements Validation Assure that the documented requirements fully specify the capabilities and characteristics necessary for the system being defined to meet its goals In particular, NASA IVV is interested in critical capabilities Requirements defects include –Missing –Superfluous –Ambiguous –Incomplete –Inconsistent –Unverifiable –Incorrect

5. IV&V Facility 1/18/2016 5 Requirements Validation Validation Approach Build a standard of reference for comparison against the documented requirements of the system of interest This standard is the System Reference Model (SRM), which represents IVV’s understanding of the system of interest Verifiable Requirements Software Requirements SRM Behaviors SRM Correct? Requirements Complete? Valid? Validated Requirements SRM Correct? Requirements in Scope? Valid? Correct, Complete, Consistent, Unambiguous, and Verifiable Requirements

6. IV&V Facility 1/18/2016 6 Requirements Validation SRM components Behaviors –Use cases and UML diagrams (e.g. activity diagrams) Structure –Placeholder for named physical and logical components of the system –Identify component roles and events important to understanding the system Independent Modeling Assertions (IMA) –Formalized IMA extracted from the other SRM components –Collection of unit tests to validate the IMAs System Scenarios –Scenarios that describe various sequences of events Nominal sequences (what the system is supposed to do) Unwanted behaviors (what the system is not supposed to do) Detection and response to adverse conditions (how the systems responds to adverse conditions)

7. IV&V Facility 1/18/2016 7 Assertion–Based Requirements Validation Pilot Assertion Statechart Definition Each Statechart Assertion is a formal specification of a single requirement. A statechart assertion is fundamentally a monitoring device that observes system behavior and determines whether that behavior is valid Observed behavior is valid when it matches the behavior specification coded into the assertion, and invalid when it violates the specification An assertion is run against observable behavior, typically supplied by some executable artifact running under a test scenario

8. IV&V Facility 1/18/2016 8 Requirements Validation Correlation-Based Requirements Validation Capture independent understanding of system behavior or characteristic in a SRM Correlate project requirements with SRM behaviors Analyze results to identify –Ambiguous, incorrect, incomplete, inconsistent, or non-verifiable requirements that correspond to SRM behaviors –Requirements without matching SRM behavior –SRM behavior without matching requirements SRM Correlations with SRM Project Requirements Analyze Results Findings

9. IV&V Facility 1/18/2016 9 Assertion–Based Requirements Validation Motivation Formalization of natural language requirements reveal errors in the specification Leveraging formal assertions for requirements validation increases the utility of formalized requirements –Formalization of requirements is a necessary step in leveraging the SRM for verification Utilizing assertions more closely mimics the highly parallel threads of execution of real time, reactive systems, including timing and synchronization, allowing more in-depth examination of complex behavioral interactions Discovering requirements and representing them as assertions is more readily done early in the modeling process while the SRM is under development – so we might as well use them!

10. IV&V Facility 1/18/2016 10 Assertion–Based Requirements Validation Dependencies Assertion Based Validation Depends on: –SRM was built to the appropriate level of depth and details –The requirements were traced to the SRM –The set of system scenarios provided an appropriate coverage of the system behaviors –The naming convention for the assertion events match the events of the formalized unit tests, system scenarios, and assertion state charts

11. IV&V Facility 1/18/2016 11 Assertion–Based Requirements Validation Process Assertion-Based Requirements Validation Formally capture developer’s documented requirements as Project Based Assertions (PBA) Exercise PBAs and IMAs using validated System Scenarios Analyze results SRM Correlations with SRM Project Requirements Analyze Results Findings Develop & Validate PBAs Execute Scenarios in Context of PBAs Develop & Validate IMAs Develop System Scenarios

12. IV&V Facility 1/18/2016 12 Assertion–Based Requirements Validation Process Develop & Validate Project–Based Assertions Translate the documented requirements from the development project into a formal representation that is conducive to comparison with the SRM Requires detailed understanding of the requirements Exposes errors in the requirements specifications SRM Correlations with SRM Project Requirements Analyze Results Findings Develop & Validate PBAs Execute Scenarios in Context of PBAs Identify Relevant Documented requirements Interpret Requirements Formalize Requirements as PBAs Validate PBAs

13. IV&V Facility 1/18/2016 13 Assertion–Based Requirements Validation Process Execute scenarios in Context of PBAs We are interested in how the PBAs react to the system scenarios –Which system scenarios ended as expected? –Which system scenarios ended differently than expected? –Which PBAs were not exercised? SRM Correlations with SRM Project Requirements Analyze Results Findings Develop & Validate PBAs Execute Scenarios in Context of PBAs Include PBAs in executable environment Execute system scenarios Collect results for further analysis

14. IV&V Facility 1/18/2016 14 Assertion–Based Requirements Validation Process Analyze Results Examine each assertion failure for: –Errors in constructing the SRM (system scenarios, IMAs, etc.) –Errors in constructing the PBA(s) –Errors in a requirement Fix the errors in the SRM Rerun test until no SRM or PBA construction errors are discovered. Therefore all unexpected results indicate findings SRM Correlations with SRM Project Requirements Analyze Results Findings Develop & Validate PBAs Execute Scenarios in Context of PBAs Ensure system scenario set is correct and complete Ensure PBA set is complete Ensure that PBA is in scope Ensure PBA is correct

15. IV&V Facility 1/18/2016 15 Assertion–Based Requirements Validation Pilot Develop and Validate IMAs Examine behavior in UML activity diagrams –Begin with preconditions, triggers, constraints –Assert important sequences that must occur in order (under all conditions or adverse conditions) –Constraints on looping (will continue to try until …) –Important executions if a critical behavior is unavailable –Any possible/impossible executions that will occur during off- nominal events –Look at the whole picture and observe modeling issues as well SRM Correlations with SRM Project Requirements Analyze Results Findings Develop & Validate PBAs Execute Scenarios in Context of PBAs Develop System Scenarios Develop & Validate IMAs Identify Candidate IMAs Analyze Candidate IMAs Represent Candidate IMAs in Natural Language Formalize IMAs as assertion state charts Validate IMAs

16. IV&V Facility 1/18/2016 16 Assertion–Based Requirements Validation Pilot Identify candidate IMAs -Actively Damp Nutation [After burn maneuver, need to stabilize damping before selecting Idle Mode to downlink data] Select Damping Mode Determine Attitude Configure Damping Filter Estimate Nutation Angle IF Angle > target: Then compute damping torque prior to controlling fired thruster burn IF Angle < target: continue process Estimate target Angle Check if < target Track time passing Once stablized for minimal time, GNC subsytem can Select Idle Mode

17. IV&V Facility 1/18/2016 17 Assertion–Based Requirements Validation Pilot Analyze candidate IMAs Analysis Note: The nutation angle must be less than or equal to target nutation angle for the minimum amount of stable time to maintain an acceptable and stable nutation angle. [P = T2 (Tmin)] Q1. The nutation angle must (/can?) be less than or equal to target nutation angle for the minimum amount of stable time to select Idle Mode. Domain Answer– Actually both Ground and Fault Protection Mode can select Idle Mode at any time even if the above is not true ([P <= R and T1 >= T2 (Tmin)]). Assumption: If there was a satellite timeout occurring, and if the fault protection mode has not been entered (due to # attempts or timeout), then the above is true. Q1b. The Reference Orientation (GN&C sub system) must maintain a nutation angle that is less than or equal to the target nutation angle for a minimal amount of stable time to select Idle Mode. (See Figure 16 and 20) Q2. What occurs if the goal is met (nutation angle stabilized and maintainable) and something causes the angle to exceed the nutation angle again? Domain Answer– The issue is monitored and the active damp nutation behavior is called again.

18. IV&V Facility 1/18/2016 18 Assertion–Based Requirements Validation Pilot Analyze candidate IMAs Analyze each candidate assertion and create good and bad scenarios to represent the requirements Use the diagrams in the SRM to derive scenarios or unit tests that will cause the requirement to succeed and fail Use the scenarios to challenge the assertion against –redundancy of events –improper sequencing –Unknowns –misuse of system boundaries –Dependencies –and constraints (i.e time) Determine whether or not the sequencing, repetition, looping, constraints, and concurrence always (“must”) occur this way or conditionally throughout the entire SRM

19. IV&V Facility 1/18/2016 19 Assertion–Based Requirements Validation Pilot Analyze candidate IMAs Good and Bad Scenarios for Q1B: Exceed Target Nutation Angle Reach Target Nutation Angle Time passed = 2 Time passed = 3 Select Idle Mode Exceed Target Nutation Angle Reach Target Nutation Angle Time passed = 4 Select Idle Mode Good Scenario Bad Scenario TimerTminis the minimal amount of stable time required to reach a maintainable angle state and allow ability to select Idle Mode. Tmin= 5. Exceed Target Nutation Angle Reach Target Nutation Angle Time passed = 4 The Reference Orientation must maintain anutationangle that is less than or equal to the targetnutationangle for a minimal amount of stable time to select Idle Mode.

20. IV&V Facility 1/18/2016 20 Assertion–Based Requirements Validation Pilot Represent the candidate IMAs as NLR Natural Language Requirement for Q1b –The Reference Orientation (GN&C sub system) shall achieve a stable nutation angle that is less than or equal to an acceptable target nutation angle for a minimal amount of time prior to selecting Idle Mode.

21. IV&V Facility 1/18/2016 21 Assertion–Based Requirements Validation Pilot Formalize each Natural Language Requirement

22. IV&V Facility 1/18/2016 22 Assertion–Based Requirements Validation Pilot Validate each formalized IMA Validate the IMA is unambiguous, correct, and consistent with the SRM behavior and scenarios. –Validate each model-based formalized requirement has been modeled correctly to represent the intended requirement Create formalized unit tests for the scenarios Validate the IMA against the formalized unit tests publicvoid testDampNutation_IMA001_sc001(){ // this is testing a good scenario for RO selecting Idle Mode printState(IMA001); IMA001.reachTargetNutationAngle(); printState(IMA001); IMA001.incrSimTime(1); printState(IMA001); IMA001.exceedTargetNutationAngle(); printState(IMA001); IMA001.reachTargetNutationAngle(); printState(IMA001); IMA001.incrSimTime(5); printState(IMA001); IMA001.incrSimTime(1); printState(IMA001); IMA001.ROselectIdleMode(); printState(IMA001); assertTrue(IMA001.isSuccess()); System.out.println("CompletedtestDampNutation_IMA001_sc001"); System.out.println(); System.out.println("-------------------------"); System.out.println(); } publicvoid testDampNutation_IMA001_sc002(){ // this is testing a b scenario for RO selecting Idle Mode printState(IMA001); IMA001.reachTargetNutationAngle(); printState(IMA001); IMA001.incrSimTime(1); printState(IMA001); IMA001.exceedTargetNutationAngle(); printState(IMA001); IMA001.reachTargetNutationAngle(); printState(IMA001); IMA001.incrSimTime(4); printState(IMA001); IMA001.ROselectIdleMode(); printState(IMA001); assertFalse(IMA001.isSuccess()); System.out.println("CompletedtestDampNutation_IMA001_sc002"); }

23. IV&V Facility 1/18/2016 23 Assertion–Based Requirements Validation Pilot Develop & Validate Project–Based Assertions Translate the documented requirements from the development project into a formal representation that is conducive to comparison with the SRM Requires detailed understanding of the requirements Exposes errors in the requirements specifications SRM Correlations with SRM Project Requirements Analyze Results Findings Develop & Validate PBAs Execute Scenarios in Context of PBAs Develop & Validate IMAs Develop System Scenarios Identify Relevant Documented requirements Interpret Requirements Formalize Requirements as PBAs Validate PBAs

24. IV&V Facility 1/18/2016 24 Assertion–Based Requirements Validation Pilot Develop & Validate Project–Based Assertions

25. IV&V Facility 1/18/2016 25 public void testTorqueVector_IMA001(){ printState(pba001); pba001.beginJOIPhase(); printState(pba001); pba001.SRUUpdate(); printState(pba001); pba001.incrSimTime(4); pba001.beginFireThrusters(); printState(pba001); pba001.endFireThrusters(); printState(pba001); assertTrue(pba001.isSuccess()); System.out.println("Completed testTorqueVector_pba001"); System.out.println(); System.out.println("-------------------------"); System.out.println(); } Assertion–Based Requirements Validation Pilot Validate each formalized PBA Validate the PBA is unambiguous, correct, and consistent. Validate each project-based formalized requirement has been modeled correctly to represent the intended requirement Create formalized unit tests for the scenarios

26. IV&V Facility 1/18/2016 26 Assertion–Based Requirements Validation Pilot Add system scenarios to SRM Derive system scenarios –Nominal, expected behavioral sequences –Behavioral sequences that should not be permitted –Detection and responses to adverse conditions SRM Correlations with SRM Project Requirements Analyze Results Findings Develop & Validate PBAs Execute Scenarios in Context of PBAs Develop & Validate IMAs Develop System Scenarios Identify events Define system scenarios Implement system scenarios Validate system scenarios

27. IV&V Facility 1/18/2016 27 Assertion–Based Requirements Validation Pilot Identify Events Identify events in the activity diagrams –Activities model the control of execution of one or more actions. Activities do not model sequences of event occurrences. –Activities do not often model timing constraints –Action names and control flow guards are defined in natural language and thus prone to ambiguity

28. IV&V Facility 1/18/2016 28 Assertion–Based Requirements Validation Pilot Define System Scenarios “Good” and “Bad” Scenarios –Used the semantics of UML sequence diagrams to define a scenario, by modeling sequence of event occurrences (i.e. action execution start/finish events) –Used guard conditions in activity diagrams that should not hold true in the context of the goal as means of creating “bad scenarios”

29. IV&V Facility 1/18/2016 29 Assertion–Based Requirements Validation Pilot Implement System Scenarios Scenarios are implemented in J-Unit and reference events in assertions

30. IV&V Facility 1/18/2016 30 Assertion–Based Requirements Validation Pilot Validate System Scenarios Use the IMAs to ensure that the scenarios return the expected results –Q1 scenarios complete without errors, as expected –Q2 scenarios do not cause any assertions to “pop” –Q3 scenarios result in detection of an adverse condition and respond in a correct way, as expected Scenarios represent a test oracle

31. IV&V Facility 1/18/2016 31 Assertion–Based Requirements Validation Pilot Execute scenarios in Context of PBAs Interested in how the PBAs react to the system scenarios –Which system scenarios ended as expected? –Which system scenarios ended differently than expected? –Which PBAs were not exercised? SRM Correlations with SRM Project Requirements Analyze Results Findings Develop & Validate PBAs Execute Scenarios in Context of PBAs Develop & Validate IMAs Develop System Scenarios Include PBAs in executable environment Execute system scenarios Collect results for further analysis

32. IV&V Facility 1/18/2016 32 Assertion–Based Requirements Validation Process Analyze Results Examine each assertion failure for: –Errors in constructing the SRM (system scenarios, IMAs, etc.) –Errors in constructing the PBA(s) –Errors in a requirement Fix the errors in the SRM Rerun test until no SRM or PBA construction errors are discovered. Therefore all unexpected results indicate findings SRM Correlations with SRM Project Requirements Analyze Results Findings Develop & Validate PBAs Execute Scenarios in Context of PBAs Develop & Validate IMAs Develop System Scenarios Ensure system scenario set is correct and complete Ensure PBA set is complete Ensure that PBA is in scope Ensure PBA is correct

33. IV&V Facility 1/18/2016 33 Assertion–Based Requirements Validation Pilot Observations Developing IMAs helps identify defects and deficiencies in the SRM. Doing this early in the SRM development process yields early benefits The process of examining the documented requirements helps to identify omissions in the correlation process The depth of understanding required to create IMAs and PBAs results in more rapid understanding of the system of interest Assertions best represent requirements of specific behaviors When creating PBAs for high level requirements such as “The system shall have the capability to do X”, these behavioral details need to be supplied from domain knowledge and analysis

34. IV&V Facility 1/18/2016 34 Assertion–Based Requirements Validation Pilot Observations It was easier than expected to: –Discover early questions and concerns that could lead to good assertions –Develop a high level understanding of what the system is supposed to do –Create higher level system scenarios that describe what the system is supposed to do –Develop Formal Assertion State Charts from Natural Language –Observe the lack of model depth, details –Discover possible missing behavioral IMAs –Write unit tests for IMAs and PBAs (although easier for IMAs if the SRM provides more information) –Write formalized Junit Tests –Test assertions against the System Scenarios

35. IV&V Facility 1/18/2016 35 Assertion–Based Requirements Validation Pilot Observations It was harder than expected to: –Vet good/worthwhile NL (informal assertions) from the list of answered questions –Create System Scenarios with the lack of model depth and details –Create System Scenarios that describe what the system is not supposed to do, and what the system is supposed to do under adverse conditions –Correlate system scenario events, IMA events, and PBA events without a data dictionary or complete Domain Model –Discover worthwhile PBAs from documented requirements that define and monitor a single event, and/or provide limited details and constraints –Discover matching IMAs and PBAs when there are limited requirement details –Discover matching IMAs and PBAs when the model is not developed to needed level of coverage/detail –Validate Requirements when the model is not developed to the needed level of coverage/detail –Set up the test rig for the first time

36. IV&V Facility 1/18/2016 36 Assertion–Based Requirements Validation Recommendations Develop IMAs early in the SRM development process Consider developing IMAs as part of the process for reviewing activity diagrams Continue refining techniques for using assertion- based methods for requirements validation

37. IV&V Facility 1/18/2016 37 Assertion–Based Requirements Validation Summary Model–Based Requirements Validation is part of the SRM Validation process –The purposes for using assertions include: Have ability to test conflicts and validate complex requirements Provides an advantage of using formal requirements over natural language Sets groundwork for automated testing –Use assertions to validate requirements that correspond to SRM behaviors and IMAs as unambiguous, correct, complete, consistent, and verifiable. –Use assertions to ensure that the right behaviors have been defined and the behaviors are of high quality. The right behaviors are those that adequately describe: What the system is supposed to do What the system in not supposed to do What the system is supposed to do under adverse conditions Assertion–based validation leverages our independent modeling to find issues earlier in the life–cycle

38. IV&V Facility 1/18/2016 38 Assertion–Based Requirements Validation Panel Discussion Michael Facemire John Ryan Bill Stanton Patrick Theeke

39. IV&V Facility 1/18/2016 39 References D. Drusinsky, “From AD to Assertions”; informal presentation, 2008. D. Drusinsky, J. B. Michael, and M. Shing, “A Framework for Computer-aided Validation”; Naval Postgraduate School, NPS- CS-07-008, 08/2007. K. Hurley, and D. Kranz, “Executable Reference Model”; informal presentation, 2008. Juno IVV Team, “Juno Level-4 Flight Software Requirements Validation Report”; NASA IV&V, 2008. D. Kranz, and J. Ryan, “Juno and Assertion Based Validation”; NASA IV&V, white paper, 2008. NASA IV&V Facility, “System Level Procedure 09-1”; 2007. S. Raque, “SRM Example Overview”; informal presentation, 2008. P. Theeke, “A process architecture for reference model based validation”; NASA IV&V, white paper, 2007. P. Theeke, “System Reference Models and Executable Assertions for Requirements Validation”; NASA IV&V, white paper, 2007.