1. Mapping Assurance to the Software Engineering Process Alfred H. Kromholz, Ph.D. The MITRE Corporation kromholz@ mitre.org+1-703.883.7331 Copyright © 2004 The MITRE Corporation

2. 2 Relation between Waterfall Process & V-Chart Plans & Rqmts Prod. Design Detail Design Unit Test Integ. Test Syst. Test Code Concept Accep. Test Concept Plans and Requirements Product Design Code Detail Design Unit Test Integration Test System Test Acceptance Test Change the shape from waterfall to fall and rise Traditional Waterfall The obvious question:Why bother?

3. Copyright © 2004 The MITRE Corporation 3 Conceptual Plans and Requirements Product Design Code Detail Design Unit Test Integration Test System Test Acceptance Test Needs System Requirements Subsystem Requirements Elements Element Requirements Element Test Subsystem Test Development V-Chart Starts with a Product Focus to Process Abstraction Analysis phase (Decomposition) Synthesis phase (Verification) Development V-Chart Shift from Product Focus to Process Abstraction

4. Copyright © 2004 The MITRE Corporation 4 For efficiency, focus on interfaces & interactions System Requirements Subsystem Requirements Element Requirements Subsystem Test Development V-Chart Process and Product Needs Elements Element Test System Test Acceptance Test Analysis phase (Product Explosion) Synthesis phase (Product Implosion) The structure of the decomposition is the product architecture. Analysis phaseSynthesis phase

5. Copyright © 2004 The MITRE Corporation 5 Development V-Chart Typical Generic Products of the Process Needs System Requirements Subsystem Requirements Elements Element Requirements Element Test Subsystem Test System Test Acceptance Test Needs Stmt System Design Spec Sub-syst Design Spec Detail Design Spec Test Report Test Report +

6. Copyright © 2004 The MITRE Corporation 6 Needs System Requirements Subsystem Requirements Elements Element Requirements Element Test Subsystem Test System Test Acceptance Test Development V-Chart When Do We Establish Test Criteria? Test Report Test Report + Needs Stmt Design Spec Design Spec Design Spec Accept Criteria Test Spec Test Spec Test Spec Development V-Chart Typically, We Establish Test Criteria on the Way Up Same products on this side Earlier products on this side Accept Criteria Test Spec Test Spec Test Spec Development V-Chart We Should Establish Test Criteria on the Way Down Generally, just barely before we test

7. Copyright © 2004 The MITRE Corporation 7 Validate / Verify V&V Development V-Chart Verify and Validate Before Staring Upward Needs Stmt Design Spec Design Spec Design Spec Accept Criteria Test Spec Test Spec Test Spec Test Report + Test Report Needs System Requirements Subsystem Requirements Elements Element Requirements Element Test Subsystem Test System Test Acceptance Test

8. Copyright © 2004 The MITRE Corporation 8 Validate / Verify V&V Development V-Chart Validate, Plan for Verify & DFT/DFM Early Needs Stmt Design Spec Design Spec Design Spec Accept Criteria Test Spec Test Spec Test Spec Test Report + Test Report Needs System Requirements Subsystem Requirements Elements Element Requirements Element Test Subsystem Test System Test Acceptance Test DFT/DFM Feedback DFT/DFM

9. Copyright © 2004 The MITRE Corporation 9 Needs System Requirements Subsystem Requirements Element Requirements Needs Stmt System Design Spec Sub-syst Design Spec Detail Design Spec Element Test Subsystem Test System Test Acceptance Test Lets Go Back to the Original Development V-Chart Rationale Test Report + Test Report Elements The Missing Rationale How did we get from here How did we get from here to here? Or here? Does it need to be captured? Recording the rationale for decomposition can help us

10. Copyright © 2004 The MITRE Corporation 10 Where Does Assurance Fit In? Two broad customer circumstances Functional expectations but no assurance requirements –Expectations may be informal – mass-market, shrink-wrapped products –Expectations may be formal – supplier requirements documents & specifications –Customer says, Convince me that the product meets expectations. –Customer may say, Why should I have confidence in your claims? Both functional requirements and assurance requirements –Customer tells supplier what it takes to convince –Assurance requirements may be direct – i.e., included in the specifications –Assurance requirements may be indirect – i.e, specifications identify product and/or process standards to be followed

11. Copyright © 2004 The MITRE Corporation 11 Customer ActivitiesSupplier Activities These activities take place during the a posteriori phase after the product is produced Sub-claim evaluation Claim evaluation Acceptance Assurance V-Chart Assurance Claim Evidence Body of Evidence Sub-Claims No customer-specified assurance requirements Two groups of assurance-related activities

12. Copyright © 2004 The MITRE Corporation 12 Expectations, standards, desiderata Claims re Expectations Sub-claims re Expectations a priori phase a posteriori phase Body of Evidence Sub-claim evaluation Claim evaluation Acceptance Argument The specific problem domain implies both the applicable standards and the Assurance Level required Assurance V-Chart (Augmented) Customer-specified assurance requirements But how do we know these are necessary and sufficient with respect to the previous level? Evidence-Item Classes We add an a priori phase that sets up an assurance case before a product is designed and produced As in the previous chart Tells the supplier what evidence is expected (and maybe in what form to present it) are decomposed into We add an argument that justifies how sub- claims support a claim and evidence supports the subclaims – are decomposed into for each decomposition activity

13. Copyright © 2004 The MITRE Corporation 13 DFT/DFM Feedback Evidence DFT/DFM Evidence DFT/DFM Evidence Needs System Requirements Subsystem Requirements Elements Element Requirements Element Test Subsystem Test System Test Evidence Acceptance Test Development V-Chart Evidence for Assurance Comes from All Development Activities Validate&Verify Evidence V&V Evidence V&V Evidence V&V Evidence Evidence is gathered in accordance with the evidence classes identified in the a priori phase. Remember the rationale for decomposition? This is where it comes in handy. Remember the rationale for decomposition? This is where it comes in handy. Remember the rationale for decomposition? This is where it comes in handy. Remember the rationale for decomposition? This is where it comes in handy.

14. Copyright © 2004 The MITRE Corporation 14 Assurance p rocess Development process Minimum Separation Development starts after 1 or 2 levels of Assurance decomposition Ideal Separation All necessary evidence identified before Development starts Assurance Completion Occurs after all evidence from Development is received Assurance Synthesis Starts as early as evidence classes are identified and evidence instantiations are available Time Relationship of Assurance & Development V-Charts How much time? Clearly, assurance requirements should be identified before development begins.

15. Current Work & Future Directions

16. Copyright © 2004 The MITRE Corporation 16 Using Adelard Safety Case Editor to Argue a Safety Case This illustrates the idea of a supplier making a claim to a purchaser that a product is safe. Thus the arrows go upward from evidence to argument to claim.

17. Copyright © 2004 The MITRE Corporation 17 Arguing a Safety Case: Basic Supplier Approach CLAIM Architecture is safe ARGUMENT Partitioning is proper ARGUMENT Multi-versioning Dissimilarity is properly provided ARGUMENT Safety Monitoring is provided EVIDENCE Partitioning Documentation EVIDENCE Multi-versioning Documentation EVIDENCE Safety-Monitoring Documentation Note that there is no justification provided to support the assertion that the given arguments support the claim. Argument is made on the basis of the evidence. Evidence is produced prior to making the claim. Collection of arguments is used to support the claim.

18. Copyright © 2004 The MITRE Corporation 18 Arguing a Safety Case: Standards-Based Supplier Approach CLAIM 2.3 Architecture is safe ARGUMENT 2.3.1 Partitioning is proper ARGUMENT 2.3.2 Multi-versioning Dissimilarity is properly provided ARGUMENT 2.3.3 Safety Monitoring is provided EVIDENCE 2.3.1 Partitioning Documentation EVIDENCE 2.3.2 Multi-versioning Documentation EVIDENCE 2.3.3 Safety-Monitoring Documentation However, there is still no justification provided. Indeed, many standards documents not only omit but deliberately exclude rationale. The supplier can include references to governing documents as part of the assurance package to help support assertions. (Numbering here is adapted from DO-178B, Software Considerations in Airborne Systems and Equipment Certification)

19. Copyright © 2004 The MITRE Corporation 19 CLAIM 2.3 Architecture is safe CLAIM 2.3.1 Partitioning is proper CLAIM 2.3.2 Multi-versioning Dissimilarity is properly provided CLAIM 2.3.3 Safety Monitoring is provided EVIDENCE 2.3.1 Partitioning Documentation EVIDENCE 2.3.2 Multi-versioning Documentation EVIDENCE 2.3.3 Safety-Monitoring Documentation ARGUMENT Partitioning, multi-versioning, and safety monitoring are necessary and sufficient for a safe architecture Arguing a Safety Case: Justified Supplier Approach Justification can be provided by the supplier to support the claim... but there is no guarantee that this will satisfy the customer.

20. Copyright © 2004 The MITRE Corporation 20 REQUIREMENT 2.3 Architecture is safe REQUIREMENT 2.3.1 Partitioning is proper REQUIREMENT 2.3.2 Multi-versioning Dissimilarity is properly provided REQUIREMENT 2.3.3 Safety Monitoring is provided EVIDENCE class 2.3.1 Partitioning Documentation EVIDENCE class 2.3.2 Multi-versioning Documentation EVIDENCE class 2.3.3 Safety-Monitoring Documentation ARGUMENT 2.3 Partitioning, multi-versioning, and safety monitoring are necessary and sufficient for a safe architecture Arguing a Safety Case: Requirements-Based Approach In the Requirements- Based approach, the arrows are top-down Customer defines what safe architecture means (at least internally to the acquiring organization) Customer identifies expectations regarding proof that architecture is safe Note:The bottom level calls for classes of evidence, not specific documents. Customer defines that architecture must be safe Customer establishes requirements based on decomposed definition

21. Copyright © 2004 The MITRE Corporation 21 REQMENT 2.3 Architecture is safe REQMENT 2.3.1 Partitioning is proper REQMENT 2.3.2 Multi-versioning Dissimilarity is properly provided REQMENT 2.3.3 Safety Monitoring is provided EVIDENCE Class 2.3.1 Partitioning Documentation EVIDENCE Class 2.3.2 Multi-versioning Documentation EVIDENCE Class 2.3.3 Safety- Monitoring Documentation ARGUMENT 2.3 Partitioning, multi- versioning, and safety monitoring are necessary and sufficient for a safe architecture Arguing a Safety Case: Consolidated Flow CLAIM 2.3 Architecture is safe EVIDENCE 2.3.1 Partitioning Documentation EVIDENCE 2.3.2 Multi-versioning Documentation EVIDENCE 2.3.3 Safety- Monitoring Documentation CLAIM 2.3.1 Partitioning is proper CLAIM 2.3.2 Multi-versioning Dissimilarity is properly provided CLAIM 2.3.3 Safety Monitoring is provided ARGUMENT 2.3 Partitioning, multi- versioning, and safety monitoring are necessary and sufficient for a safe architecture Customer specifications (a priori) Supplier evidence (a posteriori) Works remarkably like a V-Chart, doesnt it?

22. Copyright © 2004 The MITRE Corporation 22