Mapping Assurance to the Software Engineering Process Alfred H. Kromholz, Ph.D. The MITRE Corporation kromholz@ mitre.org +1-703.883.7331 Copyright © 2006 The MITRE Corporation

Relation between Waterfall Process & V-Chart Concept Plans & Rqmts Prod. Design Detail Design Code Traditional Waterfall Unit Test Integ. Test Syst. Test Accep. Test Concept Plans and Requirements Product Design Code Detail Design Unit Test Integration Test System Test Acceptance Test Change the shape from waterfall to fall and rise The obvious question: Why bother? Copyright © 2006 The MITRE Corporation

Shift from Product Focus to Process Abstraction Development V-Chart Starts with a Product Focus to Process Abstraction Conceptual Plans and Requirements Product Design Code Detail Design Unit Test Integration Test System Test Acceptance Test Conceptual Plans and Requirements Product Design Code Detail Design Unit Test Integration Test System Test Acceptance Test Elements Needs System Requirements Subsystem Requirements Element Requirements Element Test Subsystem Test Copyright © 2006 The MITRE Corporation

Partition the Process Abstraction Development V-Chart Starts with a Product Focus to Process Abstraction Analysis phase (Decomposition) Synthesis phase (Verification) Needs System Requirements Subsystem Requirements Elements Element Requirements Element Test Subsystem Test Acceptance Test System Test Copyright © 2006 The MITRE Corporation

Development V-Chart Process and Product Analysis phase Synthesis phase Analysis phase (Product Explosion) Synthesis phase (Product Implosion) Needs Acceptance Test The structure of the decompo-sition is the product architecture The structure of the synthesis is the test architecture For efficiency, focus on interfaces & interactions System Requirements System Test Subsystem Requirements Subsystem Test Element Requirements Element Test Elements Copyright © 2006 The MITRE Corporation

Typical Generic “Paper” Products of the Process Development V-Chart Typical Generic “Paper” Products of the Process Needs Stmt Needs System Requirements Subsystem Requirements Elements Element Requirements Element Test Subsystem Test System Test Acceptance Test Test Report + System Design Spec Test Report Sub-syst Design Spec Test Report Detail Design Spec Test Report Copyright © 2006 The MITRE Corporation

Typically, We Establish Test Criteria on the Way Up Development V-Chart Typically, We Establish Test Criteria on the Way Up Development V-Chart When Do We Establish Test Criteria? Generally, just barely before we test Needs System Requirements Subsystem Requirements Elements Element Requirements Element Test Subsystem Test System Test Acceptance Test Design Spec Test Report Test Report + Needs Stmt Accept Criteria Accept Criteria Test Spec Test Spec Test Spec Test Spec Test Spec Test Spec Copyright © 2006 The MITRE Corporation

We Should Establish Test Criteria on the Way Down Development V-Chart We Should Establish Test Criteria on the Way Down Needs System Requirements Subsystem Requirements Elements Element Requirements Element Test Subsystem Test System Test Acceptance Test Design Spec Test Report Test Report + Needs Stmt Accept Criteria Accept Criteria Test Spec Test Spec Test Spec Test Spec Earlier products on this side Original products on this side Copyright © 2006 The MITRE Corporation

Verify and Validate Before Starting Upward Development V-Chart Verify and Validate Before Starting Upward Needs Stmt Validate / Verify Accept Criteria Test Report + Needs Acceptance Test Design Spec V&V Test Report System Requirements Test Spec System Test Design Spec V&V Test Report Subsystem Requirements Test Spec Subsystem Test The principle: Be sure we’re asking for – and testing for – the right stuff before we continue downward. Design Spec V&V Test Report Element Requirements Test Spec Element Test Elements Copyright © 2006 The MITRE Corporation

Validate, Plan for Verify & DFT/DFM Early Development V-Chart Validate, Plan for Verify & DFT/DFM Early DFT – Design for Test(ability) DFM – Design for Maintainability Needs Stmt Validate / Verify DFT/DFM Feedback Accept Criteria Test Report + Needs Acceptance Test Design Spec V&V Test Report System Requirements DFT/DFM Test Spec System Test Design Spec V&V Test Report Subsystem Requirements DFT/DFM Test Spec Subsystem Test Testers provide the designers with early feedback – better still, participate in the design process. Design Spec V&V Test Report Element Requirements Test Spec Element Test Elements Copyright © 2006 The MITRE Corporation

How Does This Apply to High-Confidence Software? That’s the Traditional Product/Process V-Chart – with a small recommended change How Does This Apply to High-Confidence Software?

Let’s Go Back to the Original Development V-Chart The “Missing” Rationale Needs Stmt How did we get from here How did we get from here to here? Needs Test Report + Acceptance Test System Design Spec System Requirements Test Report System Test Or here? Sub-syst Design Spec Subsystem Requirements Test Report Subsystem Test Or here? Detail Design Spec Element Requirements Test Report Element Test Or here? Elements Copyright © 2006 The MITRE Corporation

Let’s Go Back to the Original Development V-Chart The “Missing” Rationale Recording the rationale for decomposition can help us Needs Stmt Needs Test Report + Rationale Acceptance Test System Design Spec System Requirements Test Report System Test Rationale Sub-syst Design Spec Subsystem Requirements Does it need to be captured? Yes, for two reasons: 2. When the requirements change, you can retrace the design process and know where to change the design. 1. To provide confidence that the design was done right in the first place. Test Report Subsystem Test Rationale Detail Design Spec Element Requirements Test Report Element Test Rationale Elements Copyright © 2006 The MITRE Corporation

High-Confidence and Assurance What do we mean by “High-Confidence” systems? Functional expectations are met Of course Performance expectations are met The “ilities-plus” – reliability, maintainability, safety, security, … What do we mean by “Assurance”? Extent of verification by test and inspection is reasonable Agreed-to by supplier and customer (and/or regulator) Sufficient documentation is provided to demonstrate that the untested aspects were adequately taken into account Product documentation – Analysis, design, build, test Process documentation – ditto, plus QA, Req Mgt, Config Mgt, Risk Mgt but it’s unfeasible – technically & financially – to test software 100% so we need assurance that the rest of the system meets expectations as well Copyright © 2006 The MITRE Corporation

Where Do High-Confidence and Assurance Fit In? Two broad customer circumstances Functional expectations but no assurance requirements Expectations may be informal – mass-market, “shrink-wrapped” products Expectations may be formal – supplier requirements documents & specifications Customer says, “Convince me that the product meets expectations.” Customer may say, “Why should I have confidence in your claims?” Both functional requirements and assurance requirements Customer tells supplier what it takes to convince Assurance requirements may be direct – i.e., included in the specifications Assurance requirements may be indirect – i.e, specifications identify product and/or process standards to be followed Copyright © 2006 The MITRE Corporation

Assurance V-Chart Customer expectations without specified assurance requirements Two groups of assurance-related activities These activities take place during the a posteriori phase after the product is produced Supplier Activities Customer Activities produce evidence evaluate evidence Assurance Claim Acceptance Evidence Claim evaluation Sub-Claims Sub-claim evaluation Sub-Claims Sub-claim evaluation Sub-Claims Body of Evidence Copyright © 2006 The MITRE Corporation

Assurance V-Chart (Augmented) Customer-specified assurance requirements We add an a priori phase that sets up assurance case expectations before a product is designed and produced Expectations, standards, desiderata a priori phase Body of Evidence Sub-claim evaluation Claim evaluation Acceptance a posteriori phase are decomposed into Specific problem domain determines both the applicable standards and the Assurance Level required Claims re Expectations are decomposed into Sub-claims re Expectations are decomposed into Sub-claims re Expectations are decomposed into Tells supplier what evidence is expected (and maybe in what form to present it) Evidence-Item Classes Copyright © 2006 The MITRE Corporation

Assurance V-Chart (Augmented) Customer-specified assurance requirements How do we know that each set of sub-claims is necessary and sufficient with respect to the previous level? We add an “argument” that justifies how sub-claims support a claim and how evidence supports the sub-claims for each decomposition activity Expectations, standards, desiderata a priori phase Body of Evidence Sub-claim evaluation Claim evaluation Acceptance a posteriori phase are decomposed into Argument Claims re Expectations are decomposed into Argument Sub-claims re Expectations are decomposed into Argument Sub-claims re Expectations are decomposed into Argument Evidence-Item Classes Copyright © 2006 The MITRE Corporation

Now Let’s Put the Development V-Chart and the Assurance V-Chart Together

Evidence for Assurance Comes from All Development Activities Development V-Chart Evidence for Assurance Comes from All Development Activities Evidence is gathered in accordance with the evidence classes identified in the a priori phase. Validate&Verify Evidence Needs DFT/DFM Feedback Evidence Acceptance Test Evidence Evidence V&V Evidence System Requirements System Test DFT/DFM Evidence Evidence Evidence V&V Evidence Subsystem Requirements DFT/DFM Evidence Subsystem Test Evidence Evidence V&V Evidence Element Requirements Element Test Remember the “rationale” for decomposition? This is where it comes in handy. Evidence Evidence Elements Evidence Copyright © 2006 The MITRE Corporation

Time Relationship of Assurance & Development V-Charts Assurance process 4. Assurance Completion Occurs after all evidence from Development is received Development process How much time? Assurance requirements should be identified before development begins. Assurance Process starts before and ends after Development Process 2. Minimum Separation Development starts after 1 or 2 levels of Assurance decomposition 1, Ideal Separation All necessary evidence identified before Development starts 3. Assurance Synthesis Starts as early as evidence classes are identified and evidence instantiations are available Copyright © 2006 The MITRE Corporation

Recent Work & Future Directions

Using Adelard Safety Case Editor to Argue a Safety Case Next slide focuses here This illustrates the idea of a supplier making a claim to a purchaser that a product is safe. Thus the arrows go upward from evidence to argument to claim. Copyright © 2006 The MITRE Corporation

Arguing a Safety Case: Basic Supplier Approach CLAIM Architecture is safe Collection of arguments is used to support the claim. ARGUMENT Partitioning is proper ARGUMENT Multi-versioning Dissimilarity is properly provided ARGUMENT Safety Monitoring is provided Argument is made on the basis of the evidence. EVIDENCE Partitioning Documentation EVIDENCE Multi-versioning Documentation EVIDENCE Safety-Monitoring Documentation Evidence was produced prior to making the claim. Note that there is no justification provided to support the assertion that the given arguments support the claim. Copyright © 2006 The MITRE Corporation

Arguing a Safety Case: Standards-Based Supplier Approach CLAIM 2.3 Architecture is safe ARGUMENT 2.3.1 Partitioning is proper 2.3.2 Multi-versioning Dissimilarity is properly provided 2.3.3 Safety Monitoring is provided EVIDENCE Partitioning Documentation Multi-versioning Documentation Safety-Monitoring Documentation The supplier can include references to governing documents as part of the assurance package to help support assertions. (Numbering here is adapted from DO-178B, Software Considerations in Airborne Systems and Equipment Certification) However, there is still no justification provided. Indeed, many standards documents not only omit but deliberately exclude rationale. Copyright © 2006 The MITRE Corporation

Arguing a Safety Case: Justified Supplier Approach CLAIM 2.3 Architecture is safe Justification can be provided by the supplier to support the claim ... ARGUMENT Partitioning, multi-versioning, and safety monitoring are necessary and sufficient for a safe architecture but there is no guarantee that this will satisfy the customer. CLAIM 2.3.1 Partitioning is proper CLAIM 2.3.2 Multi-versioning Dissimilarity is properly provided CLAIM 2.3.3 Safety Monitoring is provided EVIDENCE 2.3.1 Partitioning Documentation EVIDENCE 2.3.2 Multi-versioning Documentation EVIDENCE 2.3.3 Safety-Monitoring Documentation Copyright © 2006 The MITRE Corporation

Arguing a Safety Case: Requirements-Based Approach 2.3 Architecture is safe 2.3.1 Partitioning is proper 2.3.2 Multi-versioning Dissimilarity is properly provided 2.3.3 Safety Monitoring is provided EVIDENCE class Partitioning Documentation Multi-versioning Documentation Safety-Monitoring Documentation ARGUMENT 2.3 Partitioning, multi-versioning, and safety monitoring are necessary and sufficient for a safe architecture In the Requirements-Based approach, the arrows are top-down Customer defines that architecture must be safe Customer defines what “safe architecture” means (at least internally to the acquiring organization) Customer establishes requirements based on decomposed definition Customer identifies expectations regarding proof that architecture is safe Note: The bottom level calls for classes of evidence, not specific documents or content (i.e, what but not how) Copyright © 2006 The MITRE Corporation

Arguing a Safety Case: Consolidated Flow Customer specifications (a priori) Supplier evidence (a posteriori) REQ’MENT 2.3 Architecture is safe CLAIM 2.3 Architecture is safe Works remarkably like a V-Chart, doesn’t it? ARGUMENT 2.3 Partitioning, multi-versioning, and safety monitoring are necessary and sufficient for a safe architecture ARGUMENT 2.3 Partitioning, multi-versioning, and safety monitoring are necessary and sufficient for a safe architecture REQ’MENT 2.3.1 Partitioning is proper 2.3.2 Multi-versioning Dissimilarity is properly provided 2.3.3 Safety Monitoring is provided CLAIM 2.3.1 Partitioning is proper 2.3.2 Multi-versioning Dissimilarity is properly provided 2.3.3 Safety Monitoring is provided EVIDENCE Class 2.3.1 Partitioning Documentation 2.3.2 Multi-versioning Documentation 2.3.3 Safety-Monitoring Documentation EVIDENCE 2.3.1 Partitioning Documentation 2.3.2 Multi-versioning Documentation 2.3.3 Safety-Monitoring Documentation Copyright © 2006 The MITRE Corporation