1. Mapping Assurance to the Software Engineering Process
Alfred H. Kromholz, Ph.D.
The MITRE Corporation
mitre.org
Copyright © 2006 The MITRE Corporation

2. Relation between Waterfall Process & V-Chart
Concept
Plans & Rqmts
Prod. Design
Detail Design
Code
Traditional Waterfall
Unit Test
Integ. Test
Syst. Test
Accep. Test
Concept
Plans and Requirements
Product Design
Code
Detail Design
Unit Test
Integration Test
System Test
Acceptance Test
Change the shape from waterfall
to fall and rise
The obvious question:
Why bother?
Copyright © 2006 The MITRE Corporation

3. Shift from Product Focus to Process Abstraction Development V-Chart
Starts with a Product Focus to Process Abstraction
Conceptual
Plans and Requirements
Product Design
Code
Detail Design
Unit Test
Integration Test
System Test
Acceptance Test
Conceptual
Plans and Requirements
Product Design
Code
Detail Design
Unit Test
Integration Test
System Test
Acceptance Test
Elements
Needs
System Requirements
Subsystem Requirements
Element Requirements
Element Test
Subsystem Test
Copyright © 2006 The MITRE Corporation

4. Partition the Process Abstraction Development V-Chart
Starts with a Product Focus to Process Abstraction
Analysis phase
(Decomposition)
Synthesis phase
(Verification)
Needs
System Requirements
Subsystem Requirements
Elements
Element Requirements
Element Test
Subsystem Test
Acceptance Test
System Test
Copyright © 2006 The MITRE Corporation

5. Development V-Chart Process and Product
Analysis phase
Synthesis phase
Analysis phase
(Product Explosion)
Synthesis phase
(Product Implosion)
Needs
Acceptance Test
The structure
of the decompo-sition is the product architecture
The structure
of the synthesis is the test architecture
For efficiency, focus on interfaces & interactions
System Requirements
System Test
Subsystem Requirements
Subsystem Test
Element Requirements
Element Test
Elements
Copyright © 2006 The MITRE Corporation

6. Typical Generic “Paper” Products of the Process
Development V-Chart
Typical Generic “Paper” Products of the Process
Needs
Stmt
Needs
System Requirements
Subsystem Requirements
Elements
Element Requirements
Element Test
Subsystem Test
System Test
Acceptance Test
Test Report +
System Design
Spec
Test Report
Sub-syst Design
Spec
Test Report
Detail Design
Spec
Test Report
Copyright © 2006 The MITRE Corporation

7. Typically, We Establish Test Criteria on the Way Up
Development V-Chart
Typically, We Establish Test Criteria on the Way Up
Development V-Chart
When Do We Establish Test Criteria?
Generally, just barely before we test
Needs
System Requirements
Subsystem Requirements
Elements
Element Requirements
Element Test
Subsystem Test
System Test
Acceptance Test
Design
Spec
Test Report
Test Report +
Needs
Stmt
Accept
Criteria
Accept
Criteria
Test
Spec
Test
Spec
Test
Spec
Test
Spec
Test
Spec
Test
Spec
Copyright © 2006 The MITRE Corporation

8. We Should Establish Test Criteria on the Way Down
Development V-Chart
We Should Establish Test Criteria on the Way Down
Needs
System Requirements
Subsystem Requirements
Elements
Element Requirements
Element Test
Subsystem Test
System Test
Acceptance Test
Design
Spec
Test Report
Test Report +
Needs
Stmt
Accept
Criteria
Accept
Criteria
Test
Spec
Test
Spec
Test
Spec
Test
Spec
Earlier products on this side
Original products
on this side
Copyright © 2006 The MITRE Corporation

9. Verify and Validate Before Starting Upward
Development V-Chart
Verify and Validate Before Starting Upward
Needs
Stmt
Validate / Verify
Accept
Criteria
Test Report +
Needs
Acceptance Test
Design
Spec
V&V
Test Report
System Requirements
Test
Spec
System Test
Design
Spec
V&V
Test Report
Subsystem Requirements
Test
Spec
Subsystem Test
The principle: Be sure we’re asking for – and testing for – the right stuff before we continue downward.
Design
Spec
V&V
Test Report
Element Requirements
Test
Spec
Element Test
Elements
Copyright © 2006 The MITRE Corporation

10. Validate, Plan for Verify & DFT/DFM Early
Development V-Chart
Validate, Plan for Verify & DFT/DFM Early
DFT – Design for Test(ability)
DFM – Design for Maintainability
Needs
Stmt
Validate / Verify
DFT/DFM Feedback
Accept
Criteria
Test Report +
Needs
Acceptance Test
Design
Spec
V&V
Test Report
System Requirements
DFT/DFM
Test
Spec
System Test
Design
Spec
V&V
Test Report
Subsystem Requirements
DFT/DFM
Test
Spec
Subsystem Test
Testers provide the designers with early feedback – better still, participate in the design process.
Design
Spec
V&V
Test Report
Element Requirements
Test
Spec
Element Test
Elements
Copyright © 2006 The MITRE Corporation

11. How Does This Apply to High-Confidence Software?
That’s the Traditional Product/Process V-Chart – with a small recommended change
How Does This Apply to High-Confidence Software?

12. Let’s Go Back to the Original Development V-Chart
The “Missing” Rationale
Needs
Stmt
How did we get from here
How did we get from here to here?
Needs
Test Report +
Acceptance Test
System Design
Spec
System Requirements
Test Report
System Test
Or here?
Sub-syst Design
Spec
Subsystem Requirements
Test Report
Subsystem Test
Or here?
Detail Design
Spec
Element Requirements
Test Report
Element Test
Or here?
Elements
Copyright © 2006 The MITRE Corporation

13. Let’s Go Back to the Original Development V-Chart
The “Missing” Rationale
Recording the rationale for decomposition can help us
Needs
Stmt
Needs
Test Report +
Rationale
Acceptance Test
System Design
Spec
System Requirements
Test Report
System Test
Rationale
Sub-syst Design
Spec
Subsystem Requirements
Does it need to be captured?
Yes, for two reasons:
2. When the requirements change, you can retrace the design process and know where to change the design.
1. To provide confidence that the design was done right in the first place.
Test Report
Subsystem Test
Rationale
Detail Design
Spec
Element Requirements
Test Report
Element Test
Rationale
Elements
Copyright © 2006 The MITRE Corporation

14. High-Confidence and Assurance
What do we mean by “High-Confidence” systems?
Functional expectations are met
Of course
Performance expectations are met
The “ilities-plus” – reliability, maintainability, safety, security, …
What do we mean by “Assurance”?
Extent of verification by test and inspection is reasonable
Agreed-to by supplier and customer (and/or regulator)
Sufficient documentation is provided to demonstrate that the untested aspects were adequately taken into account
Product documentation – Analysis, design, build, test
Process documentation – ditto, plus QA, Req Mgt, Config Mgt, Risk Mgt
but it’s unfeasible – technically & financially – to test software 100%
so we need assurance that the rest of the system meets expectations as well
Copyright © 2006 The MITRE Corporation

15. Where Do High-Confidence and Assurance Fit In?
Two broad customer circumstances
Functional expectations but no assurance requirements
Expectations may be informal – mass-market, “shrink-wrapped” products
Expectations may be formal – supplier requirements documents & specifications
Customer says, “Convince me that the product meets expectations.”
Customer may say, “Why should I have confidence in your claims?”
Both functional requirements and assurance requirements
Customer tells supplier what it takes to convince
Assurance requirements may be direct – i.e., included in the specifications
Assurance requirements may be indirect – i.e, specifications identify product and/or process standards to be followed
Copyright © 2006 The MITRE Corporation

16. Assurance V-Chart
Customer expectations without specified assurance requirements
Two groups of assurance-related activities
These activities take place during the
a posteriori phase
after the product is produced
Supplier Activities
Customer Activities
produce evidence
evaluate evidence
Assurance
Claim
Acceptance
Evidence
Claim evaluation
Sub-Claims
Sub-claim evaluation
Sub-Claims
Sub-claim evaluation
Sub-Claims
Body of Evidence
Copyright © 2006 The MITRE Corporation

17. Assurance V-Chart (Augmented)
Customer-specified assurance requirements
We add an a priori phase that sets up assurance case expectations before a product is designed and produced
Expectations, standards, desiderata
a priori phase
Body of Evidence
Sub-claim evaluation
Claim evaluation
Acceptance
a posteriori phase
are decomposed into
Specific problem domain determines both the applicable standards and the Assurance Level required
Claims re Expectations
are decomposed into
Sub-claims re Expectations
are decomposed into
Sub-claims re Expectations
are decomposed into
Tells supplier what evidence is expected (and maybe in what form to present it)
Evidence-Item Classes
Copyright © 2006 The MITRE Corporation

18. Assurance V-Chart (Augmented)
Customer-specified assurance requirements
How do we know that each set of sub-claims is necessary and sufficient with respect to the previous level?
We add an “argument” that justifies how sub-claims support a claim and how evidence supports the sub-claims for each decomposition activity
Expectations, standards, desiderata
a priori phase
Body of Evidence
Sub-claim evaluation
Claim evaluation
Acceptance
a posteriori phase
are decomposed into
Argument
Claims re Expectations
are decomposed into
Argument
Sub-claims re Expectations
are decomposed into
Argument
Sub-claims re Expectations
are decomposed into
Argument
Evidence-Item Classes
Copyright © 2006 The MITRE Corporation

19. Now Let’s Put the Development V-Chart and the Assurance V-Chart Together

20. Evidence for Assurance Comes from All Development Activities
Development V-Chart
Evidence for Assurance Comes from All Development Activities
Evidence is gathered in accordance with the evidence classes identified in the a priori phase.
Validate&Verify
Evidence
Needs
DFT/DFM Feedback
Evidence
Acceptance Test
Evidence
Evidence
V&V
Evidence
System Requirements
System Test
DFT/DFM
Evidence
Evidence
Evidence
V&V
Evidence
Subsystem Requirements
DFT/DFM
Evidence
Subsystem Test
Evidence
Evidence
V&V
Evidence
Element Requirements
Element Test
Remember the “rationale” for decomposition?
This is where it comes in handy.
Evidence
Evidence
Elements
Evidence
Copyright © 2006 The MITRE Corporation

21. Time Relationship of Assurance & Development V-Charts
Assurance process
4. Assurance Completion
Occurs after all evidence from Development is received
Development process
How much time?
Assurance requirements should be identified before development begins.
Assurance Process starts before and ends after Development Process
2. Minimum Separation
Development starts after 1 or 2 levels of Assurance decomposition
1, Ideal Separation
All necessary evidence identified before Development starts
3. Assurance Synthesis
Starts as early as evidence classes are identified and evidence instantiations are available
Copyright © 2006 The MITRE Corporation

22. Recent Work & Future Directions

23. Using Adelard Safety Case Editor to Argue a Safety Case
Next slide focuses here
This illustrates the idea of a supplier making a claim to a purchaser that a product is safe. Thus the arrows go upward from evidence to argument to claim.
Copyright © 2006 The MITRE Corporation

24. Arguing a Safety Case: Basic Supplier Approach
CLAIM
Architecture is safe
Collection of arguments is used to support the claim.
ARGUMENT
Partitioning is proper
ARGUMENT
Multi-versioning Dissimilarity is properly provided
ARGUMENT
Safety Monitoring is provided
Argument is made on the basis of the evidence.
EVIDENCE
Partitioning Documentation
EVIDENCE
Multi-versioning Documentation
EVIDENCE
Safety-Monitoring Documentation
Evidence was produced prior to making the claim.
Note that there is no justification provided to support the assertion that the given arguments support the claim.
Copyright © 2006 The MITRE Corporation

25. Arguing a Safety Case: Standards-Based Supplier Approach
CLAIM
2.3
Architecture is safe
ARGUMENT
2.3.1
Partitioning is proper
2.3.2
Multi-versioning Dissimilarity is properly provided
2.3.3
Safety Monitoring is provided
EVIDENCE
Partitioning Documentation
Multi-versioning Documentation
Safety-Monitoring Documentation
The supplier can include references to governing documents as part of the assurance package to help support assertions.
(Numbering here is adapted from DO-178B, Software Considerations in Airborne Systems and Equipment Certification)
However, there is still no justification provided. Indeed, many standards documents not only omit but deliberately exclude rationale.
Copyright © 2006 The MITRE Corporation

26. Arguing a Safety Case: Justified Supplier Approach
CLAIM
2.3
Architecture is safe
Justification can be provided by the supplier to support the claim ...
ARGUMENT
Partitioning, multi-versioning, and safety monitoring are necessary and sufficient for a safe architecture
but there is no guarantee that this will satisfy the customer.
CLAIM
2.3.1
Partitioning is proper
CLAIM
2.3.2
Multi-versioning Dissimilarity is properly provided
CLAIM
2.3.3
Safety Monitoring is provided
EVIDENCE
2.3.1
Partitioning Documentation
EVIDENCE
2.3.2
Multi-versioning Documentation
EVIDENCE
2.3.3
Safety-Monitoring Documentation
Copyright © 2006 The MITRE Corporation

27. Arguing a Safety Case: Requirements-Based Approach
2.3
Architecture is safe
2.3.1
Partitioning is proper
2.3.2
Multi-versioning Dissimilarity is properly provided
2.3.3
Safety Monitoring is provided
EVIDENCE class
Partitioning Documentation
Multi-versioning Documentation
Safety-Monitoring Documentation
ARGUMENT 2.3
Partitioning, multi-versioning, and safety monitoring are necessary and sufficient for a safe architecture
In the Requirements-Based approach, the arrows are top-down
Customer defines that architecture must be safe
Customer defines what “safe architecture” means (at least internally to the acquiring organization)
Customer establishes requirements based on decomposed definition
Customer identifies expectations regarding proof that architecture
is safe
Note: The bottom level calls for classes of evidence,
not specific documents or content (i.e, what but not how)
Copyright © 2006 The MITRE Corporation

28. Arguing a Safety Case: Consolidated Flow
Customer specifications
(a priori)
Supplier evidence
(a posteriori)
REQ’MENT
2.3
Architecture is safe
CLAIM
2.3
Architecture is safe
Works remarkably like a V-Chart, doesn’t it?
ARGUMENT 2.3
Partitioning, multi-versioning, and safety monitoring are necessary and sufficient for a safe architecture
ARGUMENT 2.3
Partitioning, multi-versioning, and safety monitoring are necessary and sufficient for a safe architecture
REQ’MENT
2.3.1
Partitioning is proper
2.3.2
Multi-versioning Dissimilarity is properly provided
2.3.3
Safety Monitoring is provided
CLAIM
2.3.1
Partitioning is proper
2.3.2
Multi-versioning Dissimilarity is properly provided
2.3.3
Safety Monitoring is provided
EVIDENCE Class
2.3.1
Partitioning Documentation
2.3.2
Multi-versioning Documentation
2.3.3
Safety-Monitoring Documentation
EVIDENCE
2.3.1
Partitioning Documentation
2.3.2
Multi-versioning Documentation
2.3.3
Safety-Monitoring Documentation
Copyright © 2006 The MITRE Corporation