1. © 2013 Carnegie Mellon University Measuring Assurance Case Confidence using Baconian Probabilities Charles B. Weinstock John B. Goodenough Ari Z. Klein May 19, 2013

2. 2 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University Copyright 2013 Carnegie Mellon University and IEEE This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. This material has been approved for public release and unlimited distribution. This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu.permission@sei.cmu.edu DM-0000387

3. 3 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University The Problem How confident in C1? Why? What does it mean to have confidence? What could be done to improve confidence? Why?

4. 4 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University How is Confidence in a Hypothesis Increased? A classic philosophical problem: Determining the basis for belief in a hypothesis when it is impossible to examine every possible circumstance covered by the hypothesis Use Induction Enumerative: number of confirming instances (Pascalian) Eliminative: variety of reasons for doubt (Baconian) Measuring support for a hypothesis/claim – Enumerative: support increases with number of confirmations – Eliminative: support increases with the number of excluded alternative explanations, i.e., by eliminating reasons for doubting the claim Defeaters: reasons for doubting a claim

5. 5 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University A Baconian Theory of AC Confidence Confidence is the degree of belief We grade “degree of belief” in terms of the number of eliminated defeaters (reasons for doubt) i out of n reasons for doubt (i/n) 0/n – no confidence n/n – no remaining reasons for doubt 8/10 – residual doubt Fundamental principle: Build confidence by eliminating reasons to doubt the validity of: Claims (look for counter-examples and why they can’t occur) Evidence (look for reasons the evidence might be invalid and show those conditions do not hold) Inference rules (look for conditions under which the rule is not valid and why those conditions do not hold) As reasons for doubt are eliminated, confidence grows (eliminative induction)

6. 6 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University

7. 7 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University Citigroup Tower – New York The design passed all reviews The wind tunnel (and other) tests were successful All parties involved were convinced the building was safe But… The design accounted for straight on wind, not quartering wind Still had adequate safety margins The design accounted for straight on wind, not quartering wind Still had adequate safety margins But construction replaced welded joints with bolted joints The margin of safety was lost

8. 8 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University Concerns with the Baconian Approach What if a relevant defeater has not been identified? What if a defeater cannot be completely eliminated? Surely not all defeaters are of equal importance. How is this handled? Baconian probability seems rather weak in contrast to Bayesian or Pascalian probability. What is being gained or lost? The potential number of defeaters seems incredibly large for a real system. Is this approach practical?

9. 9 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University Unidentified Defeaters Issue: Does the failure to identify a relevant defeater lead to an inflated sense of confidence? The inability to identify some defeaters is itself a reason for doubt that needs to be recognized in a case. An assessment process for reviewing a case will need to take the possibility of missing defeaters into account. A similar problem exists with the use of assurance cases. How can you be sure that your hazard analysis has covered all potential hazards? We always admit the possibility that additional defeaters will be identified in the future (this is what defeasible reasoning is all about). – When we say that we have “complete” confidence in a claim (i.e., that the claim has Baconian probability n|n) we understand that this only reflects what we knew at a particular point in time. The concepts of eliminative induction and defeasible reasoning help in developing sound and complete arguments.

10. 10 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University Partially Eliminated Defeaters Issue: How are defeaters that cannot be completely eliminated dealt with? An argument should state a claim that, if true, serves to completely eliminate an associated defeater. There may be residual doubts about the truth of such a claim, in which case, the same doubts apply to the defeater’s elimination – as an argument is refined, one eventually develops defeaters that are “obviously” eliminated by associated evidence If it is impossible to eliminate some lowest level defeaters, then the associated doubt leads to incomplete elimination of a higher level defeater (and the claim that it defeats.) A goal in developing a convincing argument is to formulate low level defeaters that can be eliminated by appropriate evidence.

11. 11 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University Defeaters of Different (Relative) Importance Issue: It seems reasonable that eliminating some defeaters would lead to higher levels of confidence than eliminating others (of lesser importance). Independent of relative defeater importance, 1|2 (some doubts eliminated) always represents more confidence than 0|2 (no doubts eliminated), and 2|2 (all doubts eliminated) always represents complete confidence Even though the hazards that are represented in a safety case have different likelihoods and impacts relative to one another, they all must be demonstrably mitigated in order to establish sufficient confidence that the system is safe. – Assessing the relative importance of hazards is not practically profitable. A system developer would not use the elimination of low impact defeaters to justify an increase in confidence any more than he would represent minimally unlikely and impactful safety hazards in a safety case.

12. 12 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University Why Baconian Probability? Issue: What is being gained and lost (over Pascalian or Bayesian probability) with this approach? With eliminative induction we learn something concrete about why a system works, and with enumerative induction, we may learn something statistical about the system. The Baconian approach allows us to articulate both the reasons why a system can fail and the reasons why the argument can be defective – Uneliminated defeaters serve to focus additional assurance efforts. the Baconian approach allows evaluation of a system prior to its operational use The Baconian approach avoids confirmation bias The Baconian, Pascalian, and Bayesian approaches inform each other. Make Pascalian claims about system reliability and then elucidate the possibilities that would make you doubt the validity of the claim. Take repeated samples to understand the operational reliability of the system.

13. 13 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University Scalability and Practicality of the Approach Issue: is this approach practical? The number of defeaters relevant to an argument is quite large for a real system. – But the amount of relevant argument and evidence for a real system is inherently quite large. We believe that the Baconian approach allows one to develop a more thorough and cost-effective basis for developing confidence in system behavior than current methods. We hope that this approach will lead to more effective and focused assurance efforts. This all remains to be seen. Our initial interactions with systems developers have been promising.

14. 14 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University Summary The identification of defeaters and how they are eliminated provides framework for assessing confidence The framework provides useful ways of thinking about assurance case deficiencies Identification of deficiencies is the first step towards their mitigation The end result is a stronger assurance case The approach is still under development. We welcome your ideas

15. 15 Measuring Assurance Case Confidence Weinstock/Goodenough/Klein, May 2013 © 2013 Carnegie Mellon University Contact Information Charles B. Weinstock Senior Member of the Technical Staff Software Solutions Division Telephone: +1 412-268-7719 Email: weinstock@sei.cmu.eduweinstock@sei.cmu.edu U.S. Mail Software Engineering Institute 4500 Fifth Avenue Pittsburgh, PA 15213-2612 USA John B. Goodenough SEI Fellow (retired) Software Solutions Division Telephone: +1 412-268-6391 Email: jbg@sei.cmu.edu Ari Z. Klein Ph.D. Candidate – Rhetoric Software Solutions Division Telephone: +1 412-268-7700 Email: azklein@sei.cmu.edu