1. 1 Department of Electrical and Computer Engineering University of Virginia Software Quality & Safety Assessment Using Bayesian Belief Networks Joanne Bechta Dugan Susan Donohue, Ganesh Pai University of Virginia

2. 2 Department of Electrical and Computer Engineering University of Virginia Problems Under Consideration GETR: How does one decide that a software system is “good enough to release”? SWQ-BBN: Can I combine process assessment and product assessment metrics to predict quality/reliability of a software system?

3. 3 Department of Electrical and Computer Engineering University of Virginia Approach: Bayesian Belief Networks (BBN) We use BBN models as the basis of both projects BBN models effectively allow the combination of quantitative and qualitative assessment (that is, measures and expert judgment) in the same model

4. 4 Department of Electrical and Computer Engineering University of Virginia GETR Approach (with S. Donohue) For the GETR (Good Enough to Release) project, we are developing a BBN model of the decision process –What evidence is used, and how is it weighed –Determining conditional probabilities from expert opinion (to get probability parameters for the model) GETR is building a mathematical framework based on BBN to understand and facilitate the decision making process

5. 5 Department of Electrical and Computer Engineering University of Virginia GETR Decision How can we investigate and document the decision process that is used to go from... to… I have an acceptable level of belief that the system will operate as specified. Quality Assurance Test ResultsPersonal and Team CMM Prototype Performance Requirements Review Is the system good enough to release? Code InspectionRisk Assessment Formal Methods for a computer-based system Engineering Judgment

6. 6 Department of Electrical and Computer Engineering University of Virginia

7. 7 Department of Electrical and Computer Engineering University of Virginia Quantifying Judgment for BBN

8. 8 Department of Electrical and Computer Engineering University of Virginia Quantifying Judgment for BBN (QJ BBN) Conditional probabilities (NPT entries) are generated as a function of the contribution of evidence to support a premise. For example, Acceptable results from testing supports the conclusion that verification is acceptable. Unacceptable documentation supports the premise that the artifact quality is unacceptable. Evidence can overlap, be disjoint or synergistic. Proofs of coherence of functions used in QJ methodology help assure rational decisions. Importance and sensitivity analysis can help guide decision makers in seeking new evidence. BN model provides a record of evidence analysis.

9. 9 Department of Electrical and Computer Engineering University of Virginia Application to NASA: Seal of Approval Process (SOAP) for PRA tools What evidence is available for review? Is the tool “fit for use”? Is it “good enough” to share? Is the tool appropriate for use in a given domain? How much influence does certain evidence have on the approval process?

10. 10 Department of Electrical and Computer Engineering University of Virginia

11. 11 Department of Electrical and Computer Engineering University of Virginia SWQ BBN Approach (with G. Pai) For the SWQ BBN project, we are developing techniques to build a BN to model the software development process and the products (artifacts) –BBN model represents causally related phases and activities within the phases. –Measurements or expert opinion can be used to determine probability parameters for the model. Model can be used to assess the process/product with respect to reliability (defect density) or other quality attribute

12. 12 Department of Electrical and Computer Engineering University of Virginia

13. 13 Department of Electrical and Computer Engineering University of Virginia Candidate BBN for design phase

14. 14 Department of Electrical and Computer Engineering University of Virginia Hypothetical illustrative example Hypothetical priors Model result –Medium defect content –Actual values dependent on the mapping between node states and range values E.g.  Vlow, Low, Medium, High, Vhigh    0-20, 20-40, 40-60, 60-80, 80-100  –Model results  Defect content would lie in 40 – 60 range Posterior Evidence

15. 15 Department of Electrical and Computer Engineering University of Virginia Feedback to the designer  greater value –Network itself can provide feedback –Propagation of evidence In this case: knowledge of high specification quality, observation of high defect content –Change in distribution indicates potential problem area

16. 16 Department of Electrical and Computer Engineering University of Virginia Application to IV&V (joint work with Titan (Khalid Lateef)) Use IVV process for use case analysis, construct BBN from process model Relevant process parameters and inputs represent parent nodes Child nodes of BBN represent features desired from the requirements specification

17. 17 Department of Electrical and Computer Engineering University of Virginia

18. 18 Department of Electrical and Computer Engineering University of Virginia Example analysis Probabilities reflect either measurement or analysts’ beliefs The state ‘true’ is less than 95%  Not mature enough.

19. 19 Department of Electrical and Computer Engineering University of Virginia Technology Readiness Level

20. 20 Department of Electrical and Computer Engineering University of Virginia Data / Case Study Availability GETR case study domain: “lightweight” V&V for in- house developed analytical tools being considered for release to other centers or research groups. Identified case studies – RAP (JPL), SIAT (IV&V), and MATT (IV&V) SWQ BBN case study domain: Case study of system development, including artifacts & defect data. OO or ODC would be great Working with Khalid Lateef to develop case study for OO requirements analysis

21. 21 Department of Electrical and Computer Engineering University of Virginia Barriers to Research or Application Case studies