1 Fault-Based Analysis: Improving IV&V Through Requirements Risk Reduction '02 Jane Hayes Rama Bireddy D.N. American SAIC Department of Computer Science University of Kentucky

2 Outline  Research Objective  Research Approach  Progress to Date  Current Plans  Future Work

3 Research Objective To improve how we focus resources for IV&V of Critical/Catastrophic High-Risk (CCHR) software functions, we use a fault-based analysis method comprised of: a requirements fault taxonomy (Phase I) a method for extending taxonomies (Phase I) a taxonomy of IV&V techniques (Phase I and II) what faults the techniques can detect (Phase II+) a fault-based risk assessment per Class and project (Phase I+) a cost-benefit analysis of technique effectiveness (validated) (Phase II+)

4 Research Approach (Phase I) Task 1 – Select a Known Fault Taxonomy Task 2, 3 – PMR 2,3 (Presentation and Milestone Meeting) Task 4 – Examine NASA-specific requirements faults Task 5 – Build a list of IV&V techniques Task 6 – Adopt or build a method for extending the taxonomy Task 7 –Implement the method to extend the fault taxonomy Task 8,9 - Year-end report and presentation (also PMR 4)

5 Progress to Date NUREG/CR-6316 basis of general taxonomy Literature survey (55 references) resulted in three additions to taxonomy Review of defect data for 3 NASA projects resulted in two additions and re-organization of taxonomy

6 General Taxonomy Taxonomy figure

7 NASA Requirement Faults Have proven difficult to obtain Level of detail varies greatly Have thus far received and examined –IV&V “comments” on requirement problems for 3 projects –Project fault reports related to requirements for 1 project Data very useful, resulted in several changes to fault taxonomy and taxonomy extension/tailoring processes

8 Task 6 Process for extending fault taxonomy split into two parts: Process A and Process B Process A - activities to develop a Class- specific taxonomy Outputs of Process A are inputs to Process B Process B – activities to develop a project- specific taxonomy

9 Processes for Extending Fault Taxonomies High level process

10 Class-Specific Taxonomy Process Process

11 Project-Specific Taxonomy Process Process

12 Current Plans Obtain requirement-related fault reports for additional NASA projects Perform Process A (Class-specific) for classes for which we have data Complete list of IV&V techniques Continue sharing information and soliciting feedback from ST-5 project Prepare final report

13 Future Work (Phase II) Build taxonomy of IV&V techniques Survey literature and determine what techniques have been shown to detect certain types of requirement faults Gather expert opinion to fill in gaps of the technique-to-fault matrix Design experiments to validate some of the technique-to-fault mappings Provide resulting information in an Advanced Risk Reduction Tool (ARRT) friendly format

14 Backup

15 How does this differ from ODC? ODC uses a fixed set of trigger and defect types Our emphasis is on building tailored taxonomies We use historical information about Classes of related projects ODC classification strives to be independent of the specifics of a product or organization

16 How does this differ from ODC? (cont’d) ODC defect types don’t map well to requirements (function, I/f, checking, assignment, algorithm, etc.) We integrate risk analysis in our taxonomy building process Our long-term goals include validated cost-benefit information for fault- technique pairs