1. 1 FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP) for the NASA Independent Verification and Validation Facility COTR: Kenneth McGill PI: Nancy Eickelmann Contract #S-54493-G September 4, 2002 Developing Risk-Based Financial Analysis Tools and Techniques to Aid IV&V Decision-Making

2. 2 Agenda Why we need ASK IVEY Consequences and Likelihood of Failure IV&V Yield Probability of IV&V Yield: Min, Max, Most Likely ROI and Magnitude of Return of IV&V What ASK IVEY can do

3. 3 Why we need ASK IVEY NASA program managers are asked to quantify the ROI and evaluate the cost/benefit of applying IV&V technologies. This is a prediction of future events based on decisions and actions taken in the present. A point estimate is likely to be inaccurate, whereas a probability of yield has a history of providing a scope of potential yield and an extent of likelihood of expected yield.

4. 4 Calculating ROI a Financial Analysis Prompt Map Yes 1 A Financial Analysis Process Map Create development cost framework: Total Cost COQ COPQ ? IV&V and IA analysis complete. Level of IV&V or IA designated Create certification cost framework 23 Apply Financial models for monetary quanitification Create probability of yield structure 4 A STOP No

5. 5 Consequences of Failure NPG 2820 IV&V Criteria

6. 6 Likelihood of Failure

7. 7 IV&V YIELD Ultimately, the yield of an IV&V program is based upon the difference between the net resource flow with IV&V and without IV&V. If the resources saved (e.g., reduced rework) or returns gained (e.g., improved customer satisfaction or increased safety) are greater than the resources consumed to save/gain these resources, we have a net benefit. Should the resources saved be less than the resources consumed, we have a net cost.

8. 8 Cost of Poor Quality Defect Leakage –If discovered internally defect management rework retesting –If discovered externally technical support complaint investigation defect notification

9. 9 Stephen Knox “Modeling the Cost of Software Quality,” Digital Technical Journal, (Fall 1993)

10. 10 Raytheon Cost of Poor Quality Haley and Dion

11. 11 How Process Maturity Levels Affect IV&V LEVEL 1 INITIAL UNPREDICTABLE & POORLY CONTROLLED LEVEL 2 REPEATABLE CAN REPEAT PREVIOUSLY MASTERED TASKS LEVEL 3 DEFINED PROCESS CHARACTERIZED, FAIRLY WELL UNDERSTOOD LEVEL 4 MANAGED PROCESS MEASURED AND CONTROLLED LEVEL 5 OPTIMIZED FOCUS ON PROCESS IMPROVEMENT KEY PROBLEMS: CONFIG. MGMT. PROJECT MGMT. SOFTWARE QA PROJECT PLANNING EASTIMATING COST SCHEDULE KEY PROBLEMS: PROPER TRAINING DEVELOPMENT OF PRACTICE & PROCEDURES STANDARDS ORGANIZATION KEY PROBLEMS: ACCURATE PROJECT MEASUREMENT OBJECTIVE PROCESS ANALYSIS QUANTITATIVE QUALITY PLANS PRIORITIES: - DECIDING WHAT TO IMPROVE FIRST - BALANCING THE PROCESS AND THE ORGANIZATION KEY PROBLEMS: CHANGING TECHNOLOGY PROBLEM ANALYSIS PROBLEM PREVENTION ORGANIZATIONAL OPTIMIZATION KEY PROBLEMS: STILL HUMAN INTENSIVE PROCESS DIFFICULT TO MAINTAIN OPTIMUM ORGANIZATION DIFFICULT TO MAINTAIN TOOLS & PRACTICES AT STATE OF THE ART IV&V SOMEWHAT UNPREDICTABLE UNABLE TO ESTIMATE NON-TECH % IV&V NON-TECH UP TO 25% NON-TECH UP TO 15% NON-TECH UP TO 6-8% NON-TECH UP TO 3-4% MATURITY

12. 12 Cost of Leakage Grows Over Time Relative cost of fixing a problem found in design/coding, testing, or after release are: –1:20:82 (Remus, 1983) –1:13:92 (Kan, 1989) –10:100:1000 (Coyle, 1999)

13. 13 Cost of Rework in Each Phase Rework product design = leakage requirements * cost-to-fix nominal * 10 Rework programming = leakage requirements * cost-to-fix nominal * 100 +leakage design * cost-to-fix nominal * 10 Rework integration = leakage requirements * cost-to-fix nominal * 1000 +leakage design * cost-to-fix nominal * 100 +leakage programming * cost-to-fix nominal * 10 Rework deployment = leakage requirements * cost-to-fix nominal * 10000 +leakage design * cost-to-fix nominal * 1000 +leakage programming * cost-to-fix nominal * 100

14. 14 Rework at Deployment Tremendous cost rework plus –product recall –technical support –field visits –cost factor may be over 10,000

15. 15 Leakage: An Example Rework product design = 49 r * cost-to-fix nominal * 10 = 490 Rework programming = 39 r * cost-to-fix nominal * 100 = 3,900 +113 d * cost-to-fix nominal * 10 = 1,130 Rework integration = 26 r * cost-to-fix nominal * 1000 = 26,000 +49 d * cost-to-fix nominal * 100 = 4,900 +418 p * cost-to-fix nominal * 10 = 4,180 Rework deployment = 8 r * cost-to-fix nominal * 10000 = 80,000 +16 d * cost-to-fix nominal * 1000 = 16,000 +56 p * cost-to-fix nominal * 100 = 5,600142,200

16. 16 CMM Maturity and Leakage There is some evidence to suggest organizations with increased maturity have reduced rework costs Knox: Percent of Budget to Rework: –Level 1: 55% –Level 2: 45% –Level 3: 35% –Level 4: 20% –Level 5: 6%

17. 17 IV&V and Defect Leakage Application of IV&V can reduce leakage to subsequent phases The goal of the financial model is to propose a range of potential savings Specific parameters will need to be established empirically

18. 18 Timing of benefits for IV&V Full In-Phase IV&V –prevention of errors starting at requirements - can potentially bar any errors from leaking through Partial IV&V –prevention of errors at point of insertion - no errors from this phase will leak Endgame IV&V –discovery of errors at the end of development - can potentially bar any errors from leaking to deployment Audit Level IV&V

19. 19 Rework and Return from IV&V By Maturity Level

20. 20 Components to Return on Investment Cost of IV&V Expected Return –cost savings - measured as hours of rework Likelihood of Returns –how effective is the organization at minimizing rework? –how effective will IV&V be?

21. 21 Independence… An organization independent from the developers study the artifacts of software production [IEEE Std. 1012-1998]. This requires: -Technical independence. Members of the IV&V team may not be personnel involved in the development of the software. -.Managerial independence. The responsibility for IV&V belongs to an organization outside the contractor and program organizations that develop the software. -Financial independence. Control of the IV&V budget is retained in an organization outside the contractor and program organization that develop the software. IV&V is often perceived as testing the code after the development is completed …..NASA IV&V is full life cycle activities

22. 22 IV&V is NOT SQA IV&V is a full life cycle set of acivities that are applied to defect prevention, defect detection, and certification. NASA IV&V conforms to IEEE Standard 1012-1998. IV&V and Software Quality Assurance (SQA) are not redundant activities. SQA as defined by DOD-Std 2168 defines 10 activities of SQA that are complemented by IV&V activities. There are 32 types of activities conducted by IV&V, of these 32, 22 are unique to IV&V and 10 are complemented by SQA.

23. 23 Ask Ivey Prototype What ASK IVEY can do…

24. 24 Ask Ivey Input Screen

25. 25 Ask Ivey Pull Down Menu

26. 26 Ask Ivey Numeric Entry

27. 27 Ask Ivey On-Line Report

28. 28 Ask Ivey Printed Report

29. 29 Ask Ivey On-Line Help

30. 30 Questions? Ask Ivey…