1. Nancy S. Eickelmann, PhD Motorola Labs 1303 E. Algonquin Rd. Annex-2 Schaumburg, IL 60196 Phone: (847) 310-0785 Fax: (847) 576-3280 Nancy.Eickelmann@motorola.com Nancy.Eickelmann@motorola.com

2. 2 FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP) for the NASA Independent Verification and Validation Facility COTR: Ken McGill PI: Nancy Eickelmann S-54493-G September 5, 2001 Developing Risk-Based Financial Analysis Tools and Techniques to Aid IV&V Decision-Making

3. 3 PROBLEM STATEMENT This research addresses NASA’s need to evaluate the ROI and cost/benefit of applying IV&V technologies. A prototype is to be developed that will provide financial valuation of IV&V for a given program. The prototype will be developed using an iterative process that will incrementally implement the models and methodology researched and developed during prior years of this effort. The tool will be evaluated for usability, accuracy, and consistency through limited use scenarios with NASA program managers.

4. 4 Return on Investment - Status This project was funded July 20, 2001 Evaluation of data sets is in progress Benchmarking for key factor target value ranges in progress Model integration and interface to existing programs in progress, Ask Pete, ARRT

5. 5 RESEARCH APPROACH Phase 1: Reduce the models we developed earlier to actionable guidelines for practice Phase 2: Introduce these models, processes and support tools to a small group of carefully selected pilot projects Evaluate the results of applying the tools and methods Phase 3: Use the feedback from step 3 to adapt the tools and methods for widespread dissemination, if warranted within the software project decision-making community at NASA.

6. 6 HYPOTHESES/OBJECTIVE The IV&V valuation methodology will be iteratively refined based on feedback from NASA program managers and statistical evaluation of the methodology and results. Specific factors to be evaluated: Hypothesis 1: The cost relative to the potential benefits of IV&V is inversely proportional to key organizational factors, such as the capability maturity of the development organization. Hypothesis 2: The realization of potential IV&V benefits is directly related to the development organizations’ acceptance of IV&V. Hypothesis 3: The cost/benefit ratio for IV&V is directly related to the criticality of the application (and its individual subsystems).

7. 7 IV&V YIELD Ultimately, the yield of an IV&V program is based upon the difference between the net resource flow with IV&V and without IV&V. If the resources saved (e.g., reduced rework) or returns gained (e.g., improved customer satisfaction or increased safety) are greater than the resources consumed to save/gain these resources, we have a net benefit. Should the resources saved be less than the resources consumed, we have a net cost.

8. 8 IV&V Yield Cost of Quality –Key components… Cost of Poor Quality –Key components…

9. 9 What we already know… 3 issues of empirical studies... June 5-6, 1986 the 1st Workshop on Empirical Studies of Programmers, Washington, D.C. Need scientific rigor…“A Plan for Empirical Studies” Victor Basili Need to look at real world variable values…“By the Way, Did Anyone Study Real Programmers” Bill Curtis Need to study PITL…“Meeting the Challenge of Programming in the Large (PITL)” Elliot Soloway

10. 10 Why is it Difficult to Apply Quantitative Management Principles for Software Engineering? SE domain has a large number of key variables that have different degrees of significance depending on the environment SE domain has key variables that have extreme variance within the same environment (i.e., programmer productivity 10:1) SE domain variables in combination may create a “critical mass” not present when variables are studied in isolation 1986 IEEE TSE, Basili, Selby and Hutchins, Surveyed software engineering empirical studies published to date. Cited 116 published studies.

11. 11 Software Requirements Analysis Software Interface Analysis Software Code Analysis Developer Test Analysis Software Design Analysis Iterative IV&V Methodology IV&V Planning - Activities- Organization- CARA - Schedules- Tools- WBS Inputs Activities Outputs Software IV&V Plan Critical/High Risk Functions List IV&V Technical Reports Software Problem Reports IV&V Traceability Matrix Findings and Recommendations IV&V Metrics Monthly Progress/Status Reports Iterative Per Software Release Source Code Software Development Folders Software Test Plans & Procedures Problem Reports Requirements Repositories Program Milestones and Schedules Phase Dependent IV&V Tasks Phase Independent IV&V Tasks Developer Documentation Software IV&V SOW - Objectives - Requirements TRACEABILITY ANALYSIS CHANGE IMPACT ANALYSIS DELIVERABLES VALIDATION TECHNICAL REVIEWS AND AUDITS SPECIAL STUDIES

12. 12 IV&V Technologies - COQ

13. 13 Empirical Research Summary Experimental Simulation Qualitative and quantitative results based on non- deterministic or hybrid simulation model Math Modeling quantitative results based on a deterministic model Mirrors a segment of the real world, control of variables is high, supports testing of causal hypothesis, results can be replicated, high internal validity and generalizability Captures real world context in which to isolate and control variables Researcher bias can be introduced through selection of variables, parameters and assumptions concerning the model. Modeling requires high degree of analytical skill, and interdisciplinary knowledge Results are not typically generalizable to other populations or environmental contexts, researcher bias is common,

14. 14 Process Modeling and Simulation Managed, measured, productivity gains through: process improvement data driven decision-making technological innovation Quantitative valuation of COQ vs COPQ

15. 15 COQ versus COPQ

16. 16 Process Simulation Models Experimental Simulation Qualitative and quantitative results based on non-deterministic or hybrid simulation model mirrors a segment of the real world control of variables is high supports testing of causal hypothesis results can be replicated high internal validity high external validity, generalizability

17. 17 IV&V Yield Organizational context factors for cost –Key components

18. 18 Independent Verification and Validation An organization independent from the developers study the artifacts of software production. This requires: -Technical independence. Members of the IV&V team may not be personnel involved in the development of the software. -.Managerial independence. The responsibility for IV&V belongs to an organization outside the contractor and program organizations that develop the software. -Financial independence. Control of the IV&V budget is retained in an organization outside the contractor and program organization that develop the software. IV&V is often perceived as testing the code after the development is completed NASA IV&V is full life cycle activities

19. 19 State of the Practice: Process Maturity Source: SEI Web Site SEMA Report for March 2000

20. 20 Measuring IV&V Effectiveness

21. 21 US Data? Average Best in Class Current Level % Improve- ment Productivity (KAELOC per staff month) 3.237.144.03 Cost (dev. cost per KAELOC) $4,334$1,962$1,008 Defect Content (defects per KAELOC) 15.68.118.8 Defect Removal Efficiency (by Technology) 95%99.5%92.7% Cost of Delivered Defects (post-release defects per KAELOC, Domain) $1.5M Industry Benchmarking Source for US Data: Capers Jones (2000) Software Assessments, Benchmarks, and Best Practice, Addison-Wesley, p 339, System Software Baseline.

22. 22 IV&V Yield System factors for cost and gain

23. 23 Prior Empirical ROI Studies ROI: Independent V&V Benefits IV&V applied early in the lifecycle has the greatest ROI. Source Jet Propulsion Laboratory TR.

24. 24 02/09/01 IMPACT of Major Air & Space Software Problems Aggregate Cost: Loss of Life: ‘93‘96‘97‘98‘99 Airbus A320 Ariane 5 3 Flight 965 160 $640 million Loss of Data: [Poseidon] [Galileo] USAF STEP [Pathfinder] [Lewis] $116.8 million Zenit 2 Delta 3 [NEAR] $255 million Titan 4B [DS-1] Orion 3 [Galileo] $1.6 billion [‘99] – NASA IV&V presentation

25. 25 Tracing Impacts to Causes… Cause-Effect Graphing Mission Success at Reduced Cost Reliability Objective Safety Objective Cost Objective Process Improvement Avoid Rework Eliminate Redundancy Efficient Resource Allocation Skilled Workforce Domain Experts Engineers V&V Experts Skills training program IT Infrastructure, Web-based reporting, DSS, ARM, PITS, RMS, Ask Pete, ARRT Communication Channels & Reporting PL Reuse Technologies Domain Engineering Knowledge Maintenance V&V Models and Methods Quality Objective Defect Prevention Defect Detection Identify and Eliminate Hazards Identify and Manage Risks Information Analysis & Information Management, Product Certification

26. 26 Strategic and Financial Goals Reliability Objective Competitive Objective Cost Objective Process Improvement Avoid Rework Eliminate Redundancy Efficient Resource Allocation Skilled Workforce Black Belts Engineers Telecom Experts Skills training program - Motorola University IT Infrastructure, Web-based reporting, COMPASS, TIGERS, TeamPlay, Communication Channels & Reporting SIX SIGMA Performance Excellence Knowledge Maintenance Communications Models and Methods Quality Objective Defect Prevention Defect Detection Optimize resource allocation & utilization Identify and Manage Risks Information Analysis & Information Management Product Certification BSC Cause and Effect Graphing

27. 27 Filter Attributes

28. 28 DTE – Rule Based

29. 29 NEURAL NETWORK

30. 30 Intelligent update of rule structure

31. 31 STATISTICAL ANALYSIS

32. 32 BENEFITS The benefits of this proposed Center Initiative would be applicable to all NASA software development organizations for whom IV&V is an option. The formalization of an objective decision-making process, along with enabling support tools would provide key capabilities to make rational budgetary decisions that impact safety and mission critical aspects of all NASA software systems. This is significant in enabling NASA to engage in effective administrative and managerial control based on objective, quantified information. The techniques proposed under this initiative will also provide NASA participants increased visibility into their process improvement efforts. The ISO-9001 certification requires that managers be able to document the benefits contributed to the organization by specific processes and process improvement effort [8]. A formalized, well-defined decision-making process would therefore make a significant contribution to NASA’s overall quality strategy.

33. 33 MILESTONES Start=July 20, 2001 + 3 mo IV&V Process Description – Product Characterization Based on prior CSIP results Start=July 20, 2001 + 6 mo Information Analysis Data gathering for methodology Start=July 20, 2001 + 6 mo Initial Prototype Demonstration(s) & Iteration(s) Delivered GSFC IV&V interface required

34. Nancy S. Eickelmann, PhD Motorola Labs 1303 E. Algonquin Rd. Annex-2 Schaumburg, IL 60196 Phone: (847) 310-0785 Fax: (847) 576-3280 Nancy.Eickelmann@motorola.com Nancy.Eickelmann@motorola.com