Download presentation
Presentation is loading. Please wait.
Published byGriffin Weaver Modified over 9 years ago
1
Civil Government Services Group 1 Return on Investment of Independent Verification and Validation: Indirect Benefits James B. Dabney, Gary Barber, Don Ohi, Kurt Woodham Software Assurance Symposium 29 July 2003
2
Civil Government Services Group 2 Overview Approach Overall Results Top Four Benefits Conclusions
3
Civil Government Services Group 3 Study Approach Benefits of Value Quantifiable Characteristics Quantification Methods
4
Civil Government Services Group 4 Benefits of Value Identify as many benefits as possible Determine whether each benefit can be substantiated by tangible data Assess quantifiability of each benefit Identify quantification Published data IV&V and project artifacts
5
Civil Government Services Group 5 Quantifiable Characteristics Score benefits Develop method to convert quantifiable benefits into consistent measure (EPM, dollars)
6
Civil Government Services Group 6 Quantification Methods Identify quantification procedure for each benefit Unique method for each benefit Refine and test
7
Civil Government Services Group 7 Results – Candidate Benefit List Identified 84 candidate benefits Refined list to 26 quantifiable benefits Scored (High=3, Medium = 2, Low = 1) Universality (weight = 1) Practicality (weight = 3) Potential value (weight = 4) Credibility (weight = 5)
8
Civil Government Services Group 8 Top Four Indirect Benefits Improved testing Reduced high-criticality errors Requirements clarification Reduced leakage to operations
9
Civil Government Services Group 9 Improved Testing Direct benefit is correction of test case deficiency Indirect benefit is error identified as consequence of corrected test case Computing return Identify IV&V issues resulting in test case change Correlate issues with defects found in testing Incorporate results into direct ROI computation Apply high-criticality and phase leakage methodology where appropriate
10
Civil Government Services Group 10 Improved Testing Example Mission-critical ground software system IV&V identified 61 test deficiencies One added test case identified flaw in redundancy management state machine Flaw would have prevented detection of serious system fault Direct ROI return for this error is small, indirect large using high-criticality methodology
11
Civil Government Services Group 11 Reduced High-Criticality Errors CARA focuses IV&V on high-criticality portions of system Removing high-criticality defects has benefits beyond cost-to-fix (direct ROI) Value of mission Value of vehicle Crew and control center time during mission Compute using expected value based on estimated probabilities of occurrence
12
Civil Government Services Group 12 Reduced High-Criticality Errors Example Incorrect handling of thermal control system failure Disable primary GN&C computer Fault not recognized – backup GN&C stays off line Vehicle placed in uncontrolled state Three undesirable operational outcomes Loss of vehicle at cost of $40B, 0.00001 probability Loss of mission at cost of $4B, 0.00001 probability Failure to annunciate fault, $4M, 0.01 probability Expected value = $480,000
13
Civil Government Services Group 13 Requirements Clarification Requirements ambiguities Wording open to misinterpretation Conflicts between different representations of same requirement Benefits of removing Reduced development cost resulting from reduced probability of implementation errors Reduced maintenance cost due to increase in understandability
14
Civil Government Services Group 14 Reduced Development Cost Computation Increment to effective SLOC for requirements ambiguity found in requirements phase is where k w is function point weight, k L function point to SLOC factor, p i probability of error of type i resulting from requirements ambiguity, k s is SLOC reduction factor, d i is ambiguous requirement removal rate phase i
15
Civil Government Services Group 15 Reduced Maintenance Cost Computation COCOMO-II reuse model includes term SU for understandability Estimate increment to SU as FP A is function points associated with ambiguous requirements FP T total function points Increment to ESLOC (effective SLOC due to reuse)
16
Civil Government Services Group 16 Reduced Error Leakage to Operations Direct ROI model assumes no defect leaks to operations Experience indicates a significant number of errors will manifest in integration testing and operations Expand direct ROI leakage model to include integration and operations Requires adjustment to direct ROI coefficients
17
Civil Government Services Group 17 Conclusions Identified and rank-ordered 26 unique indirect benefits of IV&V Analyzed top four benefits in detail Improved testing Reduced high-criticality errors Requirements clarification Reduced leakage to operations Direct and indirect benefits form comprehensive ROI model and basis of predictive ROI model
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.