1. Mathematical Formulation and Validation of the Impact of Requirements Volatility on Systems Engineering Effort
March 6, 2012
Mauricio E. Peña

2. Outline Motivation and introduction Research methods
Observations from the research
Model Development
Results
Model Calibration
Evaluation of Model Performance
Sensitivity Analysis
Cross-Validation
Conclusion

3. Importance of Understanding Requirements Volatility
Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects1
Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans2
The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure3
System developers often lack effective methods and tools to account for and manage requirements volatility
Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO

4. Requirements Volatility is Expected
Changes to requirements are a part of our increasingly complex systems & dynamic business environment
Stakeholders needs evolve rapidly
The customer may not be able to fully specify the system requirements up front
New requirements may emerge as knowledge of the system evolves
Requirements often change during the early phases of the project as a result of trades and negotiations
Requirements volatility must be anticipated and managed
Sources: Kotonya and Sommerville (1995); Reifer (2000)

5. CSSE Parametric Cost Models
The Constructive Systems Engineering Cost Model (COSYSMO) was developed by the USC Center for Software and Systems Engineering (CSSE) in collaboration with INCOSE and Industry affiliates
COSYSMO is the first generally-available parametric cost model designed to estimate Systems Engineering effort
Built on experience from COCOMO 1981, COCOMO II
During the development of COSYSMO, volatility was identified as a relevant adjustment factor to the model’s size drivers
Source: 7th Annual Practical Software and Systems Measurement Conference. COSYSMO Workshop, Boehm

6. COSYSMO Operational Concept
Volatility Factor
COSYSMO 2.0 assumes COSYSMO hypothesis holds true
Size drivers can adequately estimate the amount of effort for a nominal systems engineering project
Application of categories to size drivers adequately captures the effect of systems engineering reuse
Looking back at previous hypothesis and only addressing reuse in the size drivers of COSYSMO 2.0:
Four size drivers represent the majority of the explanatory power of the model, effect of reuse will more directly influence the estimate
Minimize potential overlap, avoid any confusion with cost drivers, i.e. Tool Support: availability of reusable tools
By retaining number and scope of drivers, backward compatibility of model, data, and estimates is maintained
Size drivers are continuous variables, cost drivers are discrete, so size drivers provides wider range of possible values
Source: Valerdi (2005) 6 6

7. Research Methods Data gathered from 25 industry projects
Literature Review & 6 Workshops completed
We are here
Source: Boehm et al (2000)

8. Organizations that Participated in the Research
The Aerospace Corporation
Northrop Grumman Corporation
The Boeing Company
Raytheon
United Launch Alliance
BAE
TI Metricas Ltda.
IBM
Distributed Management
MIT
USC
Lockheed Martin
Ericsson España
Samsung SDS
Rolls Royce
Softstar
Texas Tech
The US Army
The US Navy
The US Air force
The Australian Department of Defense

9. Requirements Volatility Observations
Requirements volatility is caused by an identifiable set of project and organizational factors
The level of requirements volatility is a function of the system life cycle phase
Requirements volatility leads to an increase in project size and cost
The cost and effort impact of a requirements change increases the later the change occurs in the system life cycle
The impact of requirements volatility may vary depending on the type of change: added, deleted, or modified

10. Model Form
Incorporated volatility effects through a scale factor (SF) added to the diseconomies of scale exponent (E)
Similar approach used to model volatility effects in Ada COCOMO1
Prior research points to the compounding or exponential effect of project factors with variable life cycle impact2
Source: 1: Boehm, B. and Royce, W. (1989); 2: Wang, G. et al., (2008)

11. Volatility Scale Factor
Expected REVL is rated as Very Low, Low, Moderate, High and Very High
Where
REVL = The % of the baseline requirements that is expected to change over the system lifecycle
wvl = aggregate lifecycle phase volatility weighting factor
And:
wl = weighting factor for each life cycle phase1
Θl = % of total requirements changes per life cycle phase
1Life Cycle Phases: Conceptualize, Development, Operational Test and Evaluation, and Transition to Operation

12. Data Collection
Collected expert judgment on REVL and weighting factors through surveys and workshops
Software and Systems Engineers with 20+ years of experience
Variety of industries represented with an emphasis on Aerospace and Defense
Historical Data Collection
Data were collected from 25 projects from the Aerospace and Defense application domain
Collected COSYSMO size, effort, and cost driver data
Collected volatility data: added, modified, and deleted requirements over time

13. Updated parameter mean and variance
Model Calibration
A Priori Model
Historical data
Mean and Variance
Data-determined Model
Regression Analysis; OLS
Initial parameter
mean and variance
Expert-judgment estimates
A Posteriori Model
Updated parameter mean and variance
Bayesian Analysis
Formally combines expert judgment
With sample data
Optimized combination
of data sources to increase precision
Source: Boehm et al. (2000); Nguyen (2010)

14. Requirements Volatility (REVL) Rating Levels
Developed based on surveys of experienced S/W and Systems Engineers (N =38)

15. Requirements Volatility Profile
Based on expert judgment collected through three workshops (N = 36) and historical project data (N = 25)

16. Regression and Bayesian Calibration
COSYSMO can be described as multiple regression model
The model is linearized using a logarithmic transformation
The data-determined coefficient β1 is combined with the a-priori expert judgment to develop the Bayesian calibrated model
A posteriori Bayesian Update
1.96
2.09
2.01
Data Analysis
A priori Expert Judgment

17. Model Performance Comparison
Model performance evaluated using the baseline model and a model with local calibration
The performance of COSYSMO improves by including the requirements volatility factor

18. Coefficient of Determination (n=25)
Academic COSYSMO
COSYSMO with Requirements Volatility Factor
Estimated Systems Engineering Size (with diseconomies of scale)
Estimated Systems Engineering Size (with diseconomies of scale)
* Due to proprietary reasons only the analysis of the model accuracy is shown, not the data itself

19. Sensitivity Analysis
Scenario 1
Evaluated the sensitivity of the model’s results to the variability of key parameters
Standard deviation in the volatility life cycle profile (Θl )
Scenarios 1 and 2
Standard deviation in the requirements volatility weighting factor (wl)
Scenarios 3 and 4
Combination of cases:
Scenario 1-3, 1-4
Scenario 2-3, 2-4
% Volatility per Life Cycle Phase
Conceptualize
Development
Operational Test & Eval
Transition to Operations
Scenario 2
% Volatility per Life Cycle Phase
Conceptualize
Development
Operational Test & Eval
Transition to Operations
Source: Guidelines for the Economic Analysis of Projects (1997)

20. Cross-Validation Results
The 25 projects were randomly divided into K=6 subsets
One of the subsets is excluded from the data set from which the model is built
The resulting model is used to predict effort for the excluded cases
Calibrated Model
Estimation Accuracy
PRED(.15)
PRED(.20)
PRED(.30)
MMRE
Academic COSYMO
52%
64%
84%
20%
Req. Volatility Model
76%
88%
15%
Cross-Validation
80%
16%
Cross-Val Scenario 1
72%
Cross-Val Scenario 2
Cross-Val Scenario 3
Cross-Val Scenario 4
68%
Cross-Val Scenario 1-3
17%
Cross-Val Scenario 1-4
Cross-Val Scenario 2-3
Cross-Val Scenario 2-4
Improvement holds across the scenarios

21. Conclusions
Observations from the literature and workshop surveys were used to develop a mathematical framework for quantifying the impact of requirements volatility on systems engineering effort
An evaluation of the model was performed by comparing its prediction accuracy against Academic COSYSMO
The results indicate an improvement in effort prediction accuracy and MMRE
Cross validation and sensitivity analysis were performed to demonstrate the accuracy of the model in predicting effort for new projects

22. Future Work Obtain additional project data from other organizations
The external validity of this study will be limited by the engineering organizations that contribute data and the background of the industry experts that participate in the research
Evaluate the effect of reuse on the results and its potential interaction with requirements volatility
Further work is needed to complete the characterization of the model performance depending on the type of change: added, modified or deleted

23. References
B. Boehm and W. Royce. Ada COCOMO and the Ada Process Model, TRW Defense Systems Group, 1989
B. Boehm, Software risk management: Principles and practices, IEEE Software 1 (1991),
B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E., Horowitz, R. Madachy, D.J. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, New York, NY, 2000
Economics and Development Resource Center Guidelines for the Economic Analysis of Projects.
S. Conte, H. Dunsmore, and V. Shen. Software Engineering Metrics and Models. Benjamin/Cummings Publishing Company, 1986.
General Accounting Office, "Stronger management practices are needed to improve DoD’s software-intensive weapon acquisitions (GAO )," 2004.
ISO/IEC, "ISO/IEC 15288:2002 (e) systems engineering - system life cycle processes," 2002.
G. Kotonya and I. Sommerville, Requirements engineering: Processes and techniques, John Wiley & Sons, New York, NY, 1998.
MIL-STD-498, "Software development and documentation," 1994.
D. Reifer, Requirements management: The search for nirvana, IEEE Software 17(3) (2000), 45-47
D. Rhodes, R. Valerdi and G. Roedler, Systems engineering leading indicators for assessing program and technical effectiveness, Systems Engineering 12(1) (2009),
Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department.
Wang, G., Boehm, B., Valerdi, R., and Shernoff, A. (2008). “Proposed Modification to COSYSMO Estimating Relationship.” Technical Report. University of Southern California, Center for Systems and Software Engineering.

24. Back-up

25. COSYSMO Cost Estimating Relationships (CERs)
SE_Hrs = Systems Engineering effort (hours)
A = calibration constant derived from historical project data
SIZE = measure of functional size of the system (number of requirements, interfaces, algorithms, operational scenarios)
n = number of cost drivers (14)
EM = effort multiplier for the ith cost driver
The exponent (E) accounts for diseconomies of scale
Source: Valerdi (2005)

26. Expert Judgment Life Cycle Weighting Factors (n = 27)
Systems Engineering Effort Penalty Due to Volatility
Operational Test & Evaluation
Conceptualize
Development
Transition to Operation
Data collected from two workshops: 25th Annual USC CSSE COCOMO and the 2011 USC-CSSE ARR

27. Evaluation of Model Performance
The cost estimation accuracy of COSYSMO was compared to the accuracy of the model with the requirements volatility factor
The model was evaluated using predictive accuracy levels, Mean Magnitude of Relative Error (MMRE) and the coefficient of determination (R2)
The prediction accuracy level is defined as:
Where
k = number of projects in the set whose Magnitude of Relative Error is ≤ l
Source: Conte et al., (1986)

28. Model Comparison Test (F-ratio)
The F-test is typically used to compare regression models and determine whether the null hypothesis is supported or refuted
In this case, the null hypothesis is that the simpler model (without a requirements volatility factor) is correct
Values of the F ratio near one (1) support the null hypothesis, while larger values favor the alternative hypothesis
The F-value was calculated to be 7.62 with > 95% confidence level: Supports the alternative hypothesis