October 18, 2012 Mauricio E. Peña Ricardo Valerdi

Slides:



Advertisements
Similar presentations
Full Day Workshop: Thursday, November 3, 2005, 8:00am – 5:00pm SAIC, Science Drive, Orlando, Florida Registration/Sign-In Begins at 7:30am Building.
Advertisements

Printed by CONIPMO (Constructive Infrastructure Protection Model): ACCURATELY ESTIMATING HOW MUCH IT WILL COST TO SET UP NETWORK.
On Representing Uncertainty In Some COCOMO Model Family Parameters October 27, 2004 John Gaffney Fellow, Software & Systems.
A Sizing Framework for DoD Software Cost Analysis Raymond Madachy, NPS Barry Boehm, Brad Clark and Don Reifer, USC Wilson Rosa, AFCAA
May 18, 2004CS WPI1 CS 562 Advanced SW Engineering Lecture #6 Tuesday, May 18, 2004.
University of Southern California Center for Systems and Software Engineering 2012 COCOMO Forum 1 October 18, 2012 Mauricio E. Peña Ricardo Valerdi Quantifying.
COCOMO Suite Model Unification Tool Ray Madachy 23rd International Forum on COCOMO and Systems/Software Cost Modeling October 27, 2008.
COSYSMO 2.0 Workshop Summary (held Monday, March 17 th 2008) USC CSSE Annual Research Review March 18, 2008 Jared Fortune.
University of Southern California Center for Systems and Software Engineering ©USC-CSSE1 Ray Madachy, Ricardo Valerdi USC Center for Systems and Software.
USC 21 st International Forum on Systems, Software, and COCOMO Cost Modeling Nov 2006 University of Southern California Center for Software Engineering.
March 2002 COSYSMO: COnstructive SYStems Engineering Cost MOdel Ricardo Valerdi USC Annual Research Review March 11, 2002.
A study of the Causes of Requirements Volatility and its Impact on Systems Engineering Effort COSYSMO Workshop Center for Software and Systems Engineering,
University of Southern California Center for Software Engineering CSE USC COSYSMO: Constructive Systems Engineering Cost Model Barry Boehm, USC CSE Annual.
11/08/06Copyright 2006, RCI1 CONIPMO Workshop Out-brief 21 st International Forum on COCOMO and Software Cost Modeling Donald J. Reifer Reifer Consultants,
Integration of Software Cost Estimates Across COCOMO, SEER- SEM, and PRICE-S models Tom Harwick, Engineering Specialist Northrop Grumman Corporation Integrated.
1 COSYSMO 3.0: Future Research Directions Jared Fortune University of Southern California 2009 COCOMO Forum Massachusetts Institute of Technology.
University of Southern California Center for Software Engineering CSE USC ©USC-CSE 10/23/01 1 COSYSMO Portion The COCOMO II Suite of Software Cost Estimation.
COSYSMO Reuse Extension 22 nd International Forum on COCOMO and Systems/Software Cost Modeling November 2, 2007 Ricardo ValerdiGan Wang Garry RoedlerJohn.
University of Southern California Center for Systems and Software Engineering © 2010, USC-CSSE 1 COCOMO II Maintenance Model Upgrade Vu Nguyen, Barry Boehm.
1 Systems Engineering Reuse Principles Jared Fortune, USC Ricardo Valerdi, MIT COSYSMO COCOMO Forum 2010 Los Angeles, CA.
1 Results of Reuse Survey Jared Fortune, USC Ricardo Valerdi, MIT Gan Wang, BAE COSYSMO COCOMO Forum 2008 Los Angeles, CA.
University of Southern California Center for Systems and Software Engineering ©USC-CSSE1 Ray Madachy, Barry Boehm USC Center for Systems and Software Engineering.
University of Southern California Center for Systems and Software Engineering 1 November 2010 Mauricio Peña Dr. Ricardo Valerdi COSYSMO Requirements Volatility.
University of Southern California Center for Systems and Software Engineering 1 © USC-CSSE A Constrained Regression Technique for COCOMO Calibration Presented.
Introduction Wilson Rosa, AFCAA CSSE Annual Research Review March 8, 2010.
University of Southern California Center for Systems and Software Engineering © 2009, USC-CSSE 1 Assessing and Estimating Corrective, Enhancive, and Reductive.
University of Southern California Center for Software Engineering CSE USC USC-CSE Annual Research Review COQUALMO Update John D. Powell March 11, 2002.
COSYSMO Reuse Extension 22 nd International Forum on COCOMO and Systems/Software Cost Modeling November 2, 2007 Ricardo ValerdiGan Wang Garry RoedlerJohn.
University of Southern California Center for Systems and Software Engineering 1 Mauricio E. Peña Dr. Ricardo Valerdi March 8, 2011 Characterizing the Impact.
USC 21 st International Forum on Systems, Software, and COCOMO Cost Modeling Nov 2006 University of Southern California Center for Software Engineering.
System-of-Systems Cost Modeling: COSOSIMO July 2005 Workshop Results Jo Ann Lane University of Southern California Center for Software Engineering.
University of Southern California Center for Systems and Software Engineering 1 November 2010 Mauricio Peña Dr. Ricardo Valerdi CHARACTERIZING THE IMPACT.
1 Discussion on Reuse Framework Jared Fortune, USC Ricardo Valerdi, MIT COSYSMO COCOMO Forum 2008 Los Angeles, CA.
Expert COSYSMO Update Raymond Madachy USC-CSSE Annual Research Review March 17, 2009.
COCOMO II Database Brad Clark Center for Software Engineering Annual Research Review March 11, 2002.
Systems Engineering Reuse: A Report on the State of the Practice Jared Fortune, USC Ricardo Valerdi, MIT Gan Wang, BAE Systems COCOMO Forum 2008 Los Angeles,
1 COSYSMO 2.0: A Cost Model and Framework for Systems Engineering Reuse Jared Fortune University of Southern California Ricardo Valerdi Massachusetts Institute.
COSOSIMO* Workshop Outbrief 14 March 2006 Jo Ann Lane University of Southern California Center for Software Engineering CSE.
Local Bias and its Impacts on the Performance of Parametric Estimation Models Accepted by PROMISE2011 (Best paper award) Ye Yang, Lang Xie, Zhimin He (iTechs)
University of Southern California Center for Systems and Software Engineering ©USC-CSSE1 Ray Madachy USC Center for Systems and Software Engineering
COSYSMO Reuse Extension COSYSMO Workshop – USC CSSE Annual Research Review March 17, 2008 Ricardo ValerdiGan Wang Garry RoedlerJohn Rieff Jared Fortune.
©2006 BAE Systems. Practical Implementation of COSYSMO Reuse Extension Gan Wang, Aaron Ankrum, Cort Millar, Alex Shernoff, Ricardo Valerdi.
Towards COSYSMO 2.0: Update on Reuse Jared Fortune, USC Ricardo Valerdi, MIT USC ARR 2009 Los Angeles, CA.
Copyright © 2001, Software Productivity Consortium NFP, Inc. SOFTWARE PRODUCTIVITY CONSORTIUM SOFTWARE PRODUCTIVITY CONSORTIUM COSYSMO Overview INCOSE.
1 Prediction of Software Reliability Using Neural Network and Fuzzy Logic Professor David Rine Seminar Notes.
ESD web seminar1 ESD Web Seminar February 23, 2007 Ricardo Valerdi, Ph.D. Unification of systems and software engineering cost models.
“Simulation Approach for Impact Analysis of Requirement Volatility Considering Dependency Change” Tiemen Roest.
Slide 1 Estimating Performance Below the National Level Applying Simulation Methods to TIMSS Fourth Annual IES Research Conference Dan Sherman, Ph.D. American.
University of Southern California Center for Systems and Software Engineering COSATMO/COSYSMO Workshop Jim Alstad, USC-CSSE Gan Wang, BAE Systems Garry.
Gan Wang 22 October th International Forum on COCOMO® and Systems/Software Cost Modeling in conjunction with the Practical Software and Systems.
University of Southern California Center for Systems and Software Engineering COCOMO Suite Toolset Ray Madachy, NPS Winsor Brown, USC.
University of Southern California Center for Systems and Software Engineering Dr. Mauricio Peña January 28, 2013.
Proposed Metrics Definition Highlights Raymond Madachy Naval Postgraduate School CSSE Annual Research Review March 8, 2010.
University of Southern California Center for Systems and Software Engineering Reducing Estimation Uncertainty with Continuous Assessment: Tracking the.
Local Calibration: How Many Data Points are Best? Presented by Barry Boehm on behalf of Vu Nguyen, Thuy Huynh University of Science Vietnam National University.
University of Southern California Center for Systems and Software Engineering Reducing Estimation Uncertainty with Continuous Assessment Framework Pongtip.
University of Southern California Center for Systems and Software Engineering 1 Requirements Management and Volatility Mauricio E. Peña, Ph.D. January.
University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 1 November 2 nd, 2011 Mauricio E. Peña Dr. Ricardo.
Project Cost Management
Estimate Testing Size and Effort Using Test Case Point Analysis
Status Report Jim VanGaasbeek Ricardo Valerdi
Mathematical Formulation and Validation of the Impact of Requirements Volatility on Systems Engineering Effort March 6, 2012 Mauricio E. Peña.
The Effects of Reuse on Legacy DoD Systems
Requirements Volatility
Towards COSYSMO 2.0: Update on Reuse
Working Group Meeting Report
University of Southern California Center for Software Engineering
COSYSMO: Constructive Systems Engineering Cost Model
Mathematical Formulation and Validation of the Impact of Requirements Volatility on Systems Engineering Effort March 6, 2012 Mauricio E. Peña.
Generalized Reuse Model for COSYSMO Workshop Outbrief
Presentation transcript:

October 18, 2012 Mauricio E. Peña Ricardo Valerdi Quantifying the Impact of Requirements Volatility on Systems Engineering Effort October 18, 2012 Mauricio E. Peña Ricardo Valerdi

Outline Motivation and Introduction Research Methods Model Development Results Cross-Validation and Sensitivity Analysis Conclusion Threats to Validity Future Research

Importance of Understanding Requirements Volatility Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects1 Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans2 The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure3 System developers often lack effective methods and tools to account for and manage requirements volatility Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393

Requirements Volatility is Expected Changes to requirements are a part of our increasingly complex systems & dynamic business environment Stakeholders needs evolve rapidly The customer may not be able to fully specify the system requirements up front New requirements may emerge as knowledge of the system evolves Requirements often change during the early phases of the project as a result of trades and negotiations Requirements volatility must be anticipated and managed Sources: Kotonya and Sommerville (1995); Reifer (2000)

CSSE Parametric Cost Models The Constructive Systems Engineering Cost Model (COSYSMO) was developed by the USC Center for Software and Systems Engineering (CSSE) in collaboration with INCOSE and Industry affiliates COSYSMO is the first generally-available parametric cost model designed to estimate Systems Engineering effort Built on experience from COCOMO 1981, COCOMO II During the development of COSYSMO, volatility was identified as a relevant adjustment factor to the model’s size drivers Source: 7th Annual Practical Software and Systems Measurement Conference. COSYSMO Workshop, Boehm

Research Methods Data gathered from 25 industry projects Literature Review & 6 Workshops completed We are here Source: Boehm et al (2000)

Organizations that Participated in the Research The Aerospace Corporation Northrop Grumman Corporation The Boeing Company Raytheon United Launch Alliance BAE TI Metricas Ltda. IBM Distributed Management MIT USC Lockheed Martin Ericsson España Samsung SDS Rolls Royce Softstar Texas Tech The US Army The US Navy The US Air force The Australian Department of Defense

Observations from the Literature and Research Workshops Requirements volatility is caused by an identifiable set of project and organizational factors The level of requirements volatility is a function of the system life cycle phase Requirements volatility leads to an increase in project size and cost The cost and effort impact of a requirements change increases the later the change occurs in the system life cycle The impact of requirements volatility may vary depending on the type of change: added, deleted, or modified Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Ferreira (2002); Kulk and Verhoef 2008

COSYSMO Cost Estimating Relationships (CERs) SE_Hrs = Systems Engineering effort (hours) A = calibration constant derived from historical project data SIZE = measure of functional size of the system (number of requirements, interfaces, algorithms, operational scenarios) n = number of cost drivers (14) EM = effort multiplier for the ith cost driver The exponent (E) accounts for diseconomies of scale Source: Valerdi (2005) 9

Model Form Incorporated volatility effects through a scale factor (SF) added to the diseconomies of scale exponent (E) Similar approach used to model volatility effects in Ada COCOMO Prior research points to the compounding or exponential effect of requirements volatility Linearized form: Source: Boehm, B. and Royce, W. (1989); Kulk and Verhoef (2008); Wang, G. et al., (2008)

Volatility Scale Factor Expected REVL is rated as Very Low, Low, Moderate, High, and Very High Where REVL = The % of the baseline requirements that is expected to change over the system lifecycle C = Scale factor constant = 0.05 wvl = aggregate lifecycle phase volatility weighting factor And: wl = weighting factor for each life cycle phase1 Θl = % of total requirements changes per life cycle phase l = life cycle phases 1Life Cycle Phases: Conceptualize, Development, Operational Test and Evaluation, and Transition to Operation

Data Collection Collected expert judgment on the causes of volatility, expected REVL, and life cycle weighting factors through surveys conducted at 6 workshops Software and Systems Engineers with 20+ years of experience Variety of industries represented with an emphasis on Aerospace and Defense Historical Data Collection Data were collected from 25 projects from the Aerospace and Defense application domain Collected COSYSMO size, effort, and cost driver data Collected volatility data: added, modified, and deleted requirements over time

Causes of Requirements Volatility (N = 37)

Requirements Volatility (REVL) Rating Levels Developed based on surveys of experienced S/W and Systems Engineers (N =38)

Requirements Volatility Profile Based on expert judgment collected through three workshops (N = 36) and historical project data (N = 25)

Updated parameter mean and variance Model Calibration A Priori Model Historical data Mean and Variance Data-determined Model Regression Analysis; OLS Initial parameter mean and variance Expert-judgment estimates A Posteriori Model Updated parameter mean and variance Bayesian Analysis Formally combines expert judgment With sample data Optimized combination of data sources to increase precision Source: Boehm et al. (2000); Nguyen (2010)

Regression and Bayesian Calibration The coefficient β1 for the requirements volatility scale factor was calculated through linear regression analysis using data from the 25 projects The data-determined coefficient β1 was formally combined with the a-priori expert judgment to develop the Bayesian calibrated model   Coefficient Standard Error t Stat P-value Predictor 1.961 0.851 2.304 0.03 A posteriori Bayesian Update 1.96 2.09 2.01 * Data Analysis A priori Expert Judgment 17

Model Performance Comparison Model performance evaluated using the baseline model and a model with local calibration The performance of COSYSMO improves by including the requirements volatility factor

Coefficient of Determination (n=25) Academic COSYSMO COSYSMO with Requirements Volatility Factor * Relative Scale

Cross-Validation Results The 25 projects were randomly divided into K=6 subsets One of the subsets is excluded from the data set from which the model is built The resulting model is used to predict effort for the excluded cases This method is repeated for all subsets and the mean magnitude of relative errors (MMRE) and prediction accuracy PRED (l) across all K trials is calculated Calibrated Model Estimation Accuracy PRED(.15) PRED(.20) PRED(.30) MMRE Academic COSYMO 52% 64% 84% 20% Req. Volatility Model 76% 88% 15% Cross-Validation 80% 16%

Sensitivity Analysis Scenario 1 Evaluated the sensitivity of the model’s results to the variability of key parameters 2 Scenarios were developed based on the standard deviation in the volatility life cycle profile (Θl) Scenario 1: None of the requirements changes occur in the conceptualize phase Scenario 2: 66% of the changes occur in the conceptualize phase % Volatility per Life Cycle Phase Conceptualize Development Operational Test & Eval Transition to Operations Scenario 2 % Volatility per Life Cycle Phase Conceptualize Development Operational Test & Eval Transition to Operations Source: Guidelines for the Economic Analysis of Projects (1997)

Sensitivity Analysis Results The performance of the model scenarios was calculated using the K-fold cross-validation method The improvement in performance of the model with the requirements volatility scale factor over Academic COSYSMO remains largely unchanged Calibrated Model Estimation Accuracy PRED(.15) PRED(.20) PRED(.30) MMRE Academic COSYMO 52% 64% 84% 20% Req. Volatility Model 76% 88% 15% Cross-Val Scenario 1 72% 80% 16% Cross-Val Scenario 2

Conclusions The volatility of requirements throughout a system’s life cycle was found to be statistically significant predictor (p = 0.03) of systems engineering effort The prediction accuracy of Academic COSYSMO improved by incorporating a requirements volatility factor into the model Cross validation and sensitivity analyses were performed to demonstrate the robustness of the model in predicting effort for new projects

Threats to Validity Internal validity External validity Reliability Data collected from an engineering organization is influenced by many factors and all the relevant variables cannot be fully controlled as they would be in a true experiment External validity Limited by the organizations that contribute data and the background of the industry experts that participate in the research While a model calibrated to a single organization’s data can be useful, it is less than fully definitive for other organizations. Reliability Potential inaccuracies in effort and requirements metrics due to changes in business systems and practices over the time span of the projects Mitigated though: The use of consistent counting rules and involvement from personnel familiar with the projects Holding workshops to ensure survey respondents had a consistent understanding of the questions 24

Future Research Increase the quantity and diversity of the data – calibrate the model using data from additional organizations Evaluate the potential interaction between requirements volatility and reuse Further investigation of the impact of systems engineering effort depending on the type of change: added, modified, and deleted

References Boehm, B., Abts, C., Brown, A.W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D.J., and Steece, B. (2000). Software Cost Estimation with COCOMO II. Prentice Hall Department of Defense (2010). Quadrennial Defense Review Report Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements volatility in software engineering by using analytical modeling and software process simulation.” The Journal of Systems and Software. Vol. 82, pp 1568-1577. Wang, G., Boehm, B., Valerdi, R., and Shernoff, A. (2008). “Proposed Modification to COSYSMO Estimating Relationship.” Technical Report. University of Southern California, Center for Systems and Software Engineering. General Accounting Office (2004). Stronger Management Practices are Needed to Improve DOD’s Software-intensive Weapon Acquisitions (GAO-04-393). Defense Acquisitions. Hammer, T., Huffman, L., and Rosenberg, L. (1998). “Doing requirements right the first time.” Crosstalk, the Journal of Defense Software Engineering. Pp 20-25. Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State University ISO/IEC (2008). ISO/IEC 15288:2008 (E) Systems Engineering - System Life Cycle Processes. Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and Sons, Ltd. Malaiya, Y., and Denton, J. (1999). “Requirements Volatility and Defect Density.” Proceedings of the International Symposium on Software Reliability Engineering. MIL-STD-498. 1994. Software Development and Documentation. U.S. Department of Defense. Nguyen, V. and Boehm, B. (2010). A COCOMO Extension for Software Maintenance. 25th International Forum on COCOMO and Systems/Software Cost Modeling Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department. Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project Performance. Proceedings of the Ninth Asia-Pacific Software Engineering Conference 26

Back-up

Scale Factor Correlation Matrix The correlation between the Requirements Volatility Scale factor and COSYSMO cost drivers that are potential sources of volatility was evaluated   Volatility Scale Factor (SF) Requirements Understanding Personnel experience Technology Risk Volatility Scale Factor (SF) 1 Requirements Understanding 0.3878 Personnel experience 0.0074 0.4815 Technology Risk 0.4462 0.2060 0.0226 The correlations between predictors were below the correlation threshold of 0.66 used for COCOMO II and COSYSMO 28

Cumulative Volatility 54% 38% REVL (% of baseline requirements changes) 22% 6% Conceptualize Development Operational Test & Eval Transition to Operations Back 29