University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 1 November 2 nd, 2011 Mauricio E. Peña Dr. Ricardo Valerdi Mathematical Formulation and Validation of the Impact of Requirements Volatility on Systems Engineering Effort

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 2 Outline Motivation and introduction Implications to COSYSMO Research methods Observations from exploratory research Proposed model of the impact of requirements volatility on Systems Engineering effort Preliminary model validation Conclusions and next steps

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 3 Importance of Understanding Requirements Volatility Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects 1 Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans 2 The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure 3 System developers often lack effective methods and tools to account for and manage requirements volatility Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 4 Requirements Volatility is Expected Changes to requirements are a part of our increasingly complex systems & dynamic business environment –Stakeholders needs evolve rapidly –The customer may not be able to fully specify the system requirements up front –New requirements may emerge as knowledge of the system evolves –Requirements often change during the early phases of the project as a result of trades and negotiations Sources: Kotonya and Sommerville (1995); Reifer (2000) Requirements volatility must be anticipated and managed

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 5 Requirements Volatility Definitions Requirements volatility is typically defined as the change in requirements (added, deleted, and modified) over a given time interval Also known as: Requirements creep: An increase in scope and number of system requirements Requirements churn: Instability in the requirements set – requirements are modified or re-worked without necessarily resulting in an increase in the total number of requirements Source: MIL-STD-498 (1994)

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 6 CSSE Parametric Cost Models The Constructive Systems Engineering Cost Model (COSYSMO) was developed by the USC Center for Software and Systems Engineering (CSSE) in collaboration with INCOSE and Industry affiliates COSYSMO is the first generally-available parametric cost model designed to estimate Systems Engineering effort Built on experience from COCOMO 1981, COCOMO II During the development of COSYSMO, volatility was identified as a relevant adjustment factor to the model’s size drivers Source: 7th Annual Practical Software and Systems Measurement Conference. COSYSMO Workshop, Boehm

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 7 COSYSMO 2.0 Operational Concept Source: Fortune (2009) Proposed Volatility Factor

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 8 Life Cycle Phase Definition Conceptualize phase focuses on identifying stakeholder needs, exploring different solution concepts, and proposing candidate solutions. The Development phase involves refining the system requirements, creating a solution description, and building a system. The Operational Test & Evaluation Phase involves verifying/validating the system and performing the appropriate inspections before it is delivered to the user The Transition to Operation Phase involves the transition to utilization of the system to satisfy the users’ needs. Source: Valerdi (2008); ISO/IEC (2002)

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 9 Research Methods In Progress Source: Boehm et al (2000) Literature Review & 6 Workshops completed to date

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 10 Observations from the Literature and Exploratory Research 1.Requirements volatility is caused by an identifiable set of project and organizational factors. 2.The level of requirements volatility is a function of the system life cycle phase. 3.Requirements volatility leads to an increase in project size and cost 4.The cost and effort impact of a requirements change increases the later the change occurs in the system life cycle 5.The impact of requirements volatility varies depending on the type of change: added, deleted, or modified

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 11 Life Cycle Effort Penalty due to Volatility (n = 27) Data collected from two workshops: 25th Annual USC CSSE COCOMO and the 2011 USC-CSSE Annual Research Review Systems Engineering Effort Penalty Due to Volatility Conceptualize Development Operational Test & Evaluation Transition to Operation

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 12 Estimated Life Cycle Effort Penalty per Change Category (n = 9) Results from a two-round Delphi Survey held at the 2011 Practical Software and Systems Measurement Conference Conceptualize Development Operational Test & Evaluation Transition to Operation Estimated SE Effort Penalty due to Volatility

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 13 Mathematical Formulation of Requirements Volatility ( 1 of 2) Where, R 0 = Baseline number of requirements Req = Equivalent number of requirements w v = Requirements Volatility weighting factor The effective increase in the number of requirements would result in an associated increase in systems engineering effort Sources: Boehm, B., et al (2000)

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 14 Where w v = Requirements volatility weighting factor w x,l = Weighting factor for added, deleted, or modified requirements Θ x,l = % of total requirements changes that were added, deleted or modified l = lifecycle phases Pena (2010) Mathematical Formulation of Requirements Volatility ( 2 of 2)

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 15 Preliminary Model Validation Data were collected from nine projects from a space systems application domain –Number of requirements, interfaces, algorithms operational scenarios (estimated systems engineering size) –Added, modified, and deleted requirements over time The cost estimation accuracy of COSYSMO was compared to the accuracy of the model with the requirements volatility size driver adjustment factor The models were tested using the coefficient of determination (R 2 ) and predictive accuracy levels The relationship between requirements volatility and systems engineering size was also examined

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 16 Baseline Coefficient of Determination (n = 9) * Due to proprietary reasons only the analysis of the model accuracy is shown, not the data itself

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 17 Coefficient of Determination with Volatility Factor (n = 9)

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 18 Prediction Accuracy Level The prediction accuracy at a particular level (l) is defined as Where, n = set of projects k = number of projects in the set whose Magnitude of Relative Error is ≤ l Source: Conte, Dunsmore, and Shen (1986)

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 19 Relationship Between Requirements Volatility and Sys. Engr. Size

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 20 Relationship Between the Volatility Adjustment Factor and Sys. Engr. Size

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 21 Conclusions – Next Steps Observations from the literature and workshops were used to develop a mathematical framework for quantifying the impact of requirements volatility on systems engineering effort A preliminary validation of the model was performed by comparing its prediction accuracy against COSYSMO The preliminary results indicate an improvement in systems engineering effort prediction accuracy when a requirements volatility factor is added to the model The correlation between the requirements volatility factor and the estimated systems engineering size was examined – the results were not conclusive Additional project data will be collected to complete the model validation

University of Southern California Center for Systems and Software Engineering 26 th Annual COCOMO Forum 22 References B. Blanchard and W. Fabrycky, Systems engineering & analysis, Prentice Hall, New York, NY, B. Boehm, Software risk management: Principles and practices, IEEE Software 1 (1991), B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E., Horowitz, R. Madachy, D.J. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, New York, NY, 2000 S. Conte, H. Dunsmore, and V. Shen. Software Engineering Metrics and Models. Benjamin/Cummings Publishing Company, Department of Defense, “Instruction Operation of the Defense Acquisition System,” S. Ferreira, J. Collofello, D. Shunk and G. Mackulak, Understanding the Effects of Requirements Volatility in Software Engineering by Using Analytical Modeling and Software Process Simulation, Journal of Systems and Software 82 (2009) General Accounting Office, "Stronger management practices are needed to improve DoD’s software-intensive weapon acquisitions (GAO )," D. Houston, "A software project simulation model for risk management," Ph.D. Dissertation, Arizona State University, ISO/IEC, "ISO/IEC 15288:2002 (e) systems engineering - system life cycle processes," C. Jones, Assessment and Control of Software Risks, Prentice Hall, Inc. Englewood Cliffs, New Jersey, C. Jones, Strategies for managing requirements creep, IEEE Computer, Vol 20 (1996), pp G. Kotonya and I. Sommerville, Requirements engineering: Processes and techniques, John Wiley & Sons, New York, NY, G.P. Kulk, and C. Verhoef, Quantifying Requirements Volatility Effects, Science of Computer Programming, 72, , Y. Malaiya, and J. Denton, Requirements Volatility and Defect Density, Proceedings of the International Symposium on Software Reliability Engineering, 1999 MIL-STD-498, "Software development and documentation," S. Nidumolu, Standardization, Requirements Uncertainty and Software Project Performance, Information and Management. Vol 31 (No. 3) (1996), pp M. Pena, Characterizing the Impacts of Requirements Volatiliy, Proceedings of the 25th Annual COCOMO Forum, 2010 D. Pfahl, and K. Lebsanft, Using Simulation to Analyze the Impact of Software Requirements Volatility on Project Performance, Information and Software Technology, Vol 42 (No. 14) (2000), pp D. Reifer, Requirements management: The search for nirvana, IEEE Software 17(3) (2000), D. Rhodes, R. Valerdi and G. Roedler, Systems engineering leading indicators for assessing program and technical effectiveness, Systems Engineering 12(1) (2009), R. Valerdi, The Constructive Systems Engineering Cost Model (COSYSMO): Quantifying the Costs of Systems Engineering Effort in Complex Systems, Saarbrücken, Germany, VDM Verlag, D. Zowghi and N. Nurmuliani, A study of the impact of requirements volatility on software project performance, Proceedings of the Ninth Asia-Pacific Software Engineering Conference, 2002.