1. University of Southern California Center for Systems and Software Engineering 1 November 2010 Mauricio Peña Dr. Ricardo Valerdi COSYSMO Requirements Volatility Workshop

2. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 2 Workshop Agenda Introductions & Objectives8:30 – 8:45 am SE Leading Indicators &8:45 – 9:00 am Requirements Volatility Background Causal Model and Feedback9:00 – 9:15 am Survey Results9:15 – 9:30 am Requirements Volatility Weighting Factor9:30 – 10:00 am Break10:00 – 10:10 am Inputs and Discussion10:10 – 11:00 am

3. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 3 Objectives of the Workshop Discuss the causes of requirements volatility and its impact on systems engineering effort Obtain feedback on a proposed extension to COSYSMO to incorporate a requirements volatility factor Determine the expected level of requirements volatility over the system lifecycle phases Obtain inputs on the effort penalty due to late changes in requirements (volatility weighting factor) Provide an opportunity for participants to exchange lessons learned on requirements volatility and influence the direction of future research

4. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 4 Overview of Research Findings Field research validated findings from prior studies: –Requirements volatility is linked to an increase in rework and project size –The impact of changing a requirement increases the later the change occurs in the system lifecycle The research provided additional insights: 1.Causal model linking volatility to a number of technical, organizational and contextual factors 2.The level of volatility is a function of lifecycle phase 3.Respondents from S/W intensive projects tend to expect more volatility than those who work on H/W intensive systems 4.There are spikes in volatility after the transitions between lifecycle phases 5.Requirements changes early in the lifecycle may not be considered “volatility”

5. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 5 Motivating Questions When in the lifecycle do we start accounting for volatility (post-SRR, milestone B)? Should requirements volatility be modeled as a cumulative value or per lifecycle phase? What cost penalty factor should be used for changes occurring after the requirements baseline? Should a different cost penalty factor be used for added, deleted, and modified requirements? How should we aggregate these factors to determine the impact to systems engineering effort?

6. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 6 Requirements Volatility Definitions Requirements volatility is the % change in requirements (added, deleted, and modified) over a given time interval Also known as: Requirements creep: An increase in scope and/or number of system requirements Requirements churn: Instability in the requirements set – requirements are frequently modified or reworked without necessarily resulting in an increase in the total number of requirements Costello, R. and Liu, D. (1995). “Metrics for Requirements Engineering,” Journal of Systems and Software. Vol 29 (No. 1), pp. 39-63 MIL-STD-498. 1994. Software Development and Documentation. U.S. DoD Volatility Metrics (Monthly) % of total Requirements Changing

7. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 7 SE Leading Indicators Guide Leading Indicators are defined as “measures for evaluating the effectiveness of the systems engineering activities on a program in a manner that provides information about impacts that are likely to affect the system or program performance objectives.” Rhodes, D., Valerdi, R., and Roedler, G. (2009). “Systems engineering leading indicators for assessing program and technical effectiveness.” Systems Engineering Vol. 12 (No. 1), pp 21-35.

8. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 8 Requirements Trends as a Systems Engineering Leading Indicator Evaluates trends in the growth and change of the system requirements It helps to determine the stability and completeness of the system requirements which could potentially impact project performance Systems Engineering Leading Indicators Guide, Version 2.0, 2010

9. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 9 Implications to COSYSMO During the development of COSYSMO, volatility was identified as a relevant adjustment factor to the model’s size drivers However, there was insufficient data to incorporate volatility effects into the initial version of the model The primary objective of the research is to complete the requirements volatility extension to COSYSMO within the existing structure and scope of the model Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department.

10. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 10 COSYSMO Volatility Factor Fortune, J. (2009). Estimating systems engineering reuse with the constructive systems engineering cost model (COSYSMO 2.0). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department.

11. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 11 Method First Phase of the Study: Review of relevant literature Data collected through field research: surveys and discussions conducted at industry/academic conferences and workshops Boehm, B. (1981). Software Engineering Economics. Prentice Hall.

12. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 12 Literature Background Most of the requirements volatility research to date has been focused on software systems Various research methods have been utilized to investigate the causes and effects of requirement volatility, including: There is still a lack of empirical data on the quantitative impact of requirements volatility on for a broader base of engineering projects

13. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 13 Cost Commitment on Projects Blanchard and Fabrycky (1998), Systems Engineering & Analysis, Prentice Hall, 1998 Changes to the System are more difficult to implement the later they occur in the lifecycle

14. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 14 Causal Model (normative) Based on the review of the literature, a causal model was developed that relates technical, organizational and contextual project factors to requirements volatility Survey results were used to rank the level of subject- matter expert agreement with each of the postulated causes of requirements volatility.

15. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 15 Questions for Discussion Are we missing key causes of requirements volatility? In your view, is there a relationship between the project schedule and volatility? Do you agree with the rankings? Do you agree with the sign of the relationships between variables? Do you find the causal model useful? If not, what would you add?

16. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 16 Exploratory Survey Developed to gather the perspectives of subject-matter experts on the causes, impacts, and expected level of requirements volatility for a given system of interest Piloted at the 2010 USC-CSSE Annual Research Review Incorporated feedback and administered the survey at the 2010 Lean Advancement Initiative (LAI) knowledge exchange event in Dana Point, CA Organizations represented: –The Aerospace Corporation, Northrop Grumman Corporation –The Boeing Company, Softstar Systems, Raytheon –United Launch Alliance, Massachusetts Institute of Technology, University of Southern California, and –United States Army –Unites States Navy

17. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 17 Expected Level of Volatility Most respondents expect >20% volatility during the conceptualize phase of the project, decreasing to <5% in the transition to operation phase

18. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 18 Impact of Hardware/Software Project Breakdown on Expected Volatility Operational Test & Evaluation Lifecycle Phase Transition to Operation Lifecycle Phase Respondents from S/W intensive projects tend to expect more volatility later in the lifecycle

19. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 19 % of respondents Impacts of Volatility In general, results of the survey support observations from the literature and causal model Most respondents stated that requirements volatility will cause a moderate to large increase in the number of system requirements and rework Moderate decreaseModerate Increase No impactLarge Increase

20. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 20 Survey Exercise Survey Exercise administered during the 2010 Practical Software and Systems Measurement Conference Participants were asked to: 1.Draw a requirements volatility profile across the lifecycle phases covered by COSYSMO 2.Draw an “ease of change” profile across the same life cycle phases to determine the volatility weighting factor 3.Discuss variation in 1 and 2 above for: 1.Large and Small Projects 2.Hardware and Software Projects 3.Development and Recurring Projects

21. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 21 Expected Requirements Volatility Profile Conceptualize Development Operational Test & Evaluation Transition to Operation % of Requirements, Added, Deleted or Modified 30% 15% 4 out of 9 participants indicated that requirements changes should not be considered volatility during the conceptualize phase Localized peaks in volatility due to the transitions between lifecycle phases Profile Representative of Participant Feedback No significant differences between type of projects

22. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 22 Ease of Change Profile Cost Penalty defined as 1 / ease of change Average Ease of Change Factor (Estimated) Average Cost Penalty (Estimated) 0.8 0.40.10.05 1.25 2.510 20

23. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 23 Requirements Volatility Effect on SE Effort Profile Ease of Change COSYSMO Effort Profile Conceptualize Develop Operational Test & Evaluation Transition to Operation % Volatility Volatility Profile

24. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 24 Volatility Adjustment Factor (1 of 3) REVL is defined as the percentage of the baseline set of requirements that is likely to change due to the technical and organizational factors captured in the causal model This relationship is expressed through the following equation: Where, R 0 = Baseline number of requirements R eff = Effective number of requirements at the end of the project The effective increase in the number of requirements would result in an associated increase in systems engineering effort Boehm, B., Abts, C., Brown, A.W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D.J., and Steece, B. (2000). Software Cost Estimation with COCOMO II. Prentice Hall.

25. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 25 Volatility Adjustment Factor (2 of 3) In COSYSMO, the requirements are categorized by level of complexity as “easy,” “nominal,” and “difficult” Applying the three categories to the equation below results in the following relationship Where, R e,r = Initial number of requirements classified as “easy” R n,r = Initial number of requirements classified as “nominal” R d,r = Initial number of requirements classified as “difficult”

26. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 26 Volatility Adjustment Factor (3 of 3) Observations from the literature indicate that requirements added post-SRR carry an effort penalty due to the potential rework and collateral impact to other engineering products A weighting factor is added to account for this additional effort by increasing the effective functional size of the project Where, w v = Requirements volatility weighting factor Wv = 1 / ease of change

27. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 27 Measuring Volatility ConceptualizeDevelop Operational Test & Evaluation Transition to Operation % of Requirements, Added, Deleted or Modified (REVL) 100 80 60 40 20 0 Cumulative REVL Lifecycle phase REVL 20% 13% 5% 2% 33% 38% 40%

28. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 28 Questions for Discussion Should REVL be modeled as a cumulative value or per lifecycle phase? Should a different weighting factor be used for each lifecycle phase? Should a different weighting factor be used for added, deleted, and modified requirements? How should we aggregate these factors to determine the impact to systems engineering effort?

29. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 29 REVL Breakdown Example Cum REVL = 37% % of the total number of changes

30. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 30 Proposed Aggregate Weighting Factor As previously discussed, the effort penalty due to requirements volatility is likely to increase across lifecycle phases In order to capture this potential lifecycle phase dependency in the model, the following relationship is proposed: Where w v = Requirements volatility weighting factor w x,l = Weighting factor for added, deleted, or modified Θ x,l = % of total changes that were added, deleted or modified l = lifecycle phases

31. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 31 Weighting Factor Example R eff = 1 + ((REVL/100)*W v ) R eff = (1+(0.37*2.8)) R eff = 2.0 = double the original estimate of effective # of requirements and systems engineering effort

32. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 32 Expected Volatility (1 of 2) What % percentage of the baseline set of requirements that is likely to change over the lifecycle phases covered by COSYSMO for your system of interest? REVL_________ % What is your estimate of the REVL per lifecycle phase? Conceptualize________% Development________ % Operational Test & Evaluation________ % Transition to Operation________ %

33. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 33 Expected Volatility (2 of 2) Based on your experience, what is the breakdown of the type of requirements changes for your system of interest? Change type: Added__________ % Deleted__________ % Modified__________ % 100%

34. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 34 Inputs Requested – Weighting Factor Please input a weighting factor (effort penalty) per lifecycle phase (1 = no penalty) Is the weighting factor different, depending on the type of change? Y / N If yes, please fill out the following table:

35. University of Southern California Center for Systems and Software Engineering 35 Thank you!

36. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 36 References Boehm, B. (1991). Software Risk Management: Principles and Practices. IEEE Software 8(1), pp 32-41. Blanchard, B. and Fabrycky, W. (1998). Systems Engineering & Analysis, Prentice Hall, New York. Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements volatility in software engineering by using analytical modeling and software process simulation.” The Journal of Systems and Software. Vol. 82, pp 1568-1577. Fortune, J. (2009). Estimating systems engineering reuse with the constructive systems engineering cost model (COSYSMO 2.0). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department. General Accounting Office (2004). Stronger Management Practices are Needed to Improve DOD’s Software- intensive Weapon Acquisitions (GAO-04-393). Defense Acquisitions Honour, E. (2004). “Understanding the Value of Systems Engineering.” INCOSE 14th Annual International Symposium: Systems Engineering Managing Complexity and Change. Toulouse, France Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State University INCOSE Systems Engineering Handbook, Version 3, INCOSE, June 2006 ISO/IEC (2008). ISO/IEC 15288:2008 (E) Systems Engineering - System Life Cycle Processes. Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and Sons, Ltd. MIL-STD-498. 1994. Software Development and Documentation. U.S. Department of Defense. Roedler, G. and Rhodes, D. (2007). Systems engineering leading indicators guide. Version 1. Massachusetts Institute of Technology, INCOSE, and PSM. Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department. Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project Performance. Proceedings of the Ninth Asia-Pacific Software Engineering Conference

37. University of Southern California 25 th Annual COCOMO Forum Center for Systems and Software Engineering 37 Call for Participation Background The USC Center for Systems and Software Engineering (CSSE) and the Lean Advancement Initiative (LAI) at MIT in collaboration with the INCOSE Measurement Working Group has embarked on an effort to develop an update to COSYSMO, the Constructive Systems Engineering Cost Model. This incremental update aims to improve the estimation power of the model by accounting for requirements volatility. To perform an industry calibration, we are seeking industry data for system engineering projects in the form of systems engineering effort actuals (labor hours) and requirements volatility: the number of requirements, added, deleted, and modified after the requirements baseline (i.e.. SRR, milestone B). Benefits By providing data for this model your organization will: Improve its ability to estimate the impact of requirements changes on project cost learn to tailor and calibrate the model for their specific application domain be able to claim in CMMI reviews that their systems engineering cost estimates are based on calibrated industry models Proven Methodology COSYSMO (Constructive Systems Engineering Cost Model) employs a proven methodology developed for the COCOMO (Constructive Cost Model), the most widely used software cost model in the world. Proven Process USC CSSE and LAI at MIT have proven processes in place to ensure the confidentiality of the data with its Corporate Affiliates and Consortium Members. Successful data protection has enabled it to attract the participation of several organizations in this effort including Boeing, Raytheon, Northrop Grumman, Lockheed Martin, General Dynamics, SAIC, L-3 Communications BAE Systems and Air Center Communications, Systems, the US Force Space & Missile Systems Center. Contact: Mauricio E. Peña [mauricip@usc.edu] Ricardo Valerdi [rvalerdi@mit.edu]