1. University of Southern California Center for Systems and Software Engineering ©USC-CSSE1 3/16/08 Modeling System and Software Engineering Processes with System Dynamics Ray Madachy USC Center for Systems and Software Engineering madachy@usc.edu Annual Research Review March 17, 2008

2. University of Southern California Center for Systems and Software Engineering ©USC-CSSE2 3/16/08 Outline Background and Introduction System Dynamics Examples –Optimizing quality assurance, defect dynamics –Brooks’s Law model –Process model for very large SISOS References Detailed Backups –Inspection model –Value-based business and software process/product model –Spiral hybrid process model for SISOS –Process concurrence

3. University of Southern California Center for Systems and Software Engineering ©USC-CSSE3 3/16/08 Background The evaluation of process strategies for the architecting and engineering of complex systems involves many interrelated factors. Effective systems and software engineering requires a balanced view of technology, business or mission goals, and people. System dynamics is a rich and integrative simulation framework used to quantify the complex interactions and the strategy tradeoffs between cost, schedule, quality and risk.

4. University of Southern California Center for Systems and Software Engineering ©USC-CSSE4 3/16/08 System Dynamics Principles Major concepts – Defining problems dynamically, in terms of graphs over time – Striving for an endogenous, behavioral view of the significant dynamics of a system – Thinking of all real systems concepts as continuous quantities interconnected in information feedback loops and circular causality – Identifying independent levels in the system and their inflow and outflow rates – Formulating a model capable of reproducing the dynamic problem of concern by itself – Deriving understandings and applicable policy insights from the resulting model – Implementing changes resulting from model-based understandings and insights. The continuous view –Individual events are not tracked –Entities are treated as aggregate quantities that flow through a system

5. University of Southern California Center for Systems and Software Engineering ©USC-CSSE5 3/16/08 System Dynamics Notation System represented by x’(t)= f(x,p). x: vector of levels (state variables), p: set of parameters Legend: Example system:

6. University of Southern California Center for Systems and Software Engineering ©USC-CSSE6 3/16/08 Model Elements

7. University of Southern California Center for Systems and Software Engineering ©USC-CSSE7 3/16/08 Model Elements (continued)

8. University of Southern California Center for Systems and Software Engineering ©USC-CSSE8 3/16/08 Modeling Criteria Overview Process V&VDoes not catch specific defects, but can model the aggregate effects of processes on defect rates/levels. Modeling the effects on n different types of defects involves scaling up a model multiplicatively by n model elements (flow chains, rates, levels and parameters). E.g. Dynamic ODC COQUALMO has defect chains replicated for 10 types (see slides later) Ambiguity ToleranceNo tolerance for ambiguity in flow chain representation, but parameters can be stochastic. ScalabilityThe size of process is unlimited. As the size/complexity grows, the technique can cut the level of entity aggregation higher. No overhead for increasing # of entities at a chosen aggregation. E.g. Extremely large SISOS processes we have modeled at the capability level (see slides later) ExtensibilityRequires little to moderate training in simulation to extend existing models. CoverageWell suited to estimate high-level budgets, schedules, and quality levels. Uniquely suited to monitor progress with respect to plans interactively and/or continuously. E.g. Earned Value model and others in Software Process Dynamics can generate time-phased plans, track them against actuals, and recalibrate new plans based on actuals No independent composition of documents or process elements. DynamismPerfectly suited to model instant or time-phased changes to processes. E.g. Brooks’s Law model of adding people to a late project (see slides later) E.g. Dynamic ODC COQUALMO model of quality process changes on defect introduction and removal rates (see slides later) DistributionWidely supported mature commercial modeling tools, but no vendors of software process models.

9. University of Southern California Center for Systems and Software Engineering ©USC-CSSE9 3/16/08 Additional Strengths / Weaknesses Strengths –Provides a global system perspective and the ability to analyze combined strategies –Analysis of integrated process factors and feedback loops (e.g. holistic enterprise models, positive and negative feedback, process delays, process concurrence) –Finding the right process balance (e. g., quality assurance levels, staffing levels) –Adapting to change; strategic long-term policy analysis –Can model inherent tradeoffs between schedule, cost and quality –Attractive for schedule analysis accounting for critical path flows, task interdependencies and bottlenecks not available with static models or PERT/CPM methods Weaknesses –Assumption of homogeneity of entities; cannot attach attributes to individual entities –Sequential activities difficult to represent –Queuing and other discrete statistics not captured

10. University of Southern California Center for Systems and Software Engineering ©USC-CSSE10 3/16/08 Outline Background and Introduction System Dynamics Examples –Optimizing quality assurance, defect dynamics –Brooks’s Law model –Process model for very large SISOS References Detailed Backups –Inspection model –Value-based business and software process/product model –Spiral hybrid process model for SISOS –Process concurrence

11. University of Southern California Center for Systems and Software Engineering ©USC-CSSE11 3/16/08 QA Policy Tradeoff Analysis From [Abdel-Hamid/Madnick 91]

12. University of Southern California Center for Systems and Software Engineering ©USC-CSSE12 3/16/08 Dynamic ODC COQUALMO Portion Portion for Requirements Completeness defects only:

13. University of Southern California Center for Systems and Software Engineering ©USC-CSSE13 3/16/08 Dynamic ODC COQUALMO Sample Outputs Example of applying increased V&V for Execution Testing and Tools at 18 months:

14. University of Southern California Center for Systems and Software Engineering ©USC-CSSE14 3/16/08 Brooks’s Law Model “Adding manpower to a late software project makes it later” [Brooks 75]. A simple model based on the following assumptions: –New personnel require training by experienced personnel to come up to speed –More people on a project entail more communication overhead –Experienced personnel are more productive then new personnel, on average.

15. University of Southern California Center for Systems and Software Engineering ©USC-CSSE15 3/16/08 Brooks’s Law Model Output Sensitivity of Software Development Rate to Varying Personnel Allocation Pulses (1: no extra hiring, 2: add 5 people on 100th day, 3: add 10 people on 100th day) Days Function points/day

16. University of Southern California Center for Systems and Software Engineering ©USC-CSSE16 3/16/08 Large SISOS Model Overview Built around a cyclic flow chain for capabilities –Arrayed for multiple increments Each team is represented with a level and corresponding staff allocation rate Changes arrive a-periodically via the volatility trends time function and flow into the level for capability changes Changes are processed by the agile team and allocated to increments per the deferral policies –Constant or variable staffing for the agile team For each increment the required capabilities are developed into developed capabilities and then V&V’ed into V & V’ed capabilities –Productivities and team sizes for development and V&V calculated with a Dynamic COCOMO variant and continuously updated for scope changes –Flow rates between capability changes and V & V’ed capabilities are bi-directional for capability “kickbacks” sent back up the chain User-driven changes from the field are identified as field issues that flow back into the capability changes

17. University of Southern California Center for Systems and Software Engineering ©USC-CSSE17 3/16/08 Sample Response to Volatility An unanticipated change occurs at month 8 shown as a volatility trend [1] pulse It flows into capability changes [1] which declines to zero as the agile team processes the change The change is non-deferrable for increment 1 so total capabilities [1] increases Development team staff size dynamically responds to the increased scope * [1] refers to increment #1

18. University of Southern California Center for Systems and Software Engineering ©USC-CSSE18 3/16/08 Outline Background and Introduction System Dynamics Examples –Optimizing quality assurance, defect dynamics –Brooks’s Law model –Process model for very large SISOS References Detailed Backups –Inspection model –Value-based business and software process/product model –Spiral hybrid process model for SISOS –Process concurrence

19. University of Southern California Center for Systems and Software Engineering ©USC-CSSE19 3/16/08 References Abdel-Hamid T, Madnick S, Software Project Dynamics, Englewood Cliffs, NJ, Prentice-Hall, 1991 Kellner M, Madachy R, Raffo D, Software Process Simulation Modeling: Why? What? How?, Journal of Systems and Software, Spring 1999 Madachy R, System Dynamics Modeling of an Inspection-Based Process, Proceedings of the Eighteenth International Conference on Software Engineering, IEEE Computer Society Press, Berlin, Germany, March 1996 Madachy R, Boehm B, Lane J, Spiral Lifecycle Increment Modeling for New Hybrid Processes, Journal of Systems and Software, 2007 Madachy R., Software Process Dynamics, Wiley-IEEE Computer Society, 2008 Madachy R., Boehm B., Assessing Quality Processes with ODC COQUALMO, Proceedings of the 2008 International Conference on Software Process, Liepzig, Germany, 2008 USC Web Sites http://csse.usc.edu/softwareprocessdynamics http://rcf.usc.edu/~madachy/spd

20. University of Southern California Center for Systems and Software Engineering ©USC-CSSE20 3/16/08 Outline Background and Introduction System Dynamics Examples –Optimizing quality assurance, defect dynamics –Brooks’s Law model –Process model for very large SISOS References Detailed Backups –Inspection model –Value-based business and software process/product model –Spiral hybrid process model for SISOS –Process concurrence

21. University of Southern California Center for Systems and Software Engineering ©USC-CSSE21 3/16/08 Inspection Model Example Research problem addressed –What are the dynamic effects to the process of performing inspections? Model used to evaluate process quantitatively –Demonstrates effects of inspection practices on cost, schedule and quality throughout lifecycle –Can experiment with changed processes before committing project resources –Benchmark process improvement –Support project planning and management Model parameters calibrated to Litton and JPL data –Error generation rates, inspection effort, efficiency, productivity, others Model validated against industrial data

22. University of Southern California Center for Systems and Software Engineering ©USC-CSSE22 3/16/08 System Diagram

23. University of Southern California Center for Systems and Software Engineering ©USC-CSSE23 3/16/08 System Diagram (continued)

24. University of Southern California Center for Systems and Software Engineering ©USC-CSSE24 3/16/08 Inspection Policy Tradeoff Analysis Varying error generation rates shows diminishing returns from inspections [Madachy 94]:

25. University of Southern California Center for Systems and Software Engineering ©USC-CSSE25 3/16/08 Value-Based Model Background Purpose: Support software business decision- making by experimenting with product strategies and development practices to assess real earned value Description: System dynamics model relates the interactions between product specifications and investments, software processes including quality practices, market share, license retention, pricing and revenue generation for a commercial software enterprise

26. University of Southern California Center for Systems and Software Engineering ©USC-CSSE26 3/16/08 Model Features A Value-Based Software Engineering (VBSE) model covering the following VBSE elements: –Stakeholders’ value proposition elicitation and reconciliation –Business case analysis –Value-based monitoring and control Integrated modeling of business value, software products and processes to help make difficult tradeoffs between perspectives –Value-based production functions used to relate different attributes Addresses the planning and control aspect of VBSE to manage the value delivered to stakeholders –Experiment with different strategies and track financial measures over time –Allows easy investigation of different strategy combinations Can be used dynamically before or during a project –User inputs and model factors can vary over the project duration as opposed to a static model –Suitable for actual project usage or “flight simulation” training where simulations are interrupted to make midstream decisions

27. University of Southern California Center for Systems and Software Engineering ©USC-CSSE27 3/16/08 Model Sectors and Major Interfaces Software process and product sector computes the staffing and quality over time Market and sales sector accounts for market dynamics including effect of quality reputation Finance sector computes financial measures from investments and revenues

28. University of Southern California Center for Systems and Software Engineering ©USC-CSSE28 3/16/08 Software Process and Product product defect flows effort and schedule calculation with dynamic COCOMO variant

29. University of Southern California Center for Systems and Software Engineering ©USC-CSSE29 3/16/08 Finances, Market and Sales investment and revenue flows software license sales market share dynamics including quality reputation

30. University of Southern California Center for Systems and Software Engineering ©USC-CSSE30 3/16/08 Quality Assumptions COCOMO cost driver Required Software Reliability is a proxy for all quality practices Resulting quality will modulate the actual sales relative to the highest potential Perception of quality in the market matters –Quality reputation quickly lost and takes much longer to regain (bad news travels fast) –Modeled as asymmetrical information smoothing via negative feedback loop

31. University of Southern California Center for Systems and Software Engineering ©USC-CSSE31 3/16/08 Market Share Production Function and Feature Sets Cases from Example 1

32. University of Southern California Center for Systems and Software Engineering ©USC-CSSE32 3/16/08 Sales Production Function and Reliability Cases from Example 1

33. University of Southern California Center for Systems and Software Engineering ©USC-CSSE33 3/16/08 Example 1: Dynamically Changing Scope and Reliability Shows how model can assess the effects of combined strategies by varying the scope and required reliability independently or simultaneously Simulates midstream descoping, a frequent strategy to meet time constraints by shedding features Three cases are demonstrated: –Unperturbed reference case –Midstream descoping of the reference case after ½ year –Simultaneous midstream descoping and lowered required reliability at ½ year

34. University of Southern California Center for Systems and Software Engineering ©USC-CSSE34 3/16/08 Control Panel and Simulation Results Unperturbed Reference Case Case 2 Case 1 Descope Descope + Lower Reliability

35. University of Southern California Center for Systems and Software Engineering ©USC-CSSE35 3/16/08 Case Summaries CaseDelivered Size (Function Points) Delivered Reliability Setting Cost ($M) Delivery Time (Years) Final Market Share ROI Reference Case: Unperturbed 7001.04.782.128%1.3 Case 1: Descope at Time = ½ years 5501.03.701.728%2.2 Case 2: Descope and Lower Reliability at Time = ½ years 550.923.301.512%1.0

36. University of Southern California Center for Systems and Software Engineering ©USC-CSSE36 3/16/08 Example 2: Determining the Reliability Sweet Spot Analysis process –Vary reliability across runs –Use risk exposure framework to find process optimum –Assess risk consequences of opposing trends: market delays and bad quality losses –Sum market losses and development costs –Calculate resulting net revenue Simulation parameters –A new 80 KSLOC product release can potentially increase market share by 15%-30% (varied in model runs) –75% schedule acceleration –Initial total market size = $64M annual revenue Vendor has 15% of market Overall market doubles in 5 years

37. University of Southern California Center for Systems and Software Engineering ©USC-CSSE37 3/16/08 Cost Components 3-year time horizon

38. University of Southern California Center for Systems and Software Engineering ©USC-CSSE38 3/16/08 To achieve real earned value, business value attainment must be a key consideration when designing software products and processes Software enterprise decision-making can improve with information from simulation models that integrate business and technical perspectives Optimal policies operate within a multi-attribute decision space including various stakeholder value functions, opposing market factors and business constraints Risk exposure is a convenient framework for software decision analysis Commercial process sweet spots with respect to reliability are a balance between market delay losses and quality losses Model demonstrates a stakeholder value chain whereby the value of software to end-users ultimately translates into value for the software development organization Value-Based Model Conclusions

39. University of Southern California Center for Systems and Software Engineering ©USC-CSSE39 3/16/08 Value-Based Model Future Work Enhance product defect model with dynamic version of COQUALMO to enable more constructive insight into quality practices Add maintenance and operational support activities in the workflows Elaborate market and sales for other considerations including pricing scheme impacts, varying market assumptions and periodic upgrades of varying quality Account for feedback loops to generate product specifications (closed- loop control) –External feedback from users to incorporate new features –Internal feedback on product initiatives from organizational planning and control entity to the software process More empirical data on attribute relationships in the model will help identify areas of improvement Assessment of overall dynamics includes more collection and analysis of field data on business value and quality measures from actual software product rollouts

40. University of Southern California Center for Systems and Software Engineering ©USC-CSSE40 3/16/08 Outline Research introduction Value-based business and software process/product model Spiral hybrid process model for Software- Intensive System of Systems (SISOS) References Backup slides

41. University of Southern California Center for Systems and Software Engineering ©USC-CSSE41 3/16/08 Spiral Hybrid Process Introduction The spiral lifecycle is being extended to address new challenges for Software- Intensive Systems of Systems (SISOS), such as coping with rapid change while simultaneously assuring high dependability A hybrid plan-driven and agile process has been outlined to address these conflicting challenges with the need to rapidly field incremental capabilities A system-of-systems (SOS) integrates multiple independently-developed systems and is very large, dynamically evolving, unprecedented, with emergent requirements and behaviors –However, traditional static approaches cannot capture dynamic feedback loops and interacting phenomena that cause real-world complexity (e.g. hybrid processes, project volatility, increment overlap and resource contention, schedule pressure, slippages, communication overhead, slack, etc.) A system dynamics model is being developed to assess the incremental hybrid process and support project decision-making Both the hybrid process and simulation model are being evolved on a very large scale incremental project for a SISOS

42. University of Southern California Center for Systems and Software Engineering ©USC-CSSE42 3/16/08 Scalable Spiral Model Increment Activities Organize development into plan-driven increments with stable specs Agile team watches for and assesses changes, then negotiates changes so next increment hits the ground running Try to prevent usage feedback from destabilizing current increment Three team cycle plays out from one increment to the next

43. University of Southern California Center for Systems and Software Engineering ©USC-CSSE43 3/16/08 Spiral Hybrid Model Features Estimates cost and schedule for multiple increments of a hybrid process that uses three specialized teams (agile re-baseliners, developers, V&V’ers) per the scalable spiral model Considers changes due to external volatility and feedback from user-driven change requests Deferral policies and team sizes can be experimented with Includes tradeoffs between cost and the timing of changes within and across increments, length of deferral delays, and others

44. University of Southern California Center for Systems and Software Engineering ©USC-CSSE44 3/16/08 Model Input Control Panel

45. University of Southern California Center for Systems and Software Engineering ©USC-CSSE45 3/16/08 Model Overview Built around a cyclic flow chain for capabilities –Arrayed for multiple increments Each team is represented with a level and corresponding staff allocation rate Changes arrive a-periodically via the volatility trends time function and flow into the level for capability changes Changes are processed by the agile team and allocated to increments per the deferral policies –Constant or variable staffing for the agile team For each increment the required capabilities are developed into developed capabilities and then V&V’ed into V & V’ed capabilities –Productivities and team sizes for development and V&V calculated with a Dynamic COCOMO variant and continuously updated for scope changes –Flow rates between capability changes and V & V’ed capabilities are bi-directional for capability “kickbacks” sent back up the chain User-driven changes from the field are identified as field issues that flow back into the capability changes

46. University of Southern California Center for Systems and Software Engineering ©USC-CSSE46 3/16/08 Volatility Cost Functions The volatility effort multiplier for construction effort and schedule is an aggregate multiplier for volatility from different sources (e.g. COTS, mission, etc.) relative to the original baseline for increment The lifecycle timing effort multiplier models increased development cost the later a change comes in midstream during an increment

47. University of Southern California Center for Systems and Software Engineering ©USC-CSSE47 3/16/08 Sample Response to Volatility An unanticipated change occurs at month 8 shown as a volatility trend [1] pulse It flows into capability changes [1] which declines to zero as the agile team processes the change The change is non-deferrable for increment 1 so total capabilities [1] increases Development team staff size dynamically responds to the increased scope * [1] refers to increment #1

48. University of Southern California Center for Systems and Software Engineering ©USC-CSSE48 3/16/08 Sample Test Results Test case for two increments of 15 baseline capabilities each A non-deferrable change comes at month 8 (per previous slide) The agile team size is varied from 2 to 10 people Increment 1 business value also lost for agile team sizes of 2 and 4

49. University of Southern California Center for Systems and Software Engineering ©USC-CSSE49 3/16/08 System dynamics is a convenient modeling framework to deal with the complexities of a SISOS A hybrid process appears attractive to handle SISOS dynamic evolution, emergent requirements and behaviors Initial results indicate that having an agile team can help decrease overall cost and schedule –Model can help find the optimum balance Will obtain more empirical data to calibrate and parameterize model including volatility and change trends, change analysis effort, effort multipliers and field issue rates Model improvements –Additional staffing options Rayleigh curve staffing profiles Constraints on development and V&V staffing levels –More flexible change deferral options across increments –Increment volatility balancing policies –Provisions to account for (timed) business/mission value of capabilities Additional model experimentation –Include capabilities flowing back from developers and V&V’ers –Vary deferral policies and volatility patterns across increments –Compare different agile team staffing policies Continue applying the model on a current SISOS and seek other potential pilots Spiral Hybrid Model Conclusions and Future Work

50. University of Southern California Center for Systems and Software Engineering ©USC-CSSE50 3/16/08 Process Concurrence Basics Process concurrence describes interdependency constraints between tasks –can be an internal constraint within a development stage or an external constraint between stages Describes how much work becomes available for completion based on previous work accomplished Accounts for realistic bottlenecks on work availability –vs. a model driven solely by resources and productivity that can finish in almost zero time with infinite resources Concurrence relations can be sequential, parallel, partially concurrent, or other dependent relationships

51. University of Southern California Center for Systems and Software Engineering ©USC-CSSE51 3/16/08 Process Concurrence Suitability for Integration of Processes Process concurrence models the constraints on making work available between process activities. Provides a framework to characterize different approaches in terms of their ability to parallelize or accelerate activities –Can be used to derive optimal staffing profiles for different project situations Hence, it can model the integration and sequencing (in aggregate) of systems engineering and software engineering activities –E.g. The following external concurrence model can optimize staffing levels for given system architectural profiles

52. University of Southern California Center for Systems and Software Engineering ©USC-CSSE52 3/16/08 External Concurrence Model the time profile of tasks ready to elaborate ~ “ideal” staffing curve shape

53. University of Southern California Center for Systems and Software Engineering ©USC-CSSE53 3/16/08 Simulation Results and Sample Lessons flat Rayleigh COTS pulse at front N/A Critical customer decision delays slow progress - e.g. can’t design until timing performance specs are known Early stakeholder concurrence enables RAD - e.g. decision on architectural framework or COTS package