1. Tradespace, Affordability, and COCOMO III Barry Boehm, USC CSSE Annual Research Review 2014 April 30, 2014 03-19-20141

2. Tradespace, Affordability and COCOMO III Critical nature of the ilities – Major source of project overruns, failures – Significant source of stakeholder value conflicts – Poorly defined, understood – Underemphasized in project management SERC Foundations efforts – Stakeholder value-based, means-ends hierarchy – Formal analysis of ility definitions and relations – Architecture strategy synergies and conflicts – Affordability means-ends hierarchy Relations to COCOMO III 03-19-20142

3. Importance of ility Tradeoffs Major source of DoD system overruns System ilities have systemwide impact – System elements generally just have local impact ilities often exhibit asymptotic behavior – Watch out for the knee of the curve Best architecture is a discontinuous function of ility level – “Build it quickly, tune or fix it later” highly risky – Large system example below 03-19-20143

4. Example of Current Practice “The system shall have a Mean Time Between Failures of 10,000 hours” What is a “failure?” – 10,000 hours on liveness – But several dropped or garbled messages per hour? What is the operational context? – Base operations? Field operations? Conflict operations? Most management practices focused on functions – Requirements, design reviews; traceability matrices; work breakdown structures; data item descriptions; earned value management What are the effects on other –ilities? – Cost, schedule, performance, maintainability? 03-19-20144

5. 5 USC: COCOMO II-Based Tradeoff Analysis Better, Cheaper, Faster: Pick Any Two? -- Cost/Schedule/RELY: “pick any two” points (RELY, MTBF (hours)) For 100-KSLOC set of features Can “pick all three” with 77-KSLOC set of features

6. Tradespace, Affordability and COCOMO III Critical nature of the ilities – Major source of project overruns, failures – Significant source of stakeholder value conflicts – Poorly defined, understood – Underemphasized in project management SERC Foundations efforts – Stakeholder value-based, means-ends hierarchy – Formal analysis of ility definitions and relations – Architecture strategy synergies and conflicts – Affordability means-ends hierarchy Relations to COCOMO III 03-19-20146

7. SERC Value-Based ilities Hierarchy Based on ISO/IEC 9126, 25030; JCIDS; previous SERC research Individual ilities – Mission Effectiveness: Speed, Physical Capability, Cyber Capability, Usability, Accuracy, Impact, Endurability, Maneuverability, Scalability, Versatility – Resource Utilization: Cost, Duration, Personnel, Scarce Quantities (capacity, weight, energy, …); Manufacturability, Sustainability – Protection: Security, Safety – Robustness: Reliability, Availablilty, Maintainability, Survivability – Flexibility: Modifiability, Tailorability, Adaptability – Composability: Interoperability, Openness, Service-Orientation Composite ilities – Comprehensiveness/Suitability: all of the above – Dependability: Mission Effectiveness, Protection, Robustness – Resilience: Protection, Robustness, Flexibility – Affordability: Mission Effectiveness, Resource Utilization 03-19-20147

8. 8 Legacy System Repurposing Eliminate Tasks Eliminate Scrap, Rework Staffing, Incentivizing, Teambuilding Kaizen (continuous improvement) Work and Oversight Streamlining Collaboration Technology Early Risk and Defect Elimination Modularity Around Sources of Change Incremental, Evolutionary Development Risk-Based Prototyping Satisficing vs. Optimizing Performance Value-Based Capability Prioritization Composable Components,Services, COTS Affordability Improvements and Tradeoffs Get the Best from People Make Tasks More Efficient Simplify Products (KISS) Reuse Components Facilities, Support Services Tools and Automation Lean and Agile Methods Evidence-Based Decision Gates Domain Engineering and Architecture Task Automation Model-Based Product Generation Value-Based, Agile Process Maturity Means-Ends Framework: Affordability Reduce Operations, Support Costs Streamline Supply Chain Design for Maintainability, Evolvability Automate Operations Elements Anticipate, Prepare for Change Value- and Architecture-Based Tradeoffs and Balancing

9. Architecture Strategy Synergy-Conflict Matrix 03-19-20149

10. Software Development Cost vs. Quality 0.8 Very Low NominalHigh Very High 0.9 1.0 1.1 1.2 1.3 1.4 1.10 1.0 0.92 1.26 0.82 Relative Cost to Develop COCOMO II RELY Rating MTBF (hours) 1 10 300 10,000 300,000 03-19-201410

11. Software Ownership Cost vs. Quality 0.8 Very Low NominalHigh Very High 0.9 1.0 1.1 1.2 1.3 1.4 1.10 0.92 1.26 0.82 Relative Cost to Develop, Maintain, Own and Operate COCOMO II RELY Rating 1.23 1.10 0.99 1.07 1.11 1.05 70% Maint. 1.07 1.20 0.76 0.69 VL = 2.55 L = 1.52 Operational-defect cost at Nominal dependability = Software life cycle cost Operational - defect cost = 0 MTBF (hours) 1 10 300 10,000 300,000 03-19-201411

12. 03-19-201412

13. Tradespace, Affordability and COCOMO III Critical nature of the ilities – Major source of project overruns, failures – Significant source of stakeholder value conflicts – Poorly defined, understood – Underemphasized in project management SERC Foundations efforts – Stakeholder value-based, means-ends hierarchy – Formal analysis of ility definitions and relations – Architecture strategy synergies and conflicts – Affordability means-ends hierarchy Relations to COCOMO III 03-19-201413

14. 03-19-201414 Legacy System Repurposing Eliminate Tasks Eliminate Scrap, Rework Staffing, Incentivizing, Teambuilding Kaizen (continuous improvement) Work and Oversight Streamlining Collaboration Technology Early Risk and Defect Elimination Modularity Around Sources of Change Incremental, Evolutionary Development Risk-Based Prototyping Satisficing vs. Optimizing Performance Value-Based Capability Prioritization Composable Components,Services, COTS Affordability Improvements and Tradeoffs Get the Best from People Make Tasks More Efficient Simplify Products (KISS) Reuse Components Facilities, Support Services Tools and Automation Lean and Agile Methods Evidence-Based Decision Gates Domain Engineering and Architecture Task Automation Model-Based Product Generation Value-Based, Agile Process Maturity Affordability and Tradespace Framework Reduce Operations, Support Costs Streamline Supply Chain Design for Maintainability, Evolvability Automate Operations Elements Anticipate, Prepare for Change Value- and Architecture-Based Tradeoffs and Balancing

15. 15 Costing Insights: COCOMO II Productivity Ranges Productivity Range 11.21.41.61.822.22.4 Product Complexity (CPLX) Analyst Capability (ACAP) Programmer Capability (PCAP) Time Constraint (TIME) Personnel Continuity (PCON) Required Software Reliability (RELY) Documentation Match to Life Cycle Needs (DOCU) Multi-Site Development (SITE) Applications Experience (AEXP) Platform Volatility (PVOL) Use of Software Tools (TOOL) Storage Constraint (STOR) Process Maturity (PMAT) Language and Tools Experience (LTEX) Required Development Schedule (SCED) Data Base Size (DATA) Platform Experience (PEXP) Architecture and Risk Resolution (RESL) Precedentedness (PREC) Develop for Reuse (RUSE) Team Cohesion (TEAM) Development Flexibility (FLEX) Scale Factor Ranges: 10, 100, 1000 KSLOC 03-19-2014 Staffing Teambuilding Continuous Improvement

16. COSYSMO Sys Engr Cost Drivers 1603-19-2014 Teambuilding Staffing Continuous Improvement

17. 03-19-201417 Legacy System Repurposing Eliminate Tasks Eliminate Scrap, Rework Staffing, Incentivizing, Teambuilding Kaizen (continuous improvement) Work and Oversight Streamlining Collaboration Technology Early Risk and Defect Elimination Modularity Around Sources of Change Incremental, Evolutionary Development Risk-Based Prototyping Satisficing vs. Optimizing Performance Value-Based Capability Prioritization Composable Components,Services, COTS Affordability Improvements and Tradeoffs Get the Best from People Make Tasks More Efficient Simplify Products (KISS) Reuse Components Facilities, Support Services Tools and Automation Lean and Agile Methods Evidence-Based Decision Gates Domain Engineering and Architecture Task Automation Model-Based Product Generation Value-Based, Agile Process Maturity Tradespace and Affordability Framework Reduce Operations, Support Costs Streamline Supply Chain Design for Maintainability, Evolvability Automate Operations Elements Anticipate, Prepare for Change Value- and Architecture-Based Tradeoffs and Balancing

18. 03-19-2014 18 Value-Based Testing: Empirical Data and ROI — LiGuo Huang, ISESE 2005 (a) (b)

19. Value-Neutral Defect Fixing Is Even Worse % of Value for Correct Customer Billing Customer Type 100 80 60 40 20 51015 Automated test generation tool - all tests have equal value Value-neutral defect fixing: Quickly reduce # of defects Pareto 80-20 Business Value 03-19-2014 19

20. Product Line Engineering and Management: NPS 02-25-1420

21. 21 SysML Building Blocks for Cost Modeling GaTech-USC Work in RT46 Phase 2 (Oct-Dec 2013) Implemented reusable SysML building blocks – Based on SoS/COSYSMO SE cost (effort) modeling work by Lane, Valerdi, Boehm, et al. Successfully applied building blocks to healthcare SoS case study from [Lane 2009] Provides key step towards affordability trade studies involving diverse “-ilities” ( see MIM slides )

22. 22 Healthcare SoS Case Study [Lane 2009] Implemented Using SysML Building Blocks: Selected SysML Diagrams

23. Tradespace, Affordability and COCOMO III Critical nature of the ilities – Major source of project overruns, failures – Significant source of stakeholder value conflicts – Poorly defined, understood – Underemphasized in project management SERC Foundations efforts – Stakeholder value-based, means-ends hierarchy – Formal analysis of ility definitions and relations – Architecture strategy synergies and conflicts – Affordability means-ends hierarchy Relations to COCOMO III 03-19-201423

24. COCOMO II Data by 5-Year Periods 1/13/2014© USC-CSSE24

25. COCOMO II Data: Productivity Trends 1/13/2014© USC-CSSE25

26. COCOMO II Data: Process Maturity Trends 1/13/2014© USC-CSSE26

27. SRDR Data: Productivity vs. Size, CMM Level 1/13/2014© USC-CSSE27

28. Trends Confounded by Missing Variables Incremental Development Productivity Decline 12/03/2013Copyright © USC-CSSE28

29. COCOMO II Status and COCOMO III Plans Baseline COCOMO II calibrated to 161 project data points –Pred (20%; 30%) = (63%; 70%) general; (75%;80%) local Added 149 data points 2000-2009 –Pred (30%) < 40% general; some sources ~70-80% local –Some improvement with added variables (year; domain; agility) But some data-source mismatches unexplainable Vu Nguyen analysis of full dataset suggests further adjusting –Rating scales for experience, tools, reliability Proposed approach for COCOMO III –Explore models for unexplained existing sources or drop –Try added variables for mostly-general fit to existing data –Obtain more data to validate results 1/13/2014© USC-CSSE29