1 Reggie Cole Lockheed Martin Senior Fellow Garry Roedler Lockheed Martin Fellow October 23, 2013 COSYSMO Extension as a Proxy Systems Cost Estimation

2 Workshop Agenda 1:00 – 2:00COSYSMO as a Proxy for System Cost Estimation 2:00 – 2:50Deep Dive on the Proxy/Bias Function 2:50 – 3:10Break 3:10 – 4:30Discussion of the Three Key Use Cases 4:30 – 5:00Wrap-Up and Recommendations

3 Workshop Objectives Validation of the Approach – Review the basic approach and get a consensus on its overall validity Improvement of the Approach – Recommendations on how the approach might be improved Decide on Next Steps for the Community – How do we move forward with the approach?

4

5 COSYSMO as a Proxy Estimator for System Cost Need for a Top-Down System Cost Model – Better Buying Power 2.0 Mandate Perform ongoing should-cost analysis – Better Buying Power 2.0 Implication Perform design-to-cost analysis and design for affordability – There is Currently a Gap in Tools Primarily in early-stage analysis – when directions can still be changed without significant repercussions Extending COSYSMO for System Costing – Evaluate the Viability of Extending COSYSMO COSYSMO seems to have the right parameters

6 Cost Modeling Needs Change Over Time in Terms of Speed and Accuracy Problem-Space Description Cost Estimate ± 25% High-Level Solution Description Cost Estimate ± 10% Detailed Solution Description Cost Estimate ± 5% High-Level Solution Assumptions Cost Estimate ± 20% Increasing Effort and Cost-Modeling Expertise Increasingly Refined Information About the Solution Increasingly Refined Cost Estimate Increasingly Refined Solution We Have a Good Selection of Tools for Late-Stage Cost Modeling We Have Gaps in Early-Stage Cost Modeling

7 SE Effort as a Proxy Measure of Overall System Size and Complexity Proxy Measures – Proxy measures are used when you cannot directly measure what you want to measure – and when an indirect measure provides sufficient insight – Proxy measures are often used in clinical studies since direct measurement is often infeasible or can even alter the outcome – It is not always possible to directly measure what you want to measure – or directly estimate what you want to estimate System Engineering Effort is a Proxy Measure for System Cost – There is strong evidence for the link between systems engineering effort and program cost – dating back to a NASA study in the 1980s – The optimal relationship between systems engineering effort and overall program cost is 10% - 15% – Industry has long used a parametric relationship between software cost and systems engineering cost for software-intensive systems – Systems engineering effort can be an effective proxy measure for overall system cost

8 COSYSMO 2.0 Model Parameters Provide a Rich Assessment of System Size and Complexity Number of System Requirements Number of Major System Interfaces Number of Critical Algorithms Number of Operational Scenarios Size Drivers Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Level of Documentation Required Diversity of Installed Platforms Level of Design Recursion Stakeholder Team Cohesion Personnel / Team Capability Personnel Experience / Continuity Process Capability Multisite Coordination Level of Tool Support Cost Drivers Managed Elements Adopted Elements Deleted Elements Modified Elements New Elements Reuse Factors Initial Estimate of System Size Scaled Estimate of System Size Consolidated Cost Driver Factor Estimate of Systems Engineering Effort…Also a Biased Proxy Estimator for System Scope…And System Cost

9 Relationship Between SE Effort and Total Effort NASA data supports a 10%-15% optimal allocation of systems engineering effort as a portion of overall program effort W. Gruhl, Lessons Learned, Cost/Schedule Assessment Guide,” Internal Presentation, NASA Comptroller’s Office, 1992 E. Honour, “Understanding the Value of Systems Engineering,” INCOSE, 2004 INCOSE study on the value of systems engineering also supports a 10%-15% optimal allocation of systems engineering as a portion of overall program effort

10 Putting It All Together  Size Drivers (Problem Space)  Customer Requirements  System Interfaces  Major Algorithms  Operational Scenarios  Complexity Drivers (Problem/Solution)  Requirements Understanding  Architecture Understanding  Level of Service Requirements  Migration Complexity  Technology Risk  Documentation Needs  Installations/Platform Diversity  Levels of Recursion in the Design  Stakeholder Team Cohesion  Personnel/Team Capability  Personnel Experience/Continuity  Process Capability  Multisite Coordination  Tool Support  Reuse Factors (Solution Space)  New  Modified  Deleted  Adopted  Managed SE Effort is an estimator for total system cost…but it is a biased estimator Estimator Bias Function is Based on the Well- Established Relationship Between SE Effort and Overall Program Effort Estimation of Total System Cost

11 Adjusting Our View of COSYSMO Parameters Our view of the size drivers can be preserved – their context doesn’t change under COSYSMO extension Our view of the cost drivers needs to be broadened to include all aspects of the system, not just systems engineering Our view of reuse requires the most extreme adjustment – we are not just talking about systems engineering artifacts

12 Expanding Our View of Cost Drivers Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Level of Documentation Required Diversity of Installed Platforms Level of Design Recursion Stakeholder Team Cohesion Personnel / Team Capability Personnel Experience / Continuity Process Capability Multisite Coordination Level of Tool Support Cost Drivers Precedented systems and unprecedented systems are fundamentally different Need to consider the entire team – including subcontractors Need to consider the all processes and tools System modification and reuse have a significant effect on some cost drivers

13 Expanding Our View of Reuse Factors Managed Elements Adopted Elements Deleted Elements Modified Elements New Elements Reuse Factors New Elements – These are elements that need to be engineered and developed. Just reusing systems engineering artifacts is not sufficient. Modified Elements – These are elements that offer some form of reuse. Enhanced COTS or reusable components that need modification fall into this category. Adopted Elements – These are elements that offer significant reuse with minimal modification and do not require full retesting. COTS typically falls into this category. Managed Elements – These are elements that are already in the system and require minimal regression testing. A previously deployed element falls into this category. For Requirements… Think Functional Components For Interfaces… Think Connection Points For Algorithms… Think Functional Components For Scenarios… Think Implications to Both

14 Example Consider the case of large C2 system. Initially developed 20 years ago, the system was unprecedented. Twenty years later, a replacement system is needed. While the initial development was unprecedented, the replacement system is not, which drives down the size drivers (through reuse) and cost drivers.  Case 1 – Unprecedented System  Case 2 – Developed Replacement System  Case 3 – COST/GOTS-Based Replacement System Similar Size Drivers – But Significantly Different Cost Drivers Similar Cost Drivers – But Significantly Different Size Drivers

15 Part 1 Wrap-Up The Approach Based on Well-Established Approaches – COSYSMO provides the basis for estimation of systems engineering effort – and a biased proxy estimator for overall system cost – There is a well-established relationship between systems engineering effort and overall effort used to de-bias the COSYSMO-modeled effort The Approach Can Improve System Cost Modeling – It occupies an important niche – fully parametric system cost modeling in the early stages of system definition – It can serve as a powerful affordability analysis tool – supporting rapid-turnaround analysis of alternatives – But…it is not a replacement for existing models

16

17 Context of the Bias Function Deeper Discussion of the Proxy/Bias Function is Necessary – As Well as a Technique for Generating Cumulative Distribution of System Costs

18 The Bias Function VariableTypeDescription COSYSMO Calibration FactorDeterministic Scalar ValueOrganization-specific calibration factor Effort Conversion FactorTriangular Distributed Random VariableThree-point estimate of factor to convert SE effort to total program effort (nominally 0.08, 0.12 and 0.16) SE EffortTriangular Distributed Random VariableThree-point estimate for SE effort, generated using COSYSMO Labor RateTriangular Distributed Random VariableThree-point estimate for composite labor rate Material CostsTriangular Distributed Random VariableThree-point estimate for material costs Travel CostsTriangular Distributed Random VariableThree-point estimate for travel costs We are going to discuss each of these factors!

19 SE Effort (Effort SE )  Size Drivers (Problem Space)  Customer Requirements  System Interfaces  Major Algorithms  Operational Scenarios  Complexity Drivers (Problem/Solution)  Requirements Understanding  Architecture Understanding  Level of Service Requirements  Migration Complexity  Technology Risk  Documentation Needs  Installations/Platform Diversity  Levels of Recursion in the Design  Stakeholder Team Cohesion  Personnel/Team Capability  Personnel Experience/Continuity  Process Capability  Multisite Coordination  Tool Support  Reuse Factors (Solution Space)  New  Modified  Deleted  Adopted  Managed Three different COSYMO scenarios – optimistic, expected & pessimistic – provide the basis for a sampling distribution. COSYSMO provides a Proxy Estimate of the system cost. We will not try to de-bias it right now….that is the next step. Since we want to perform Monte Carlo simulation of costs, we would like to generate a distribution of the proxy costs. Optimistic Expected Pessimistic Triangular Distribution Beta Pert Distribution COSYSMO Estimate of Hours Becomes the Parameters for Either Triangular or Beta Pert Distribution

20 Effort Conversion Factor (F Conv ) Studies provide some insight into what the value should be This is probably the most important factor in the bias function! Heuristic approaches for determining SE costs for software intensive systems are also consistent with these studies All indications point to a range of 0.08 to 0.16 for F Conv And this range is consistent with all the data we’ve collected to date…for relatively healthy programs And it provides the basis for our random variable parameters Triangular Distribution Beta Pert Distribution Optimistic Expected Pessimistic

21 Other Stochastic “Adder” Factors Labor Costs – Labor costs vary – especially in early stages – and needs to be treated as a random variable – Any number of distributions would probably be OK – Beta Pert would be a good default – If hours are the item of interest rather than cost, this factor can be omitted Material Costs – Material costs vary – especially in early stages – and needs to be treated as a random variable – Any number of distributions would probably be OK – Beta Pert would be a good default but there is at least one study that looks at using a normal distribution – If hours are the item of interest rather than cost, this factor can be omitted Travel Costs – Travel costs vary – especially in early stages – and needs to be treated as a random variable – Any number of distributions would probably be OK – Beta Pert would be a good default – If hours are the item of interest rather than cost, this factor can be omitted Other – Any number of other “stochastic adders” can be treated similarly

22 Other “Deterministic Adders” Some Factors Are Well Known – To the Point They Can Be Considered Deterministic – They are often set and apply across programs – Examples include: G&A Costs Other Direct costs Management Reserve Fee

23 COSYSMO Calibration Factor Local Calibration is Important – Keeps us from over-tuning the Effort Conversion Factor This Also Serves as a Type of Organizational Efficiency Factor – Can vary across organizations within an enterprise It is a Simple Scalar Factor – Optimally, it should be 1.0

24 Calibration for Proxy Estimation It is Not Necessary to Revalidate or Recalibrate COSYSMO – The strength of this approach is that it rests on the COSYSMO foundation It is Necessary to Validate and Calibrate the Bias Function – Important to validate the relationship between system costs and systems engineering costs – Important to calibrate the COSYSMO Calibration Function

25 Our Current Approach for Validation Use a Few Long-Running Programs – They tend to have good data collection and good process discipline – so the data is reliable Treat Each Major Release as a Separate Entity – That really allows us to dig into reuse between releases – It is necessary to collect data on reuse – we found that to be a little challenging – Use some releases for validation and others for calibration

26 Run Monte Carlo Simulations and Generate Cumulative Distribution of Costs Define Parameters for Remaining Factors in Bias Function Construct the COSYSMO Scenarios Revisiting the Overall Approach 1 2 3

27 Part 2 Wrap-Up The Basic Bias Function – Concerns with the basic bias function – Suggested improvements on the basic bias function The Approach for Validation and Calibration – Concerns with the basic approach – Suggested improvements for validation and calibration

28

29 The Key Use Cases Should-Cost Analysis – Establishing a should-cost for which cost estimates from bidders can be evaluated – Establishing a DTC target for performing DTC analysis Design-to-Cost Analysis – Performing cost vs. capability trades as way to provide more affordable solutions Analysis of Alternatives – Evaluating alternative solution strategies – It’s not just about the problem space – the solution space can be evaluated too

30 Should-Cost Analysis Goal is to Establish a Target Cost – Can also be a cost range – Usually performed very early in the lifecycle Approach – Given a requirements baseline, use the extended COSYSMO approach to estimate the cost – Use “plug” numbers for adders e.g., labor, material, fee, etc.

31 COSYSMO Setup Optimistic Expected Pessimistic Assume a mature supplier, experienced in the domain, minimal scope creep, and cooperative stakeholders Assume an average supplier, with some experience in the domain, average scope creep, and generally cooperative stakeholders Assume a new supplier who will have some challenges, a fair amount of baseline volatility, and stakeholders who need to be corralled

32 Cost Analysis Approach & Results Requirements Baseline Architecture Baseline Optimistic Expected Pessimistic Requirements Interfaces Algorithms Scenarios Use a “Plug” Number for Adders  Anticipated Distribution of Labor Rates  Anticipated Distribution of Material Costs  Anticipated Travel Costs  Anticipated Supplier Fees Bias Function Risky Range Target Cost Target Reserve

33 Design-to-Cost Analysis Goal – The goal is, given a target cost, design a solution to meet the target cost Approach – Given a requirements baseline, identify and prioritize capabilities – Use the extended COSYSMO approach to estimate the cost for each capability – Evaluate the “cost for capability” against the capability priority

34 Capability Decomposition Requirements Baseline Architecture Baseline Problem: TELECOM Operations Support System Major Upgrade, Budget Limited to $40M Problem: TELECOM Operations Support System Major Upgrade, Budget Limited to $40M 1 – Service Provisioning 2 – Service Order Orchestration 3 – Pre-Provisioning Support 4 – Service Order Validation 5 – Service Activation 6 – Network Management 7 – Dashboards for Awareness & Reporting 8 – Service Catalog Management 9 – Service-to-Subscriber Mapping 10 – Auto-Discovery 11 – Service Reconciliation 12 – Billing - Service Order Creation 13 – Billing - Trouble Ticketing 14– Service Coverage Mapping 15– Resource Management Major Capabilities Evaluate Each Capability with Respect to Cost and Enterprise Utility to Determine Best-Value Baseline

35 Mission Utility Analysis of Capabilities Operational Burden Very High High Mod High Med Mod Low Low Mod LowMedMod HighHighVery High Operational Tempo 1 – Service Provisioning 2 – Service Order Orchestration 3 – Pre-Provisioning Support 4 – Service Order Validation 5 – Service Activation 6 – Network Management 7 – Dashboards for Awareness & Reporting 8 – Service Catalog Management 9 – Service-to-Subscriber Mapping 10 – Auto-Discovery 11 – Service Reconciliation 12 – Billing - Service Order Creation 13 – Billing - Trouble Ticketing 14– Service Coverage Mapping 15– Resource Management Operational Tempo is a combination of the frequency with which the capability is used operationally and its criticality to the overall mission. Operational Burden is a combination of the skill level required for the capability and the time it takes to perform.

36 Cost Analysis Approach Requirements Baseline Architecture Baseline Requirements Interfaces Algorithms Scenarios Optimistic Expected Pessimistic Three COSYSMO Scenarios for Each Capability Use a Common Number for Adders  Anticipated Distribution of Labor Rates  Anticipated Distribution of Material Costs  Anticipated Travel Costs  Anticipated Supplier Fees Bias Function Take the 80% Confidence Cost as the Capability Cost A Cost Curve is Produced for Each Capability

37 Simplified Cost Analysis Approach Expected Requirements Baseline Architecture Baseline Requirements Interfaces Algorithms Scenarios A Single COSYSMO Scenario for Each Capability Use a Common Number for Adders  Anticipated Distribution of Labor Rates  Anticipated Distribution of Material Costs  Anticipated Travel Costs  Anticipated Supplier Fees Bias Function Take the 80% Confidence Cost as the Capability Cost A Cost Curve is Produced for Each Capability A Simpler Approach is Slightly Less Robust…Bust Still Just as Valid

38 Analysis Results Capabilities Sorted By Cost Total Cost = $81.3M Colors Map Mission Utility Priorities  RED = High  YELLOW = Medium  Green = Low Capabilities Sorted By Mission Utility $34.4M $39.9M $64.1M $81.3M (Cost for Priority 1 Capabilities Only) (Capabilities at DTC Target) (Cost for Priority 1&2 Capabilities) (Cost for All Capabilities)

39 Analysis of Alternatives Goal – The goal is, given a requirements baseline, determine the most affordable solution that meets requirements Approach – Given a requirements baseline, identify possible solution alternatives – Use the extended COSYSMO approach to estimate the cost for each capability – Evaluate the cost to find the most affordable alternative that meets all requirements

40 Case Study Overview 20-Year Old C2 System – The system was unprecedented at the time – Twenty years later, a replacement system is needed due to obsolescence and needed changes Alternative Solutions – Full Replacement System Develop and deploy a new replacement system – COTS/GOTS/NDI-Based Replacement System Use a combination existing non-obsolete components and COTS components Modify components as necessary to meet requirements

41 The Two Key Alternatives Newly Developed System Refurbished System Using Modified NDI  Ground-Up Development of System – Requirements Refinement, Architecture, Detailed Design – Soup-to-Nuts  But…It is No Longer an Unprecedented System – So Requirements/Architecture Understanding, etc. Are High  Ground-Up Development of System – Requirements Refinement, Architecture, Detailed Design – Soup-to-Nuts  But…It is No Longer an Unprecedented System – So Requirements/Architecture Understanding, etc. Are High  Use of NDI is Maximized – Additional Components Developed as Necessary to Meet Requirements  Reuse Considerations Drive This Alternative  Use of NDI is Maximized – Additional Components Developed as Necessary to Meet Requirements  Reuse Considerations Drive This Alternative

42 COSYSMO Size and Cost Drivers Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Level of Documentation Required Diversity of Installed Platforms Level of Design Recursion Stakeholder Team Cohesion Personnel / Team Capability Personnel Experience / Continuity Process Capability Multisite Coordination Level of Tool Support Cost Drivers System modification and reuse have an effect on some cost drivers  Size Drivers for the Two Alternatives Are Largely the Same  Cost Drivers for the Two Alternatives Are Very Similar – With a Few Notable Exceptions  Size Drivers for the Two Alternatives Are Largely the Same  Cost Drivers for the Two Alternatives Are Very Similar – With a Few Notable Exceptions

43 COSYSMO Reuse Factors Managed Elements Adopted Elements Deleted Elements Modified Elements New Elements Reuse Factors New Elements – These are elements that need to be engineered and developed. Just reusing systems engineering artifacts is not sufficient. Modified Elements – These are elements that offer some form of reuse. Enhanced COTS or reusable components that need modification fall into this category. Adopted Elements – These are elements that offer significant reuse with minimal modification and do not require full retesting. COTS typically falls into this category. Managed Elements – These are elements that are already in the system and require minimal regression testing. A previously deployed element falls into this category. Most Elements Are New for the Development Alternative Elements That Are “Wrapped” Can Be Treated as Adopted for the NDI Alternative Most Elements Are Modified for the NDI Alternative Elements That Stay in Place Without Modification (or Wrapping) Can Be Treated as Managed Deleted Elements – For the NDI Alternative, Some Elements May Need to Be Deleted/Retired Elements That Need to Be Retired Should Be Treated as Deleted Elements

44 Cost Analysis Approach Requirements Baseline Architecture Baseline Requirements Interfaces Algorithms Scenarios Optimistic Expected Pessimistic Three COSYSMO Scenarios for Each Alternative Use a Common Number for Adders  Anticipated Distribution of Labor Rates  Anticipated Distribution of Material Costs  Anticipated Travel Costs  Anticipated Supplier Fees Bias Function Take the 80% Confidence Cost as the Capability Cost A Cost Curve is Produced for Each Capability

45 Part 3 – Wrap-Up There Are Three Very Important Use Cases – Should-Cost Analysis – DTC Analysis – Analysis of Alternatives There Are At Least a Couple More – Change Evaluation for Operational Systems – Scope Creep Monitoring – Probably Others We Haven’t Thought Of Discussion on Use Cases

46 End of Workshop Wrap-Up Validity of the Overall Approach Improvement of the Approach Thoughts on Moving Forward

47