1. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 081 Dr. Ricardo Valerdi Massachusetts Institute of Technology rvalerdi@mit.edu Workshop on Estimating and Managing Through-Life-Costs Bath University November 12, 2008 Parametric Cost Modeling for Systems Engineering

2. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 2 Roadmap (1)Systems Engineering fundamentals; (2)Explanation of COSYSMO size and cost drivers; (3)Limitations; (4)Recent developments;

3. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 The Delphic Sybil Michelangelo Buonarroti Capella Sistina, Il Vaticano (1508-1512)

4. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 4 FeasibilityPlans/Rqts.DesignDevelop and Test Phases and Milestones Relative Size Range Operational Concept Life Cycle Objectives Life Cycle Architecture Initial Operating Capability x 0.5x 0.25x 4x 2x Cone of Uncertainty Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981.

5. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 Cost Commitment on Projects Blanchard, B., Fabrycky, W., Systems Engineering & Analysis, Prentice Hall, 1998.

6. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 086 Why measure systems engineering? Cost Overrun as a Function of SE Effort NASA Data Honour, E.C., Understanding the Value of Systems Engineering, Proceedings of the INCOSE International Symposium, Toulouse, France, 2004.

7. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 Contract Engineering WBS Based On Standards 1.0 – System/Project 1.1 – Integrated Project Management (IPM) 1.2 – Systems Engineering 1.3 – Prime Mission Product (PMP) 1.3.1 – Subsystem / Configuration Item (CI) 1…n (Specify Names) 1.3.2 – PMP Application Software 1.3.3 – PMP System Software 1.3.4 – PMP Integration, Assembly, Test & Checkout (IATC) 1.3.5 – Operations/Production Support 1.4 – Platform Integration 1.5 – System Test & Evaluation (ST&E) 1.6 – Training 1.7 – Data Management 1.8 – Peculiar Support Equipment 1.9 – Common Support Equipment 1.10 – Operational / Site Activation 1.11 – Industrial Facilities Product-oriented construct, by tailoring MIL- HDBK 881A and ANSI/EIA 632 Six Functions: 1.Systems Engineering 2.Software Engineering 3.Electrical Engineering 4.Mechanical Engineering 5.Support Engineering 6.Project Engineering Management Six Functions: 1.Systems Engineering 2.Software Engineering 3.Electrical Engineering 4.Mechanical Engineering 5.Support Engineering 6.Project Engineering Management

8. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 088 COCOMO II COCOMO is the most widely used, thoroughly documented and calibrated software cost model COCOMO - the COnstructive COst MOdel –COCOMO II is the update to COCOMO 1981 –ongoing research with annual calibrations made available Originally developed by Dr. Barry Boehm and published in 1981 book Software Engineering Economics COCOMO II described in Software Cost Estimation with COCOMO II (Prentice Hall 2000)

9. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 9 COSYSMO Scope Addresses first four phases of the system engineering lifecycle (per ISO/IEC 15288) Considers standard Systems Engineering Work Breakdown Structure tasks (per EIA/ANSI 632) Conceptualize Develop Oper Test & Eval Transition to Operation Operate, Maintain, or Enhance Replace or Dismantle Valerdi, R., The Constructive Systems Engineering Cost Model: Quantifying the Costs of Systems Engineering Effort in Complex Systems, VDM Verlag, 2008

10. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 0810 How is Systems Engineering Defined? Acquisition and Supply –Supply Process –Acquisition Process Technical Management –Planning Process –Assessment Process –Control Process System Design –Requirements Definition Process –Solution Definition Process Product Realization –Implementation Process –Transition to Use Process Technical Evaluation –Systems Analysis Process –Requirements Validation Process –System Verification Process –End Products Validation Process EIA/ANSI 632, Processes for Engineering a System, 1999.

11. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 0811 COSYSMO Size Drivers Effort Multipliers Effort Calibration # Requirements # Interfaces # Scenarios # Algorithms + 3 Volatility Factors - Application factors -8 factors - Team factors -6 factors - Schedule driver WBS guided by EIA/ANSI 632 COSYSMO Operational Concept

12. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 0812 Where: PM NS = effort in Person Months (Nominal Schedule) A = calibration constant derived from historical project data k = {REQ, IF, ALG, SCN} w x = weight for easy, nominal, or difficult size driver = quantity of k size driver E = represents diseconomy of scale EM = effort multiplier for the j th cost driver. The geometric product results in an overall effort adjustment factor to the nominal effort. Model Form

13. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 COSYSMO Data Sources BoeingIntegrated Defense Systems (Seal Beach, CA) RaytheonIntelligence & Information Systems (Garland, TX) Northrop GrummanMission Systems (Redondo Beach, CA) Lockheed MartinTransportation & Security Solutions (Rockville, MD) Integrated Systems & Solutions (Valley Forge, PA) Systems Integration (Owego, NY) Aeronautics (Marietta, GA) Maritime Systems & Sensors (Manassas, VA; Baltimore, MD; Syracuse, NY) General DynamicsMaritime Digital Systems/AIS (Pittsfield, MA) Surveillance & Reconnaissance Systems/AIS (Bloomington, MN) BAE Systems National Security Solutions/ISS (San Diego, CA) Information & Electronic Warfare Systems (Nashua, NH) SAIC Army Transformation (Orlando, FL) Integrated Data Solutions & Analysis (McLean, VA) L-3 Communications Greenville, TX

14. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 0814 Academic prototype Commercial Implementations Proprietary Implementations COSYSMO-R SECOST SEEMaP Impact Academic Curricula Intelligence Community Sheppard Mullin, LLC Policy & Contracts Model 10 theses

15. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 0815 4 Size Drivers 1. Number of System Requirements 2. Number of System Interfaces 3. Number of System Specific Algorithms 4. Number of Operational Scenarios Weighted by complexity, volatility, and degree of reuse

16. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 0816 Counting Rules Example COSYSMO example for sky, kite, sea, and underwater levels where: Sky level: Build an SE cost model Kite level: Adopt EIA 632 as the WBS and ISO 15288 as the life cycle standard Sea level: Utilize size and cost drivers, definitions, and counting rules Underwater level: Perform statistical analysis of data with software tools and implement model in Excel Source: Cockburn 2001

17. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 0817 14 Cost Drivers 1. Requirements understanding 2. Architecture understanding 3. Level of service requirements 4. Migration complexity 5. Technology Risk 6. Documentation Match to Life Cycle Needs 7. # and Diversity of Installations/Platforms 8. # of Recursive Levels in the Design Application Factors (8)

18. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 0818 14 Cost Drivers (cont.) 1. Stakeholder team cohesion 2. Personnel/team capability 3. Personnel experience/continuity 4. Process capability 5. Multisite coordination 6. Tool support Team Factors (6)

19. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 0819 Cost Driver Rating Scales Very LowLowNominalHighVery High Extra HighEMR Requirements Understanding1.871.371.000.770.60 3.12 Architecture Understanding1.641.281.000.810.65 2.52 Level of Service Requirements0.620.791.001.361.85 2.98 Migration Complexity 1.001.251.551.93 Technology Risk0.670.821.001.321.75 2.61 Documentation0.780.881.001.131.28 1.64 # and diversity of installations/platforms 1.001.231.521.87 # of recursive levels in the design0.760.871.001.211.47 1.93 Stakeholder team cohesion1.501.221.000.810.65 2.31 Personnel/team capability1.501.221.000.810.65 2.31 Personnel experience/continuity1.481.221.000.820.67 2.21 Process capability1.471.211.000.880.770.682.16 Multisite coordination1.391.181.000.900.800.721.93 Tool support1.391.181.000.850.72 1.93

20. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 0820

21. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 ISO/IEC 15288 Conceptualize Develop Transition to Operation Acquisition & Supply Technical Management System Design Product Realization Technical Evaluation Operational Test & Evaluation ANSI/EIA 632

22. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 0822 Limitations of the model 1. Mostly qualitative drivers 2. Variance of Delphi responses 3. Small sample size 4. Aerospace-heavy 5. Calibration is biased by successful projects because successful projects share data, bad ones dont 6. Model will not work outside of calibrated range 7. A fool with a tool is still a fool

23. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 0823

24. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 Reuse Continuum Modified vs. New Threshold Modified Adopted New 1.0 0 Deleted Managed Reuse weight 0.65 0.51 0.43 0.15

25. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 Lessons Learned from Development Lesson #1: Scope of the model Lesson #2: Types of projects needed for data collection effort Lesson #3: Size drivers Lesson #4: Effort Multiplier Lesson #5: Systems Engineering hours across life cycle stages Lesson #6: Data collection form Lesson #7: Definition Lesson #8: Significance vs. data availability Lesson #9: Influence of data on the drivers and statistical significance Lesson #10: Data safeguarding procedure Lesson #11: Buy-in from constituents

26. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 Lessons Learned from Validation Lesson #1: Skills Needed to use COSYSMO Provide a list of assumptions/prerequisites for model use as well as the appropriate training/resources for COSYSMO understanding Lesson #2: Model Usability Understanding usability will lead to more reliable inputs to the model especially at the early phases of the life cycle where there is little project information available Lesson #3: Model Adoption Providing organizations with a sequential process driven by implementation experience will facilitate the adoption of COSYSMO Lesson #4: Accounting for Reuse Providing a way to account for reuse in systems engineering is essential for improving the accuracy of the model

27. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 Rechtins Systems Architecting Heuristics COSYSMO Model COSYSMO Tool Model Development Heuristics Model Calibration Heuristics Model Usage Heuristics Cost Estimation Heuristics inspiredimplemented in experience lead to confirmed Experiential Closed Loop

28. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 Estimating-related Heuristic Heuristic #30: Models are optimistic. Heuristic #31: People are generally optimistic. Koehler, D. J., Harvey, N. (1997). Confidence judgments by actors and observers. Journal of Behavioral Decision Making. 10, 221- 242.

29. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 08 % time added for architecture and risk resolution % time added to overall schedule % of project schedule devoted to initial architecture and risk resolution Added schedule devoted to rework (COCOMO II RESL factor) Total % added schedule Sweet Spot 0 20 40 60 80 100 01020304050 10,000 KSLOC 100 KSLOC 10 KSLOC

30. Bath University Workshop on Estimating and Managing Through-Life-Costs Nov 0830 Contact Ricardo Valerdi MIT Lean Advancement Initiative rvalerdi@mit.edu (617) 253-8583 www.valerdi.com/cosysmo