1. 1 Early Systems Costing Prof. Ricardo Valerdi Systems & Industrial Engineering rvalerdi@arizona.edu Dec 15, 2011 INCOSE Brazil Systems Engineering Week INPE/ITA São José dos Campos, Brazil

2. Take-Aways 1.Cost ≈ f(Effort) ≈ f(Size) ≈ f(Complexity) 2.Requirements understanding and “ilities” are the most influential on cost 3.Early systems engineering yields high ROI when done early and well Two case studies: SE estimate with limited information Selection of process improvement initiative 2

3. 3 Cost Commitment on Projects Blanchard, B., Fabrycky, W., Systems Engineering & Analysis, Prentice Hall, 2010.

4. 4 FeasibilityPlans/Rqts.DesignDevelop and Test Phases and Milestones Relative Size Range Operational Concept Life Cycle Objectives Life Cycle Architecture Initial Operating Capability x 0.5x 0.25x 4x 2x Cone of Uncertainty Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981.

5. The Delphic Sybil Michelangelo Buonarroti Capella Sistina, Il Vaticano (1508-1512) 5

6. NASA data (Honour 2002) Systems Engineering Effort vs. Program Cost 6

7. 7 COSYSMO Origins COSYSMO Systems Engineering 1950 Software Cost Modeling 1980 CMMI* 1990 *Capability Maturity Model Integrated (Software Engineering Institute, Carnegie Mellon University) Warfield, J. N., Systems Engineering, United States Department of Commerce PB111801, 1956. Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981. Humphrey, W. Managing the Software Process. Addison-Wesley, 1989. (Humphrey 1989) (Boehm 1981) (Warfield 1956)

8. 8 How is Systems Engineering Defined? Acquisition and Supply –Supply Process –Acquisition Process Technical Management –Planning Process –Assessment Process –Control Process System Design –Requirements Definition Process –Solution Definition Process Product Realization –Implementation Process –Transition to Use Process Technical Evaluation –Systems Analysis Process –Requirements Validation Process –System Verification Process –End Products Validation Process EIA/ANSI 632, Processes for Engineering a System, 1999.

9. COSYSMO Data Sources BoeingIntegrated Defense Systems (Seal Beach, CA) RaytheonIntelligence & Information Systems (Garland, TX) Northrop GrummanMission Systems (Redondo Beach, CA) Lockheed MartinTransportation & Security Solutions (Rockville, MD) Integrated Systems & Solutions (Valley Forge, PA) Systems Integration (Owego, NY) Aeronautics (Marietta, GA) Maritime Systems & Sensors (Manassas, VA; Baltimore, MD; Syracuse, NY) General DynamicsMaritime Digital Systems/AIS (Pittsfield, MA) Surveillance & Reconnaissance Systems/AIS (Bloomington, MN) BAE Systems National Security Solutions/ISS (San Diego, CA) Information & Electronic Warfare Systems (Nashua, NH) SAIC Army Transformation (Orlando, FL) Integrated Data Solutions & Analysis (McLean, VA) L-3 Communications Greenville, TX

10. 10 Modeling Methodology

11. 11 Results of Bayesian Update: Using Prior and Sampling Information

12. 12 COSYSMO Scope Addresses first four phases of the system engineering lifecycle (per ISO/IEC 15288) Considers standard Systems Engineering Work Breakdown Structure tasks (per EIA/ANSI 632) Conceptualize Develop Oper Test & Eval Transition to Operation Operate, Maintain, or Enhance Replace or Dismantle

13. COSYSMO Size Drivers Effort Multipliers Effort (schedule, risk) Calibration # Requirements # Interfaces # Scenarios # Algorithms - Application factors -8 factors - Team factors -6 factors WBS guided by EIA/ANSI 632 COSYSMO Operational Concept 13

14. 14 Software Cost Estimating Relationship Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981. MM = Man months a = calibration constant S = size driver E = scale factor c = cost driver(s) KDSI = thousands of delivered source instructions

15. 15 COSYSMO Model Form Where: PM NS = effort in Person Months (Nominal Schedule) A = calibration constant derived from historical project data k = {REQ, IF, ALG, SCN} w x = weight for “easy”, “nominal”, or “difficult” size driver = quantity of “k” size driver E = represents diseconomies of scale EM = effort multiplier for the j th cost driver. The geometric product results in an overall effort adjustment factor to the nominal effort.

16. 16 Size Drivers vs. Effort Multipliers Size Drivers: Additive, Incremental –Impact of adding a new item inversely proportional to current size 10 -> 11 rqts = 10% increase 100 -> 101 rqts = 1% increase Effort Multipliers: Multiplicative, system-wide –Impact of adding a new item independent of current size 10 rqts + high security = 40% increase 100 rqts + high security = 40% increase

17. 4 Size Drivers 1. Number of System Requirements 2. Number of System Interfaces 3. Number of System Specific Algorithms 4. Number of Operational Scenarios Weighted by complexity and degree of reuse

18. Number of System Requirements This driver represents the number of requirements for the system-of-interest at a specific level of design. The quantity of requirements includes those related to the effort involved in system engineering the system interfaces, system specific algorithms, and operational scenarios. Requirements may be functional, performance, feature, or service-oriented in nature depending on the methodology used for specification. They may also be defined by the customer or contractor. Each requirement may have effort associated with is such as V&V, functional decomposition, functional allocation, etc. System requirements can typically be quantified by counting the number of applicable shalls/wills/shoulds/mays in the system or marketing specification. Note: some work is involved in decomposing requirements so that they may be counted at the appropriate system-of-interest. EasyNominalDifficult - Simple to implement- Familiar- Complex to implement or engineer - Traceable to source- Can be traced to source with some effort - Hard to trace to source - Little requirements overlap - Some overlap- High degree of requirements overlap

19. Counting Rules Example COSYSMO example for sky, kite, sea, and underwater levels where: Sky level: Build an SE cost model Kite level: Adopt EIA 632 as the WBS and ISO 15288 as the life cycle standard Sea level: Utilize size and cost drivers, definitions, and counting rules Underwater level: Perform statistical analysis of data with software tools and implement model in Excel Source: Cockburn 2001

20. EasyNominalDifficult # of System Requirements0.51.005.0 # of Interfaces1.74.39.8 # of Critical Algorithms3.46.518.2 # of Operational Scenarios9.822.847.4 Size Driver Weights

21. 21 UNDERSTANDING FACTORS –Requirements understanding –Architecture understanding –Stakeholder team cohesion –Personnel experience/continuity COMPLEXITY FACTORS –Level of service requirements –Technology Risk –# of Recursive Levels in the Design –Documentation Match to Life Cycle Needs OPERATIONS FACTORS –# and Diversity of Installations/Platforms –Migration complexity PEOPLE FACTORS –Personnel/team capability –Process capability ENVIRONMENT FACTORS –Multisite coordination –Tool support Cost Driver Clusters

22. 22 Stakeholder team cohesion Represents a multi-attribute parameter which includes leadership, shared vision, diversity of stakeholders, approval cycles, group dynamics, IPT framework, team dynamics, trust, and amount of change in responsibilities. It further represents the heterogeneity in stakeholder community of the end users, customers, implementers, and development team. 1.51.221.000.810.65 ViewpointVery LowLowNominalHighVery High Culture  Stakeholders with diverse expertise, task nature, language, culture, infrastructure  Highly heterogeneous stakeholder communities  Heterogeneous stakeholder community  Some similarities in language and culture  Shared project culture  Strong team cohesion and project culture  Multiple similarities in language and expertise  Virtually homogeneous stakeholder communities  Institutionalized project culture Compatibility  Highly conflicting organizational objectives  Converging organizational objectives  Compatible organizational objectives  Clear roles & responsibilities  Strong mutual advantage to collaboration Familiarity and trust  Lack of trust  Willing to collaborate, little experience  Some familiarity and trust  Extensive successful collaboration  Very high level of familiarity and trust

23. Technology Risk The maturity, readiness, and obsolescence of the technology being implemented. Immature or obsolescent technology will require more Systems Engineering effort. ViewpointVery LowLowNominalHighVery High Lack of Maturity Technology proven and widely used throughout industry Proven through actual use and ready for widespread adoption Proven on pilot projects and ready to roll-out for production jobs Ready for pilot useStill in the laboratory Lack of Readiness Mission proven (TRL 9) Concept qualified (TRL 8) Concept has been demonstrated (TRL 7) Proof of concept validated (TRL 5 & 6) Concept defined (TRL 3 & 4) Obsolescen ce - Technology is the state-of-the- practice - Emerging technology could compete in future - Technology is stale - New and better technology is on the horizon in the near-term - Technology is outdated and use should be avoided in new systems - Spare parts supply is scarce

24. Migration complexity This cost driver rates the extent to which the legacy system affects the migration complexity, if any. Legacy system components, databases, workflows, environments, etc., may affect the new system implementation due to new technology introductions, planned upgrades, increased performance, business process reengineering, etc. ViewpointNominalHighVery HighExtra High Legacy contractor Self; legacy system is well documented. Original team largely available Self; original development team not available; most documentation available Different contractor; limited documentation Original contractor out of business; no documentation available Effect of legacy system on new system Everything is new; legacy system is completely replaced or non-existent Migration is restricted to integration only Migration is related to integration and development Migration is related to integration, development, architecture and design

25. 25 Cost Driver Rating Scales Very LowLowNominalHighVery High Extra HighEMR Requirements Understanding1.871.371.000.770.60 3.12 Architecture Understanding1.641.281.000.810.65 2.52 Level of Service Requirements0.620.791.001.361.85 2.98 Migration Complexity 1.001.251.551.93 Technology Risk0.670.821.001.321.75 2.61 Documentation0.780.881.001.131.28 1.64 # and diversity of installations/platforms 1.001.231.521.87 # of recursive levels in the design0.760.871.001.211.47 1.93 Stakeholder team cohesion1.501.221.000.810.65 2.31 Personnel/team capability1.501.221.000.810.65 2.31 Personnel experience/continuity1.481.221.000.820.67 2.21 Process capability1.471.211.000.880.770.682.16 Multisite coordination1.391.181.000.900.800.721.93 Tool support1.391.181.000.850.72 1.93

26. 26 Cost Drivers Ordered by Effort Multiplier Ratio (EMR)

27. ISO/IEC 15288 Conceptualize Develop Transition to Operation Acquisition & Supply Technical Management System Design Product Realization Technical Evaluation Operational Test & Evaluation ANSI/EIA 632 Effort Profiling

28. Benefits of Local Calibration Before local calibration After local calibration

29. 29 Prediction Accuracy PRED(30) PRED(25) PRED(20) PRED(30) = 100% PRED(25) = 57%

31. Human Systems Integration in the Air Force 31 Joint HSI Personnel Integration Working Group. “Human System Integration (HSI) Activity in DoD Weapon Acquisition Programs: Part II Program Coverage and Return on Investment.” April 16, 2007. U.S. Air Force Scientific Advisory Board. “Report on Human Systems Integration in Air Force Weapon Systems Development and Acquisition.” July, 2004.

32. HSI Early in F119 Development AF Acquisition Community-led requirements definition studies 40% fewer parts than previous engines 32

33. Parametric Cost Estimation 33

34. Parametric Cost Estimation 34 Requirements Interfaces Algorithms Operational Scenarios Tool Support Architecture Understanding Technology Risk # and diversity of installations/platforms Personnel/team capability

35. Illustrative Example 35 100 Difficult Requirements 200 Medium Requirements 200 Easy Requirements Nominal System SE Effort = 300 Person-months

36. Illustrative Example 36 110 Difficult Requirements 210 Medium Requirements 200 Easy Requirements SE Effort = 327 Person-months 5 th /95 th Percentile Manpower Tools reduction Interactive Technical Manuals 20 minute component replacement Size impact Nominal System

37. Illustrative Example 37 110 Difficult Requirements 210 Medium Requirements 200 Easy Requirements SE Effort = 368 Person-months effect of effort multipliers: High Level of service requirements High HSI Tools support Cost impact Nominal System

38. ROI Calculation Cost: 68 Person Months @ $10,000/person month – $680,000 of Human Systems Integration Benefit: assuming 30 year life cycle –Manpower (one less maintainer): $200,000 * 30 = $6,000,000 –Human factors (40% fewer parts):$300,000 * 30 = $9,000,000 –Safety (one less repair accident): = $50,000 –Survivability (one less engine failure): = $500,000 = $15,550,000 Return on Investment 38 Note: time value of money excluded

39. Relative ROI Activity% ROI CMMI®173% ISO 9001229% Systems Engineering1,700% Human Systems Integration2,186% Software inspections3,272% Personal Software Process4,133% 39 Brodman, J. G., Johnson, D. L., Return on Investment from Software Process Improvement as Measured by U.S. Industry, Crosstalk – The Journal of Defense Software, 1996. Rico, D. F., ROI of Software Process Improvement: Metrics for Project Managers and Software Engineers, J. Ross Publishing, Boca Raton, FL, 2004. Boehm, B. W., Valerdi, R. and Honour, E., “The ROI of Systems Engineering: Some Quantitative Results for Software- Intensive Systems,” Systems Engineering, 11(3), 221-234, 2008.

40. 40

41. 41 Case Study I: Albatross Budgetary Estimate You, as the Albatross SE lead, have just been asked to provide your program manager a systems engineering budgetary estimate for a new, believed-to-be critical function to the current baseline –This new functionality would add some new, nearly-standalone, capability to your Albatross program, your best educated guess by looking at the emailed requirements provided by your customer is that it adds about 10% to the requirements baseline of 1,000 and two new interfaces, you guess we need at least one new Operational Scenario. –The customer also stated that they really need this capability to be integrated into the next delivery (5 months from now) –The PM absolutely has to have your SE cost estimate within the next two hours, as the customer representative needs a not-to-exceed (NTE) total cost soon after that! Information that may prove useful in your response –Albatross is well into system I&T, with somewhat higher than expected defects –Most of the baseline test procedures are nearing completion –The SE group has lost two experienced people in the past month to attrition –So far, the Albatross customer award fees have been excellent, with meeting of schedule commitments noted as a key strength

42. 42 Case Study I: In-Class Discussion Questions What are some of the risks? What additional questions could you ask of your PM? What additional questions could the PM (or the SE Lead) ask of the Customer Representative? What role could the Albatross PM play in this situation? Is providing only “a number” appropriate in this situation? What additional assumptions did you make that can be captured by COSYSMO?

43. 43 Case Study II: Selection of Process Improvement Initiative You are asked by your supervisor to recommend which process improvement initiative should be pursued to help your project, a 5-year $100,000,000 systems engineering study. The options and their implementation costs are: Lean$1,000,000 Six Sigma$600,000 CMMI$500,000 Which one would you chose?

44. 44 Case Study II: Selection of Process Improvement Initiative Assumptions –Implementing Lean on your project will greatly improve your team’s documentation process. Using COSYSMO, you determine that the Documentation cost driver will move from “Nominal” to “Low” yielding a 12% cost savings (1.00 – 0.88 = 0.12). –Implementing Six Sigma will greatly improve your team’s requirements process. Using COSYSMO, you determine that the Requirements Understanding cost driver will move from “Nominal” to “High” yielding a 23% cost savings (1.00 – 0.77 = 0.23) –Implementing CMMI will greatly improve team communication. Using COSYSMO, you determine that the Stakeholder Team Cohesion cost driver will move from “Nominal” to “High” yielding a 19% cost savings (1.00 – 0.81 = 0.19)

45. 45 Cost Driver Rating Scales Very LowLowNominalHighVery High Extra HighEMR Requirements Understanding1.871.371.000.770.60 3.12 Architecture Understanding1.641.281.000.810.65 2.52 Level of Service Requirements0.620.791.001.361.85 2.98 Migration Complexity 1.001.251.551.93 Technology Risk0.670.821.001.321.75 2.61 Documentation0.780.881.001.131.28 1.64 # and diversity of installations/platforms 1.001.231.521.87 # of recursive levels in the design0.760.871.001.211.47 1.93 Stakeholder team cohesion1.501.221.000.810.65 2.31 Personnel/team capability1.501.221.000.810.65 2.31 Personnel experience/continuity1.481.221.000.820.67 2.21 Process capability1.471.211.000.880.770.682.16 Multisite coordination1.391.181.000.900.800.721.93 Tool support1.391.181.000.850.72 1.93

46. 46 Case Study II: Selection of Process Improvement Initiative Assumptions The systems engineers on your project spend: 5% of their time doing documentation 30% of their time doing requirements-related work 20% of their time communicating with stakeholders What is the financial benefit of each process improvement initiative? How does the financial benefit vary by project size?

47. Return-on-Investment Calculation (benefit – cost) / cost Best option is Six Sigma!

48. 48 Six Sigma CMMI Lean Return-on-Investment vs. Project Size

49. Reuse is a Universal Concept 49

50. 50 Reusable Artifacts System Requirements System Architectures System Description/Design Documents Interface Specifications for Legacy Systems Configuration management plan Systems engineering management plan Well-Established Test Procedures User Guides and Operation Manuals Etc.

51. 51 Reuse Principles 1.The economic benefits of reuse can be described either in terms of improvement (quality, risk identification) or reduction (defects, cost/effort, time to market). 2.Reuse is not free, upfront investment is required (i.e., there is a cost associated with design for reusability). 3.Reuse needs to be planned from the conceptualization phase of program

52. 52 Reuse Principles (con’t.) 4.Products, processes, and knowledge are all reusable artifacts. 5.Reuse is as much of a technical issue as it is an organizational one 6.Reuse is knowledge that must be deliberately captured in order to be beneficial.

53. 53 Reuse Principles (con’t.) 7.The relationship between project size/complexity and amount of reuse is non-linear 8.The benefits of reuse are limited to closely related domains. 9.Reuse is more successful when level of service requirements are equivalent across applications.

54. 54 Reuse Principles (con’t.) 10.Higher reuse opportunities exist when there is a match between the diversity and volatility of a product line and its associated supply chain. 11.Bottom-up (individual elements where make or buy decisions are made) and top-down (where product line reuse is made) reuse require fundamentally different strategies. 12.Reuse applicability is often time dependent; rapidly evolving domains offer fewer reuse opportunities than static domains.

55. 55 Reuse Success Factors Platform –Appropriate product or technology, primed for reuse People –Adequate knowledge and understanding of both the heritage and new products Processes –Sufficient documentation to acquire and capture knowledge applicable to reuse as well as the capability to actually deliver a system incorporating or enabling reuse

56. 56 Background on Software Reuse Main size driver = KSLOC Adapted Source Lines of Code (ASLOC) Percent of Design Modification (DM) Percent of Code Modification (CM) Percent of Integration Required for Modified Software (IM) Percentage of reuse effort due to Software Understanding (SU) Percentage of reuse effort due to Assessment and Assimilation (AA) Programmer Unfamiliarity with Software (UNFM) From COCOMO II Model Definition Manual (p. 7-11) AAF

57. Example COSYSMO 2.0 Estimate Estimated as 129.1 Person-Months by COSYSMO (without reuse) …a 30.4% difference

58. COSYSMO 2.0 Implementation Results Across 44 projects at 1 diversified organization Using COSYSMO: –PRED(.30) = 14% –PRED(.40) = 20% –PRED(.50) = 20% –R 2 = 0.50 Using COSYSMO 2.0: –PRED(.30) = 34% –PRED(.40) = 50% –PRED(.50) = 57% –R 2 = 0.72 Result: 36 of 44 (82%) estimates improved September 10, 2009

59. 59 Reuse Terminology New: –Items that are completely new Managed: –Items that are incorporated and require no added SE effort other than technical management Adopted: –Items that are incorporated unmodified but require verification and validation Modified: –Items that are incorporated but require tailoring or interface changes, and verification and validation Deleted: –Items that are removed from a legacy system, which require design analysis, tailoring or interface changes, and verification and validation Notes: New items are generally unprecedented Those items that are inherited but require architecture or implementation changes should be counted as New

60. Reuse Continuum Modified Adopted New 1.0 0 Deleted Managed Reuse weight 0.65 0.51 0.43 0.15 Modified vs. New Threshold

62. 10 Academic Theses Academic Curricula Proprietary Implementations SEEMaP COSYSMO-R SECOST Systems Eng. Cost Tool Commercial Implementations Policy & Contracts COSYSMO Model Impact

63. Take-Aways 1.Cost ≈ f(Effort) ≈ f(Size) ≈ f(Complexity) 2.Requirements understanding and “ilities” are the most influential on cost 3.Early systems engineering yields high ROI when done early and well Two case studies: SE estimate with limited information Selection of process improvement initiative 63