Working Group Meeting Ricardo Valerdi Thursday October 27, 2005 Los Angeles, CA 20 th International Forum on COCOMO and Software Cost Modeling

Agenda 8:15 AM COSYSMO Status 8:45 AM Major updates COCOMO/COSYSMO Overlap COSYSMO book outline Exercise in Psychometrics COSYSMO game Systems engineering effort estimation in O&M phase 9:15 AM SE Staffing profile survey 9:45 AM Break 10:15 AM Ongoing projects Data availability for COSYSMO (Chris Miller) Schedule implications in COSYSMO (Anthony Peterson) COSYSMOstar (Dan Ligett) 11:45 AM Lunch

COSYSMO Evolution Phase 1: Baseline ☺ Phase 2: Prototype ☺ Phase 3: Validate ☺ Phase 4: Institutionalize

Notional Estimation Example Company “Lockheed Grumman” is developing a system that has: COSYSMO Size Drivers Effort Multipliers 36 Person Months of systems engineering effort Calibration 100 easy, 50 nominal, 75 difficult requirements 2 easy, 3 difficult interfaces 4 easy algorithms 5 nominal operational scenarios High requirements und High tech risk High process capability

COSYSMO Status OrganizationData Points in Calibration BAE Systems 19 General Dynamics10 Raytheon10 Lockheed Martin8 Northrop Grumman4 SAIC 2 TOTAL 53 academicCOSYSMO, myCOSYSMO, COSYSMO Risk Add-on, and COSYSMOstar continue to be improved and can be downloaded from

Major Updates COCOMO II/COSYSMO overlap –MBASE/RUP phases and ISO –Integration/test/requirements activities Calibration factor clarification Hours = * (size)^1.06 * product (EM) PM = 38.55/152 * (size)^1.06 * product (EM) 0.25

COSYSMO Book Outline 1.Scope of COSYSMO a)Background on Systems Engineering b)COCOMO II and COSYSMO c)Estimation example 2.Model definition a)Model form b)Size drivers & counting rules c)Cost drivers & interpretations 3.Model verification & validation a)Statistical tests b)Model parsimony c)Bayesian approximation 4.Model usage a)Experience base b)Model Tailoring c)Institutionalization & training

Exercise in Psychometrics* Psychometrics is the field of study (connected to psychology and statistics) concerned with the measurement of "psychological" aspects of a person such as knowledge, skills, abilities, or personality. Source: Guilford JP. Psychometric methods. 2nd edn. New York: McGraw-Hill; Stakeholder team cohesion 2. Personnel/team capability 3. Personnel experience/continuity 4. Process Capability *Inspired by Clark and subsequently Guilford Source:

Reality Check

Stakeholder team cohesion Represents a multi-attribute parameter which includes leadership, shared vision, diversity of stakeholders, approval cycles, group dynamics, IPT framework, team dynamics, trust, and amount of change in responsibilities. It further represents the heterogeneity in stakeholder community of the end users, customers, implementers, and development team. ViewpointVery LowLowNominalHighVery High Culture  Stakeholders with diverse expertise, task nature, language, culture, infrastructure  Highly heterogeneous stakeholder communities  Heterogeneous stakeholder community  Some similarities in language and culture  Shared project culture  Strong team cohesion and project culture  Multiple similarities in language and expertise  Virtually homogeneous stakeholder communities  Institutionalized project culture Compatibility  Highly conflicting organizational objectives  Converging organizational objectives  Compatible organizational objectives  Clear roles & responsibilities  Strong mutual advantage to collaboration Familiarity and trust  Lack of trust  Willing to collaborate, little experience  Some familiarity and trust  Extensive successful collaboration  Very high level of familiarity and trust

Personnel/team capability Basic intellectual capability of a Systems Engineer (compared to the national pool of SEs) to analyze complex problems and synthesize solutions. Very LowLowNominalHighVery High 15 th percentile35 th percentile55 th percentile75 th percentile90 th percentile Personnel experience/continuity The applicability and consistency of the staff at the initial stage of the project with respect to the domain, customer, user, technology, tools, etc. Very lowLowNominalHighVery High ExperienceLess than 2 months1 year continuous experience, other technical experience in similar job 3 years of continuous experience 5 years of continuous experience 10 years of continuous experience Annual Turnover 48%24%12%6%3%

Process capability The consistency and effectiveness of the project team at performing SE processes. This may be based on assessment ratings from a published process model (e.g., CMMI, EIA- 731, SE-CMM, ISO/IEC15504). It can also be based on project team behavioral characteristics, if no assessment has been performed. Very lowLowNominalHighVery HighExtra High Assessme nt Rating (Capability or Maturity) Level 0 (if continuou s model) Level 1Level 2Level 3Level 4Level 5 Project Team Behavioral Characteri stics Ad Hoc approach to process performan ce Performed SE process, activities driven only by immediate contractual or customer requirements, SE focus limited Managed SE process, activities driven by customer and stakeholder needs in a suitable manner, SE focus is requirements through design, project-centric approach – not driven by organizational processes Defined SE process, activities driven by benefit to project, SE focus is through operation, process approach driven by organizational processes tailored for the project Quantitatively Managed SE process, activities driven by SE benefit, SE focus on all phases of the life cycle Optimizing SE process, continuous improvement, activities driven by system engineering and organizational benefit, SE focus is product life cycle & strategic applications

COSYSMO Game Purpose: to serve as a training tool for stakeholders to understand the capabilities and limitations of COSYSMO Rules The formal part of the game. A game designer designs the rules of the game directly but designs the player’s experience only indirectly. Source: Rules of Play: Game Design Fundamentals by Salen & Zimmerman, MIT Press Play Rule-bound and free-form; Players are assigned roles and given a project. Culture Simulated project which realistically mimics the real world of systems engineering.

COSYSMO Game Rules –Facilitator sets the stage for a “case study” –Deliberately creates a model clash scenario Play –Role-based decision making Culture –Expectations and politics must be highlighted –Bring forward social process of cost estimation

Cast of Characters CastFunction CustomerDevelops spec ContractorEvaluates spec 3 rd party consultant (i.e., SETA or FFRDC) Evaluates spec & bid ObserversDon’t interfere but keep track of what’s happening

Role Playing Activities PhaseActivities 1. Warm upFacilitator: Sets up scenario Makes scenario real Surfaces model clashes Explains roles 2. Select participantsParticipants: Roles are assigned 3. Set the stageFacilitator & Participants Clarifies scenario, background, and goal 4. Prepare the observersFacilitator & Participants Identify what observers are observing Assign observing tasks 5. ActionParticipants Act out scenario Observe flow, behaviors, results, etc. Source: Models of Teaching, 6th Ed., by Joyce, Weil, Calhoun, 2000.

Role Playing Activities PhaseActivities 6. DebriefParticipants: Present COSYSMO results 7. GeneralizationFacilitator & Participants: Relate to reality Comment on limitations of the model Summarize general challenges of process Source: Models of Teaching, 6th Ed., by Joyce, Weil, Calhoun, 2000.

Operate & Maintain Phase Current scope Suggestions: Need to clarify point where M&E begins Add “deleted” category (may be a different weighting) for requirements, interfaces, etc. Consider that not all current COSYSMO parameters will apply here and that the ones that do, may have different values Need to consider business strategies with respect to OM&E (e.g., problem report-driven, level of effort/priority driven, requirement changes, upgrade/technology refresh activities, etc.)—may require other size drivers Need to better define OM&E activities Consider frequency of deliveries as a driver Conceptualize Develop Oper Test & Eval Transition to Operation Operate, Maintain, or Enhance Replace or Dismantle

Inconsistent Effort Reporting Data was adjusted based on the following: –Effort distribution across EIA632 processes ISO15288 phases –Yielding Phase Conceptualiz e Develop Operational Test & Eval Transition to Operation %Effort (STDEV) 23 (12)36 (16)27 (13)14 (9)

Effort Distribution Across EIA 632 Fundamental Processes N = 18 EIA 632 Fundamental Process AverageStandard Deviation Acquisition & Supply 7%3.5 Technical Management 17%4.5 System Design 30%6.1 Product Realization 15%8.7 Technical Evaluation 31%8.7 Total = 100%

ISO/IEC Conceptualize Develop Transition to Operation EIA/ANSI 632 Acquisition & Supply Technical Management System Design Product Realization Technical Evaluation Operational Test & Evaluation Effort Profiling mini-Delphi

ConceptualizeDevelop Operation al Test & Eval. Transition to Operation Operate, Maintain, or Enhance Replace or Dismantle (check sum) Acquisition and Supply 28 (12.3)51 (18.6)13 (11.3)8 (5.0)100 Technical Management 22 (10.0)38 (9.9)25 (7.4)15 (6.4)100 System Design 34 (12.4)40 (19.4)17 (9.6)9 (6.2)100 Product Realization 13 (14.1)30 (24.3)32 (16.0)25 (20.4)100 Technical Evaluation 18 (11.4)27 (11.0)40 (17.7)15 (8.5)100 Effort Distribution of EIA 632 Fundamental Processes Across ISO Phases N = 15 In each cell: Average (Standard Deviation)

Contact Ricardo Valerdi MIT Lean Aerospace Initiative (617)