COSOSIMO October 2005 Workshop Jo Ann Lane University of Southern California Center for Software Engineering COCOMO Forum – October 2005 © USC CSE 2005

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Agenda COSOSIMO Workshop July 2005 Survey Results (~30 minutes) –Overview of results –How to interpret –Call for additional responses Using COSYSMO to estimate SoS LSI technical effort (~1 hour) –Tutorial Case Study –Call for data Break (~20 minutes) SoS LSI management model ideas (~1 hour) –Candidate ideas—Gary Constantine, Raytheon –Discussion SoS/FoS WBS ideas—Gan Wang (~1 hour)

System-of-Systems Cost Modeling: COSOSIMO July 2005 Workshop Results Jo Ann Lane University of Southern California Center for Software Engineering COCOMO Forum – October 2005 © USC CSE 2005

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum July 2005 Workshop Objectives and Summary Objectives –Clarify the LSI activities to be estimated by the COSOSIMO model –Begin converging on a relevant and complete set of parameters for the COSOSIMO model that are easily discerned in the early stages of SoS development Summary of workshop activities –Attended by 20 people representing 12 organizations –Provided overview on research conducted to date on LSI activities –Discussed/updated list of key LSI activities –Conducted survey to better determine the differences between LSI activities and more traditional SE activities –Discussed size drivers and factors that impact LSI effort –Conducted survey to identify relevant LSI size drivers and scale factors

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Workshop Recommendations and Results LSI Activities –Recommendation: LSI activities should be broken out into the following categories Management activities Technical activities –Results: Updated list of LSI activities No initial activities or issues dropped from list Additional activities added to list Additional issues added to list LSI Effort Management Effort Technical Effort Discussions to continue in this area …. Proposed scope for COSOSIMO if technical can be estimated using COSYSMO SoS calibration Investigate possibility of COSYSMO calibration to estimate this part

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Workshop Recommendations and Results (continued) Suggested additional activities –Configuration management –Common infrastructure alignment/supply Development common vocabulary Management processes Technical processes/tools –Managing interoperability with external systems –Verification and validation of the total system Development of SoS test bed/integration lab SoS level requirements in addition to oversight of lower level V&V Suggested additional activities (continued) –Transition plans –Ensure communications between various SoS orgs –Provide logistics, support centers, other -ilities –Security approach (developmental and operational) –Safety plans –Training –Post implementation communications –Disaster recovery –Tradeoffs on level of service reqs –Development of SoS infrastructure? Discussions to continue in this area …. Activities to be analyzed with respect to size and cost drivers….

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Workshop Results (continued) Suggested additional issues that may impact LSI effort –Conflict of interest –Sharing of proprietary info –Import/export concerns with international teams –Subcontractor process maturity –Supplier stabilization and synchronization –Diversity of supplier processes, methods, and tools –Synchronization of tools –Level of component independence Discussions to continue in this area …. Issues to be analyzed with respect to size and cost drivers….

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum COSOSIMO Survey Overviews LSI activity survey –Description Typical LSI effort was compared to the typical systems engineering effort defined in COSYSMO Comparison was done for the 33 activities defined in EIA632 –12 responses received to date Size Driver and Scale Factor Survey –Description Survey contained a list of current COSYSMO size drivers and scale factors as well as the proposed COSOSIMO size drivers and scale factors Respondents were asked to indicate which parameters applied to LSI management activities (by entering an M) and which parameters applied to LSI technical activities (by entering a T) –12 responses received to date

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Survey Results: LSI Activities General observations –Number of EIA/632 activities evaluated: 33 –Highest average rating: 0.67 Lowest average rating: 0.00 –Number of activities with average rating 0.5 or higher: 15 –Number of activities with average rating between 0.3 and 0.5: 9 –Number of activities with average rating 0.3 or lower: 9 -1 (less) 0 (same) 1 (more) Survey Scale:

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Survey Results: LSI Activities (continued) Activities with average rating 0.5 or higher –Product supply– Product acquisition –Supplier performance– Process implementation strategy –Technical effort definition – Outcomes management –Information dissemination– Other stakeholder requirements –Transition to use– Effectiveness analysis –Risk analysis– Requirements statements validation –Logical solution validation– End product verification –End product validation Activities with average rating between 0.3 and 0.5 –Schedule and organization– Progress against plans and schedules –Technical reviews– System technical requirements –Implementation– Tradeoff analysis –Acquirer requirements validation– Other stakeholder requirements validation –Enabling product readiness Activities with average rating 0.3 or lower –Technical plans– Work directives – Progress against requirements– Acquirer requirements – Design: Logical solutions– Design: Physical solutions – Design: Specified requirements– System technical requirements validation –Design solution verification

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Survey Results: LSI Activities (continued) LSI activity survey conclusions (so far) –All EIA 632 activities are applicable to the LSI effort (there were only two responses that indicated a specific activity was not applicable) –Most EIA 632 activities (75%) require the same or more effort in the LSI environment than in the more traditional SE projects –No survey identified additional activities to be included in this list – however, during workshop discussions, others were identified Interpretation: the other activities identified during discussions were either a clarification of the emphasis of the activity or a set of lower level activities

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Survey Results: Size Drivers and Scale Factors Management Size Drivers –Average response “high”:# of independent system component orgs –Average response “medium:# of component systems –Average response “low: # system requirements # system interfaces # operational scenarios Subsystem software size # SoS interface protocols –Average response “N/A”:# algorithms Survey Rating Scale: High Applicability. Medium Applicability, Low Applicability, or Not Applicable (N/A)

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Survey Results: Size Drivers and Scale Factors (continued) Technical Size Drivers –Average response “high”:# system interfaces # operational scenarios –Average response “medium”: # system requirements # of component systems # SoS interface protocols –Average response “low”: # algorithms Subsystem software size # of independent system component orgs –Average response “N/A”:(none) Survey Rating Scale: High Applicability. Medium Applicability, Low Applicability, or Not Applicable (N/A)

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Survey Results: Size Drivers and Scale Factors (continued) Management Scale Factors –Average response “high”:Stakeholder team cohesion Multi-site coordination Cost/schedule compression –Average response “medium”: Requirements understanding Architecture understanding Migration complexity Technology risk # and diversity of installations/platforms Personnel/team capability Personnel experience/continuity Process capability Tool support Integration risk resolution Integration stability Component readiness Component system maturity/stability Survey Rating Scale: High Applicability Medium Applicability Low Applicability Not Applicable (N/A)

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Survey Results: Size Drivers and Scale Factors (continued) Management Scale Factors –Average response “low”: Level of service requirements Documentation Recursive levels in the design Integration simplicity –Average response “N/A”:(none) Survey Rating Scale: High Applicability. Medium Applicability, Low Applicability, or Not Applicable (N/A)

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Survey Results: Size Drivers and Scale Factors (continued) Technical Scale Factors –Average response “high”:Requirements understanding Personnel/team capability Personnel experience/continuity –Average response “medium”: everything else…. Survey Rating Scale: High Applicability. Medium Applicability, Low Applicability, or Not Applicable (N/A)

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Survey Results: Size Drivers and Scale Factors (continued) Size driver survey conclusions (so far) –Management Size Drivers: Most important appear to be key drivers currently defined for COSOSIMO –Technical Size Drivers: Most of the high and medium applicability drivers are those currently defined for COSYSMO (not COSOSIMO) Exceptions –COSOSIMO “# of system components” included under medium applicability –COSOSIMO “# of SoS level interfaces” included under medium applicability (Possible interpretation: respondents may have thought this related to COSYSMO “# of system interfaces”) –COSYSMO “# of algorithms” thought to be low applicability –Rated low in both categories # of algorithms Sub-system software size –No significant suggestions for additional size drivers

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Survey Results: Size Drivers and Scale Factors (continued) Scale factor survey conclusions (so far) –Management scale factors: Most (80%) of the COSYSMO and COSOSIMO scale factors were thought, on average, to be of high or medium applicability None of the listed scale factors had an average rating of “not applicable –Technical scale factors: ALL of the proposed scale factors were thought, on average, to be of either high or medium applicability No significant suggestions for additional scale factors Questions to consider –If COSYSMO is to be used to estimate LSI technical effort, are additional scale factors required? –Or is there embedded overlap in the current set of scale factors? –Is there a better, minimal set of scale factors that are sufficient for both COSYSMO and/or COSOSIMO?

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Going Forward Continue to –Collect survey data –Identify sources of actual SoS LSI effort data Compare COSYSMO outputs with COSOSIMO outputs for selected SoS programs Topics for discussion at this week’s workshop –Impact of July 205 survey results on current COSOSIMO model LSI activities Size drivers and scale factors –Using COSYSMO to estimate LSI technical effort –Management model ideas Overview of some management model parameters Sufficient for LSI management effort? –SoS/FoS WBS ideas For the most current information on COSOSIMO, see

New Processes and Estimation Methods for Acquiring 21st Century Software-Intensive Systems of Systems ****************** Case Study Barry Boehm Jo Ann Lane Winsor Brown COCOMO Forum – October 2005 © USC CSE 2005 University of Southern California Center for Software Engineering

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Metropolitan Area Crisis Management System (MACMS) Net-Centric SoSNet-Centric SoS

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum MACMS Overview Objective: To integrate and share information between the various crisis management organizations within a metropolitan area. Integrated organizations include –911 system –Police and Sheriff’s Departments –Fire –Jail –Ambulances –City and county planning (provide information on roads, property boundaries, property owners, etc.) –Hazardous materials management –Biological hazard containment –Bomb squad –Hospitals –Local news organizations (with live video feeds that can be used by people on the ground) –Community warning systems –Related State and Federal agencies (e.g., State Highway Patrol, State Wants and Warrants, FBI, Border Patrol, Coast Guard)

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Key Features Ability to coordinate responses to metropolitan area crises –Classify type of crisis –Alert appropriate organizations –Alert/evacuate public –Identify and manage needed resources Fire trucks Airplanes Helicopters Robots/remotely controlled vehicles Medical supplies/special treatment or isolation facilities Request and coordinate support from other agencies: state, Federal, or other regional areas Support crisis management activities in other regions

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum MACMS Issues and Risks Incompatible interfaces between existing systems COTS products available to support interconnectivity, but have not been used at this level (potential scalability issues) Police and fire departments currently have on-going projects to integrate the police, fire department, and 911 systems Limited local budgets to modify other existing systems Little or no modifications expected for related State and Federal systems (potential impacts with interfaces to other metropolitan area systems) Federal funds available if system implemented within the next 5 years Both County Board of Supervisors and City Council need to approve plans and budgets Citizen privacy and security issues

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Discussion Topics Who should be the LSI? County? City? Contractor? Shared vision Feature priorities Candidate architecture(s) and increment definitions: What can be defined as “independent projects”? How does this impact cost and schedule? Candidates for critical path Risk management: What key risks should be addressed first? Where to be agile? Where to be plan-driven? Oversight and management: What to control with vendors/suppliers? What to track (suppliers and collaborating organizations)?

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum SISOS Schedule Estimation: A Composite Approach Customer, Users LSI – Agile LSI IPTs – Agile Suppliers – Agile Suppliers – PD – V&V LSI – Integrators RFP, SOW, Evaluations, Contracting Effort/Staff Proposals Similar, with added change traffic from users… Assess compatibility, short-falls Rework LCO  LCA Packages at all levels COSOSIMO -like Assess sources of change; Negotiate rebaselined LCA 2 package at all levels COSOSIMO -like Similar, with added re- baselineing risks and rework… Inception Elaboration Source SoS Selection Architecting Increment 1 Increments 2,… n Develop to spec, V&V CORADMO -like Degree of Completeness risks, rework Proposal Feasibility LCOLCA LCA 1 IOC 1 Effort/staff at all levels risks, rework Risk-manage slow- performer, completeness risks, rework Integrate COSOSIMO -like LCA 2 shortfalls risks, rework Effort COSYSMO-like. Schedule = Effort/Staff Try to model ideal staff size LCA 2

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum SISOS Effort and Schedule Estimation Exercise Notes and assumptions: –To handle the unavailability of a calibrated COSOSIMO model Will use COSYSMO to estimate LSI technical effort Will determine an additional % to address LSI management effort Exercise –Using guidelines in tutorial, estimate Inception effort and schedule –Using guidelines in tutorial, estimate Elaboration effort and schedule –Outline approach for estimating Construction and Transition effort and schedule

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum MACMS SISOS Characteristics Current size information –Key features documented in a requirements specification that contains approximately (TBD) functional and performance requirements –To date, about (TBD) system interfaces have been identified, about half of which are documented in some sort of Interface Requirements Specification –During the initial requirements specification process, about (TBD) top level use cases/scenarios were developed SISOS Approach –Integrate existing legacy systems together using a net-centric architecture that includes wireless, mobile networks for mobile units and existing networks for fixed control center connectivity –As part of this effort, the city and county planning and land use organizations would like to replace their location tracking systems with a new system that is based on city/county records and not the more general purpose map programs/ databases typically provide by Geographic Information System (GIS) vendors –No other new system components planned for the early versions of this SoS –Build on existing connectivity Some sort of connectivity exists between –City police, sheriff’s, 911, and ambulance systems –Jail information system and state and Federal agencies Most other system components are relatively closed, independent systems

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Notes from Interactive Exercise: Architecture (System of Interest) View Source: ISO/IEC Goal: Describe MACMS System of Interest for estimation purposes…

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Notes from Interactive Exercise: MACMS Architecture View

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Notes from Interactive Exercise: Assumptions

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Notes from Interactive Exercise: COSYSMO Parameters Size drivers –# requirements: –# interfaces: –# algorithms: –# operational scenarios: Scale factors –Requirements Understanding: –Architecture Understanding: –Level of Service Requirements: –Migration Complexity: –Technology Risk: –Documentation: –# and diversity of installations/platforms: –# of recursive levels in the design: –Stakeholder team cohesion: –Personnel/team capability: –Personnel experience/continuity: –Process capability: –Multi-site coordination: –Tool support:

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum COSYSMO Conceptual Phase Effort = 23% –7% Inception; 16% Elaboration Inception Phase Effort = (total effort) *.07 = ____ PM Inception Phase Schedule –Months = 1.5 * cube root (inception effort) = ___ months –Average staff size = effort/months = ____ people Notes from Interactive Exercise: Schedule Estimation

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Notes from Interactive Exercise: Schedule Estimation (continued) Sum of schedules for systems engineering, source selection, and post-selection rebaselining COSYSMO Elaboration effort = (total effort) *.16 = _____ PM Systems engr. schedule = 1.5 * cube root (elab effort) = 19 months –Average staff size = elab effort/months = ___ people Source selection schedule –Preparation in parallel: no added schedule –RFP finalization and publication: ___ month –Proposal responses: ___ months Including prototypes, architecture, Feasibility Rationale –Parallel evaluation, finalist selection: ___ months –Finalist compatibility/feasibility, Q&A: ___ months –Contracting: ___month –Total schedule: ___ months Teambuilding, LCA rebaselining: ___ months Total Elaboration Phase schedule: ___ months

COSOSIMO Workshop Results © USC CSE 2005 COCOMO Forum Summary of Estimation Results