I n t e g r i t y - S e r v i c e - E x c e l l e n c e Headquarters U.S. Air Force Next-Generation Systems and Software Cost Estimation Wilson Rosa Technical Advisor Air Force Cost Analysis Agency (AFCAA) October 28, 2008

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Problem Statement  Emerging technologies such as Systems of Systems (SoS) Model Driven Architecture (MDA) Enterprise Resource Planning (ERP) Service-Oriented Architecture (SOA) Commercial Off the Shelf (COTS) Design for Reuse (RUSE) are complicating AFCAA's job of producing accurate software cost estimates 2

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 3 Next-Generation Systems Challenges 1. Lines of Code not appropriate for  Model Driven Architecture  COTS-Based Systems (SOA, ERP, etc.) 2. No guidelines for estimating beyond software design:  Infrastructure (servers, LAN, routers, etc.)  Concurrent Users  Enterprise Services (collaboration, discovery, etc.)  Data Migration, External Interfaces  Interoperability and Interdependency 3. Unfamiliar with total system size and cost drivers 4. Lack of Empirical Research – SOA, ERP, SoS, MDA

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Data Challenges  AFCAA has multiple software datasets  Unable to combine software datasets because of inconsistencies and lack of standardization  Schedule seems to be reported at program and not CSCI level -- all CSCI’s have same schedule  No reporting of % re-design, % re-coding, % re-test  No common counting method – logical, physical, etc.  No standard application type definitions  No common code counting tool  Product size only reported in lines of code  No reporting of COCOMO, SEER, PRICE parameters  No reporting of quality measures – defects, MTBF, etc. 4

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 5 Parametric Model Challenges  Most DoD Program Offices rely on software parametric models which have not been calibrated with recent DoD data  Parametric Models only cover software design not total system – infrastructure, users, etc.  Calibration will help reduce the program office estimating error rate Electronic Systems Center (Hanscom AFB)SEER-SEM Aeronautical Systems Center (WPAFB)True-S Software Technology Center (Hill AFB)Sage Space and Missile Systems CenterSEER-SEM

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Consequence: Significant Cost Growth (%) 6 Statistics*Total System**Software Only Minimum-64%-80% Mean45%37% Median27%8% High471%623% Standard Deviation71%107% Milestone PhaseDevelopment Sample Size Year of Data Source : *John McCrillis, 36 th DOD Cost Analysis Symposium (2003) **Defense Automated Cost Information System

I n t e g r i t y - S e r v i c e - E x c e l l e n c e Headquarters U.S. Air Force 7 Software Cost Metrics Manual OVERVIEW

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 8 Scope  Cost Agencies in conjunction with University of Southern California will publish a manual to help analysts develop quick software estimates using reliable metrics from recent programs

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 9 Data Sources CommoditySourceFormatYearProjectsCSCIs Space, Ground, AirSoftware Resource Data Reports DD-Form SpaceAEHF SEER AirF-22 EMD and Increment II Boeing SpaceMILSTAR SEER1990s428 SpaceFAB-T DD-Form SpaceNPOESS SEER SpaceTSAT DD-Form Air, GroundNorthrop Grumman COCOMO, SEER Space, GroundRaytheon COCOMO Air, Ship, GroundNaval Center for Cost Analysis TECHNOMICS AirLockheed Martin COCOMO AirArmy Cost and Economics Analysis Center TECHNOMICS GroundFuture Combat System DD-Form SpaceNRO SEERTBD Space, GroundAerospace Unknown TBD Space, GroundSMC Unknown TBD SpaceNASA JPL Unknown TBD >168>598 Note: Expecting over 1600 CSCIs by 2010

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 10 Data Normalization  USC will interview program offices and developers to obtain additional information… 1. COCOMO II Parameters 2. Reuse Type – auto generated, re-hosted, translated, modified 3. Reuse Source – in-house, third party 4. Degree-of-Modification – %DM, %CM, %IM 5. Method – Model Driven Architecture, Object-Oriented, Traditional  Available Data 1. DoDAF – System Views, Operational Views, etc. 2. Software Resource Data Report – Software Size, Effort, Schedule 3. Cost Analysis Requirements Description (CARD)  System Description, Users, Infrastructure, locations, interfaces, etc.

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 11 Software Cost Manual Content Chapter 1: Basic Software Cost Estimation Chapter 2: Product Size Metrics Chapter 3: Historical Growth Chapter 4: Default Effective Size (ESLOC) Parameters Chapter 5: Historical Productivity Dataset Chapter 6: Default COCOMO Parameters Chapter 7: SLIM-ESTIMATE Calibration Chapter 8: Risk and Uncertainty Parameters Chapter 9: Data Cleansing Chapter 10: Space Software Cost Estimation Chapter 11: Software Maintenance

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Chapter 4: ESLOC Parameters 12 Reuse TypeReuse Source Design Modified Code Modified Integration ModifiedESLOC Auto GeneratedIn-House 0% 50%15% Third Party 0% 100%30% Re-HostIn-House 0% 100%30% Third Party 0%24%100%37% TranslatedIn-House 0%100% 60% Third Party 15%100% 66% ModifiedIn-House 0%100% 60% Third Party 100% UnmodifiedIn-House 0% 32%10% Third Party 0% 100%30%  Default values from recent programs  Based on Reuse Type and Reuse Source

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 13 Chapter 5: Historical Productivity  Overview and Guidelines  Historical Productivity Dataset by Application  Default Productivity Ranges by Application IOC CSCIApplication Productivity (ESLOC/MM) Raw (KSLOC) ESLOC Effort (MM) Peak Effort (FTE) Schedule (Months) 1999Signal Processing Avionics Spot Antenna Control Payload BootstrapBus

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Significance of Software Cost Metrics Manual  Collected data can be used for  Systems of Systems cost research  COCOMO improvement initiatives  Understanding relationships between Next- Generation Processes and COCOMO cost drivers can encourage researchers to explore new strategies to improve available cost models… 14

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Way Ahead  Short Term ( )  Send Software Data Call to program offices, developers, and USC Affiliates  Write Chapters 4 & 5 (2009)  Publish Software Cost Metrics Manual (2010)  Long Term ( )  ERP Cost Guide (2010)  Impact of MDA on Software Productivity (2010)  SOA Cost Study (2012) 15 Note: Any data you provide will not be attributed to your company or program, but will be combined with like data from other sources and generic zed"

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 16 Backup Slides I n t e g r i t y - S e r v i c e - E x c e l l e n c e