1. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Headquarters U.S. Air Force Next-Generation Systems and Software Cost Estimation Wilson Rosa Technical Advisor Air Force Cost Analysis Agency (AFCAA) October 28, 2008

2. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Problem Statement  Emerging technologies such as Systems of Systems (SoS) Model Driven Architecture (MDA) Enterprise Resource Planning (ERP) Service-Oriented Architecture (SOA) Commercial Off the Shelf (COTS) Design for Reuse (RUSE) are complicating AFCAA's job of producing accurate software cost estimates 2

3. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 3 Next-Generation Systems Challenges 1. Lines of Code not appropriate for  Model Driven Architecture  COTS-Based Systems (SOA, ERP, etc.) 2. No guidelines for estimating beyond software design:  Infrastructure (servers, LAN, routers, etc.)  Concurrent Users  Enterprise Services (collaboration, discovery, etc.)  Data Migration, External Interfaces  Interoperability and Interdependency 3. Unfamiliar with total system size and cost drivers 4. Lack of Empirical Research – SOA, ERP, SoS, MDA

4. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Data Challenges  AFCAA has multiple software datasets  Unable to combine software datasets because of inconsistencies and lack of standardization  Schedule seems to be reported at program and not CSCI level -- all CSCI’s have same schedule  No reporting of % re-design, % re-coding, % re-test  No common counting method – logical, physical, etc.  No standard application type definitions  No common code counting tool  Product size only reported in lines of code  No reporting of COCOMO, SEER, PRICE parameters  No reporting of quality measures – defects, MTBF, etc. 4

5. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 5 Parametric Model Challenges  Most DoD Program Offices rely on software parametric models which have not been calibrated with recent DoD data  Parametric Models only cover software design not total system – infrastructure, users, etc.  Calibration will help reduce the program office estimating error rate Electronic Systems Center (Hanscom AFB)SEER-SEM Aeronautical Systems Center (WPAFB)True-S Software Technology Center (Hill AFB)Sage Space and Missile Systems CenterSEER-SEM

6. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Consequence: Significant Cost Growth (%) 6 Statistics*Total System**Software Only Minimum-64%-80% Mean45%37% Median27%8% High471%623% Standard Deviation71%107% Milestone PhaseDevelopment Sample Size137111 Year of Data1993-20032002-2008 Source : *John McCrillis, 36 th DOD Cost Analysis Symposium (2003) **Defense Automated Cost Information System

7. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Headquarters U.S. Air Force 7 Software Cost Metrics Manual OVERVIEW

8. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 8 Scope  Cost Agencies in conjunction with University of Southern California will publish a manual to help analysts develop quick software estimates using reliable metrics from recent programs

9. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 9 Data Sources CommoditySourceFormatYearProjectsCSCIs Space, Ground, AirSoftware Resource Data Reports DD-Form 26302002-200832103 SpaceAEHF SEER2008653 AirF-22 EMD and Increment II Boeing200413112 SpaceMILSTAR SEER1990s428 SpaceFAB-T DD-Form 26302008721 SpaceNPOESS SEER2008367 SpaceTSAT DD-Form 2630200733 Air, GroundNorthrop Grumman COCOMO, SEER1997-20081532 Space, GroundRaytheon COCOMO1997-20083349 Air, Ship, GroundNaval Center for Cost Analysis TECHNOMICS1992-20012168 AirLockheed Martin COCOMO1996-200422 AirArmy Cost and Economics Analysis Center TECHNOMICS2001-200416 GroundFuture Combat System DD-Form 26302003-20081342 SpaceNRO SEERTBD Space, GroundAerospace Unknown TBD Space, GroundSMC Unknown TBD SpaceNASA JPL Unknown TBD >168>598 Note: Expecting over 1600 CSCIs by 2010

10. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 10 Data Normalization  USC will interview program offices and developers to obtain additional information… 1. COCOMO II Parameters 2. Reuse Type – auto generated, re-hosted, translated, modified 3. Reuse Source – in-house, third party 4. Degree-of-Modification – %DM, %CM, %IM 5. Method – Model Driven Architecture, Object-Oriented, Traditional  Available Data 1. DoDAF – System Views, Operational Views, etc. 2. Software Resource Data Report – Software Size, Effort, Schedule 3. Cost Analysis Requirements Description (CARD)  System Description, Users, Infrastructure, locations, interfaces, etc.

11. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 11 Software Cost Manual Content Chapter 1: Basic Software Cost Estimation Chapter 2: Product Size Metrics Chapter 3: Historical Growth Chapter 4: Default Effective Size (ESLOC) Parameters Chapter 5: Historical Productivity Dataset Chapter 6: Default COCOMO Parameters Chapter 7: SLIM-ESTIMATE Calibration Chapter 8: Risk and Uncertainty Parameters Chapter 9: Data Cleansing Chapter 10: Space Software Cost Estimation Chapter 11: Software Maintenance

12. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Chapter 4: ESLOC Parameters 12 Reuse TypeReuse Source Design Modified Code Modified Integration ModifiedESLOC Auto GeneratedIn-House 0% 50%15% Third Party 0% 100%30% Re-HostIn-House 0% 100%30% Third Party 0%24%100%37% TranslatedIn-House 0%100% 60% Third Party 15%100% 66% ModifiedIn-House 0%100% 60% Third Party 100% UnmodifiedIn-House 0% 32%10% Third Party 0% 100%30%  Default values from recent programs  Based on Reuse Type and Reuse Source

13. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 13 Chapter 5: Historical Productivity  Overview and Guidelines  Historical Productivity Dataset by Application  Default Productivity Ranges by Application IOC CSCIApplication Productivity (ESLOC/MM) Raw (KSLOC) ESLOC Effort (MM) Peak Effort (FTE) Schedule (Months) 1999Signal Processing Avionics6090000 10006971 2008Spot Antenna Control Payload3978000500020009569 2008BootstrapBus4435000310008007060

14. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Significance of Software Cost Metrics Manual  Collected data can be used for  Systems of Systems cost research  COCOMO improvement initiatives  Understanding relationships between Next- Generation Processes and COCOMO cost drivers can encourage researchers to explore new strategies to improve available cost models… 14

15. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Way Ahead  Short Term (2009-2010)  Send Software Data Call to program offices, developers, and USC Affiliates  Write Chapters 4 & 5 (2009)  Publish Software Cost Metrics Manual (2010)  Long Term (2010-2012)  ERP Cost Guide (2010)  Impact of MDA on Software Productivity (2010)  SOA Cost Study (2012) 15 Note: Any data you provide will not be attributed to your company or program, but will be combined with like data from other sources and generic zed"

16. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 16 Backup Slides I n t e g r i t y - S e r v i c e - E x c e l l e n c e