1. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Headquarters U.S. Air Force Next-Generation Systems and Software Cost Estimation Wilson Rosa Technical Advisor Air Force Cost Analysis Agency (AFCAA) October 28, 2008

2. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Problem Statement  Emerging technologies such as Systems of Systems (SoS) Model Driven Architecture (MDA) Enterprise Resource Planning (ERP) Service-Oriented Architecture (SOA) Commercial Off the Shelf (COTS) Services Reuse-Driven (RUSE) are complicating AFCAA's job of producing accurate software cost estimates 2

3. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 3 Next-Generation System Challenges 1. Lines of Code not appropriate for  Model Driven Architecture  COTS-Based Systems (SOA, ERP, etc.) 2. No guidelines for estimating beyond software design  Infrastructure (servers, LAN, routers, etc.)  Concurrent Users  Enterprise Services (collaboration, discovery, portal, etc.)  Data Migration, External Interfaces  Interoperability and Interdependency 3. Unfamiliar with total system cost drivers 4. Lack of Empirical Research – SOA, ERP, SoS, MDA

4. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Data Challenges  AFCAA has multiple software datasets  Unable to combine software datasets because of inconsistencies and lack of standardization  Schedule seems to be reported at program and not CSCI level -- all CSCI’s have same schedule  No reporting of % re-design, % re-coding, % re-test  No common counting method – logical, physical, etc.  No standard application type definitions  No common code counting tool  Product size only reported in lines of code  No reporting of COCOMO, SEER, PRICE parameters  No reporting of quality measures – defects, MTBF, etc. 4

5. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 5 Parametric Model Challenges  Most AF program offices rely on cost models  Have these models been calibrated with the most recent DoD data? (2002-2008)  Most models only cover a fraction, not total system costs – infrastructure, users, etc.  Model calibration with recent data will help reduce the program office estimating error rate

6. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Consequence: Significant Cost Growth (%) 6 StatisticsTotal System*Software Only** Minimum-64%-80% Mean45%37% Median27%8% High471%623% Standard Deviation71%107% Milestone PhaseDevelopment Sample Size137111 Year of Data1993-20032002-2008 Source : *John McCrillis, 36 th DOD Cost Analysis Symposium (2003) **Defense Automated Cost Information System

7. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Headquarters U.S. Air Force 7 Software Cost Metrics Manual OVERVIEW

8. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 8 Scope  Cost Agencies in conjunction with University of Southern California will publish a manual to help analysts develop quick software estimates using reliable metrics from recent programs

9. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 9 Software Cost Manual Content Chapter 1: Basic Software Estimation Chapter 2: Product Size Estimation Chapter 3: Schedule and Cost Growth Chapter 4: Default ESLOC Parameters Chapter 5: Historical Productivity Chapter 6: Default COCOMO/SEER/True-S Parameters Chapter 7: SLIM-ESTIMATE Calibration Chapter 8: Risk and Uncertainty Parameters Chapter 9: Data Normalization Chapter 10: Space Software Cost Guide Chapter 11: Software Maintenance

10. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 10 Data Sources CommoditySourceFormatYearProjectsCSCIs Space, Ground, AirSoftware Resource Data Reports DD-Form 26302002-200832103 SpaceAEHF SEER2008653 AirF-22 EMD and Increment II Boeing200413112 SpaceMILSTAR SEER1990s428 SpaceFAB-T DD-Form 26302008721 SpaceNPOESS SEER2008367 SpaceTSAT DD-Form 2630200733 Air, GroundNorthrop Grumman COCOMO, SEER1997-20081532 Space, GroundRaytheon COCOMO1997-20083349 Air, Ship, GroundNaval Center for Cost Analysis TECHNOMICS1992-20012168 AirLockheed Martin COCOMO1996-200422 AirArmy Cost and Economics Analysis Center TECHNOMICS2001-200416 GroundFuture Combat System DD-Form 26302003-20081342 Space, IntelIntelligence Community (NRO, NGA, DNI) SEERTBD Space, GroundAerospace, Space & Missile System Center Unknown TBD SpaceNASA JPL Unknown TBD Space, Air, GroundUSC AffiliatesUnknown TBD >168>598 Note: Expecting over 1600 CSCIs by 2010

11. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 11 Data Normalization  USC will interview program offices and developers to obtain additional information… 1. Cost Model Parameters – True S, SEER, COCOMO 2. Reuse Type – auto generated, re-hosted, translated, modified 3. Reuse Source – in-house, third party 4. Degree-of-Modification – %DM, %CM, %IM 5. Method – Model Driven Architecture, Object-Oriented, Traditional  Available Data 1. DoDAF – System Views, Operational Views, etc. 2. Software Resource Data Report – Software Size, Effort, Schedule 3. Cost Analysis Requirements Description (CARD)  System Description, Users, Infrastructure, locations, interfaces, etc.

12. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Chapter 4: ESLOC Parameters 12 Reuse TypeReuse Source Design Modified Code Modified Integration ModifiedESLOC Auto GeneratedIn-House 0% 50%15% Third Party 0% 100%30% Re-HostIn-House 0% 100%30% Third Party 0%24%100%37% TranslatedIn-House 0%100% 60% Third Party 15%100% 66% ModifiedIn-House 0%100% 60% Third Party 100% UnmodifiedIn-House 0% 32%10% Third Party 0% 100%30%  Default values from recent programs  Based on Reuse Type and Reuse Source

13. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 13 Chapter 5: Historical Productivity  Overview and Guidelines  Historical Productivity Dataset by Application  Default Productivity Ranges by Application IOC CSCIApplication Productivity (ESLOC/MM) Raw (KSLOC) ESLOC Effort (MM) Peak Effort (FTE) Schedule (Months) 1999Signal Processing Avionics6090000 10006971 2008Spot Antenna Control Payload3978000500020009569 2008BootstrapBus4435000310008007060

14. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Significance of Proposed Manual  Collected data can be used for  Systems of Systems cost research  Cost Model improvement initiatives  SEER, True-S, COCOMO, etc.  Understanding relationships between Next- Generation Processes and COCOMO cost drivers can encourage researchers to explore new strategies to improve available cost models… 14

15. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Way Ahead  Short Term  NCCA/AFCAA Software Cost Handbook (Nov 2008)  Software Cost Metrics Manual (2009)  Long Term  Cost Risk and Uncertainty Handbook Volume II (2011)  ERP Cost Guide (2010)  Impact of MDA on Cost Modeling (2010)  SOA Cost Study (2012)  Data Center Cost Study (2010) 15 Note: Any data you provide will not be attributed to your company or program, but will be combined with like data from other sources and generic zed"

16. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 16 Backup Slides I n t e g r i t y - S e r v i c e - E x c e l l e n c e