Affordability Analysis: The Role Process, Cost and ROI Modeling In Improved Program Performance SEI October 2012 Galorath Incorporated Daniel D. Galorath: Founder & CEO Copyright Galorath Incorporated 2012

Key Points Viable affordability decisions yield project achievements Parametric Models provide affordability decision support Positive ROI when decisions use affordability modeling 2 © 2012 Copyright Galorath Incorporated 2

Cost Overruns Are Everywhere “GAO: Staggering cost overruns dwarf modest improvements in Defense acquisition” R&D costs of weapons programs increased 42% over original estimates Average delay 22 months in delivering initial capabilities Evolving technical requirements Shortage of qualified government staff to manage "Every dollar of cost growth on a DoD weapon system represents a lost opportunity to pay for another national priority" “The Collins Class submarine program: Murphy was an optimist” “BG Group shares hit by $5.4B cost overruns on Australian Liquified Natural Gas project”

Software Is A Key Risk Item In Weapons Systems Navy Mobile User Objective Satellite Communication System delays to the Joint Tactical Radio System, a set of software-defined radios causes advanced MUOS capabilities to be drastically underused… GAO GAO identified 42 programs at risk for cost & schedule 1.military requirements changes 2.software development challenges 3.workforce issues National Institute of Standards and Technology (NIST) Software defects cost nearly $60 Billion Annually 80% of development costs involve identifying and correcting defects Software, not Hardware or technology readiness levels were called out

Example: Project Cost Alone Is not The Cost of IT Failure (Source: HBR) Case Study: Levi Strauss $5M ERP deployment contracted Risks seemed small Difficulty interfacing with customer’s systems Had to shut down production Unable to fill orders for 3 weeks $192.5M charge against earnings on a $5M IT project failure “IT projects touch so many aspects of organization they pose a new singular risk”

An ROI Analysis of A New System: Should We Fund This? Can we do better? Will stakeholders tolerate a loss for 3 years? What is the risk?

Bad Estimates Are A Root Cause of Project Failure An estimate is the most knowledgeable statement you can make at a particular point in time regarding: Effort / Cost Schedule Staffing Risk Reliability Estimates more precise with progress A WELL FORMED ESTIMATE IS A DISTRIBUTION © 2012 Copyright Galorath Incorporated 7

Poor Project Performance Often Points Back To Inadequate Estimating When performance doesn’t meet the estimate, there are two possible causes: 1.Poor performance 2.Poor estimates “In the software world, we have ample evidence that our estimates stink, but virtually no evidence that people in general don’t work hard or intelligently enough.” - Tom DeMarco, Why Does Software Cost So Much? Why should we care: Good people working with poor estimates cause project challenges. © 2012 Copyright Galorath Incorporated 8

Greatest Leverage Is In Tackling Inherent and Structural Costs © 2010 Copyright Galorath Incorporated 9

© 2010 Copyright Galorath Incorporated 10

June 28, 2010 Mandate September 14, 2010 Guidance November 3, 2010 Implementation Five Specific Areas of Concern: Target Affordability and Control Cost Growth Reduce Non-Productive Processes and Bureaucracy Incentivize Productivity and Innovation in Industry Promote Real Competition Improve Tradecraft in Services Acquisition US Better Buying Power Initiatives

Affordability Initiatives With “Should Cost” and “Will Cost” Should Cost Performance Cost Initiatives (Applied practices & improvements) Cost Initiatives (Applied practices & improvements) Will Cost Performance Will Cost Performance - = © 2012 Copyright Galorath Incorporated 12 Many View Bottoms up estimates as the requirement for Should Cost / Will Cost Analysis But parametrics can do analysis faster as well as provide more tradeoffs

System Description (Parametrics Can Estimate More, Earlier) Adapted from CEBOK “If you can’t tell me what it is, I can’t tell you what it costs.” -Mike Jeffers “If you can tell me the range of what it might be, I can tell you the range of cost, schedule & probability.” -Dan Galorath

Affordability Process Step 1. Procure Key Performance Parameters that are inviolate Step 2. Identify Affordability Goals & Figures of Merit (development, life cycle, payback, ROI, NPV, kill ratio, Budget constraints, etc.) Step 3. Gather Requirements, Features, Performance Step 4. Define Baseline Alternatives Step 5. Perform Technical Design Analysis for Each Alternative Step 6. Perform Cost Schedule Analysis of Each Alternative Step 7. Assess Benefits Based on Figures of Merit Step 8. Perform Probabilistic Risk Analysis Step 9. Assess Alternatives & Select Optimal Alternative Step 10. Document Analysis and Lessons Learned

Requirements And Features Analysis Preliminary Design Process Human Factors Security Reliability Availability Survivability Supportability Testability Producibility Reuse Transportability Other Factors Alternative 1 Alternative 2 Alternative 3 Alternative 4 SEER Assessment Performance Schedule Risk Assessment Life Cycle Cost Selection Process OK? Optimized? Yes Cost Performance Schedule Risk Bottoms Up Estimation as Required NoIterate Should Cost: Trade Study Flow © 2012 Copyright Galorath Incorporated 15

Top-Down versus Bottom-Up Estimating (Adapted From: Project Management: the Managerial Process, McGraw Hill 2011) ConditionMacro Level / Parametric Micro level / Bottoms up Strategic decision making X Internal, Small Projectx Fixed price contractx Customer wants detailx Unstable scopex Bottoms up can be more accurate IF All work elements are captured and Inter WBS work is not forgotten Parametrics can be more accurate since it won’t forget items Use parametrics AT LEAST for analysis of alternatives © 2010 Copyright Galorath Incorporated 16

Complex Parametric Modeling: Repeatable Based on Program Parametric Estimating Mathematical relationships between anticipated characteristics and effort, schedule, risk, etc. & Technology, programmatics, cost, schedule, To estimate Development Production Effort / cost / Total Cost of Ownership Schedule Manufacture Modification Deployment of end items Much more Part of a repeatable process Traditional Estimating Many techniques from: Guessing Simple Analogy Spreadsheets Bottoms up Quantity x price Etc. Generally manual in nature… Most are not repeatable

18 What is likely to happen Feel lucky? Firm Fixed Price? Understand the risk before you commit!

Dealing With the “Problem of Assumptions” Assumptions are essential but… Incorrect assumptions can drive an estimate to uselessness Use an assumption verification process © 2012 Copyright Galorath Incorporated Identify assumptions 2. Rank order assumptions based on estimate impact 3. Identify high ranking assumptions that are risky 4. Clarify high ranking, high risk assumptions & quantify what happens if those assumptions change 5. Adjust range of SEER inputs to describe the uncertainty in assumptions

Key Points Viable affordability decisions yield project achievements Parametric Models provide affordability decision support Positive ROI when decisions use affordability modeling 20 © 2012 Copyright Galorath Incorporated 20