DA HIEF YSTEMS NGINEER 1 ASN (RDA) Chief Systems Engineer ASN (RDA) Chief Systems Engineer DoN PoPS to ICPM Core Common Metrics and OSD DAPS NDIA ICPM DoN PoPS to ICPM Core Common Metrics and OSD DAPS NDIA ICPM 16 November 2009

DA HIEF YSTEMS NGINEER 2 DoN Probability of Program Success (PoPS) to ICPM Core Common Metrics and Defense Acquisition Program Support (DAPS)  Requested to compare/contrast ICPM and PoPS  Also looked at DAPS to PoPS

DA HIEF YSTEMS NGINEER 3 3 Program Health Factors Metrics Criteria Naval Probability of Program Success (PoPS) v2.0  6 October 2008 PDASN Memo Naval PoPS Guidance: o Required for all DoN Gate Reviews and any time program health is discussed o Standard DoN method of representing the health of all ACAT Programs and any other programs subject to the DoN Acquisition process o Use the methods and tools described in the Naval PoPS Guidebook and supplemental Handbooks  Consistent means to assess program health, criteria adjusts as Program matures  Used for Portfolio Assessments and identification of Department wide systemic issues PoPS Criteria Scoring (Example)

DA HIEF YSTEMS NGINEER 4 4 ICPM Core Common Metrics  Under Development by NDIA ICPM  Shared view of program performance by customers and industry  Establishes critical behaviors / techniques for program success  Portfolio management Metrics Program Health Factors Requirements Stability & Completeness Predictive Measures of Program Performance Critical Skills Requirements Volatility TBD/TBR Burn down Risk / Opportunity Summary Funding Stability & Contract Health Technical Performance & Productivity Variance Risk & Opportunity Management Program Cost & Schedule Staffing & Critical Skills Program Funding Plan Program Funding Status Contract Change Volume TPM (Summary quick look) TPM Progress/Regress Burn Down Chart Software Productivity Cost Performance Index Schedule Performance Index Cum CPI vs. TCPIEAC Baseline Execution Index Requirements Traceability Risk / Opportunity relative to MR Risk Mitigation Plan Late/On-time Starts Critical Path Staffing Profile Supply Chain Performance Parts Demand Fulfillment Supplier Acceptance Rate Supplier Late Starts Top 8 IT Readiness Indicators Resources Physical Inventory, Maintenance, Records, Relief of Stewardship Action Tracking / Burndown Issues/Help Needed Customer Satisfaction CPAR Award Fee Product Performance  Transparency of data to avoid surprises o Clear Understanding of Requirements o Transparency of Technical Performance o Cost / Schedule Performance, Risk Management, Resources, and Supplier Performance o Help Needed Culture – Across All Program Lines and Open Communication Relative to Issues o Clear, Continuous Feedback from the Customer Categories

DA HIEF YSTEMS NGINEER 5 Defense Acquisition Program Support (DAPS)  Defines the PSR approach to assess defense system development programs by the OSD Program Support Team. o not a checklist to be followed rigidly, nor must each assessment address all assessment areas and sub-areas listed. o Tailorable to ensure important areas are not overlooked and comprehensively examined o Includes an extensive list of programmatic and technical factors that are used to build the review plan. Mission Capabilities / Req’ts Generation DoD DAPS Readiness Levels CONOPs Analysis of Alternatives Design Considerations Resources Management Technical Process Program Performance Special Interest Areas Budget Sufficiency & Phasing Staffing Level Program Schedule Overview Acquisition Strategy Knowledge-based Decisions and Milestones Program & Project Management Contracting Effectiveness Suitability Survivability Production Capabilities Requirements Development Technical Baseline Engineering Tools Software Design Verification Supportability Planning Program Assessment Areas Criteria Sub-Areas Pre-Milestone A 3.0 Management 3.1 Acquisition Strategy Credibility - The program manager (PM) is developing a credible Acquisition Strategy that will provide the basis for meeting program objectives and therefore will be an aid in gaining program acceptance and support Q1 : How is the Acquisition Strategy realistic? Q2 : Does the program face any of the following pressures, which work against the realism of the Acquisition Strategy? Q3: How is the Acquisition Strategy stable? Q4: Are any of the potential causes of instability to the Acquisition Strategy present? Etc.

DA HIEF YSTEMS NGINEER 6 Key Assessment Areas / Factors ICPM versus PoPS version 2.0 ICPM Categories DoN PoPS Factors Requirements Stability & Completeness Program Requirements Customer Satisfaction Program Planning & Execution Technical Performance & Productivity Variance Risk & Opportunity Management Program Cost & Schedule Supply Chain Performance Resources Funding Stability & Contract Health Program Resources Staffing & Critical Skills Issues/Help Needed External Influencers ICPMDoN PoPS  10 Categories  4 Factors  30 Metrics  18 Metrics  Criteria* *Particular Criteria and number is dependent on Life Cycle Phase All ICPM Categories and Metrics are included in PoPS Factors and Metrics

DA HIEF YSTEMS NGINEER 7 ICPMDoN POPS Requirements Stability and Completeness CONOPs Program Requirements Requirements VolatilityScope Evolution TBD/TBR Burn down Requirements TraceabilityParameter Status Funding Stability & Contract Health Program Funding PlanTotal Ownership Cost Program Planning & Execution Contract Change VolumeContract Execution Program Funding StatusProgram Resources Factor- Budget Technical Performance & Productivity Variance TPM (Summary quick look)Technical Maturity Program Planning & Execution TPM Progress/Regress Burn Down Chart Software ProductivitySoftware Risk and Opportunity Management Risk/opportunity SummaryGovernment Program Office Performance Program Planning & Execution Risk/Opportunity relative to MR Risk Mitigation Plan Program Cost and Schedule Cost Performance IndexContract Planning and Execution Program Planning & Execution Schedule Performance Index Statistical EACs Cum CPI vs TCPIEAC Baseline Execution Index Late/On-Time Starts Critical PathAcquisition Management Summary ICPM versus PoPS

DA HIEF YSTEMS NGINEER 8 ICPMDoN POPS Staffing & Critical Skills Critical SkillsManning Program Resources Staffing Profile Supply Chain Performance Parts Demand FulfillmentContract Planning & Execution Program Planning & Execution Supplier Acceptance Rate Supplier Late Starts Resources Top 8 IT Readiness IndicatorsSoftware Program Planning & Execution Physical Inventory, Maintenance, Records, Relief of Stewardship Sustainment Government Program Office Performance Customer Satisfaction CPARIndustry and Company Assessment Program Planning & Execution Award FeeContract Planning and Execution Product PerformanceTechnical Maturity Test and Evaluation Program Requirements Factor – Parameter Status Issues/Help Needed Action Tracking /Burndown Assessed at DoN SETRs and Gate Reviews Summary (cont.) ICPM versus PoPS

DA HIEF YSTEMS NGINEER 9 Key Assessment Areas / Factors DAPS versus PoPS version 2.0 OSD DAPS Areas DoN PoPS Factors Mission Capabilities / Requirements Generation Program Requirements ResourcesProgram Resources ManagementExternal Influences Program Planning & Execution Technical Process Program Performance Special Interest Areas – Technical Readiness OSD DAPSDoN PoPS  6 Areas  4 Factors  22 Sub-Areas  18 Metrics  59 Factors - Criteria* Questions* - Criteria* *Particular Criteria and number is dependent on Life Cycle Phase

DA HIEF YSTEMS NGINEER 10 OSD DAPSDoN POPS Mission Capabilities / Requirements Generation CONOPs Program Requirements Analysis of AlternativesScope Evolution CapabilitiesParameter Status Resources Program Schedule OverviewAcquisition Management Program Planning & Execution Budget Sufficiency & PhasingTotal Ownership Cost Estimating Budget Program Resources Staffing LevelManning Management Acquisition StrategyFit in Vision External Influences Program Advocacy Acquisition Management Program Planning & Execution Knowledge-Based Decisions & Milestones ContractingContract Planning & Execution Industry / Company Assessment Program & Project ManagementGovernment Program Office Performance Interdependencies External Influences Technical Process Design Considerations Program Planning & Execution Requirements DevelopmentTechnical Maturity Technical Baseline Engineering Tools Software Design VerificationTest and Evaluation Supportability PlanningSustainment Special Interest Areas Readiness LevelTechnical Maturity Program Performance Effectiveness Test and Evaluation SuitabilitySustainment SurvivabilityTest and Evaluation ProductionIndustry / Company Assessment Summary DAPS versus PoPS DoN Gate Review and DoN SETR Process

DA HIEF YSTEMS NGINEER 11 Summary  ICPM and DoN PoPS o ICPM appears to serve Executive Level discussions o ICPM assessment factors similar to those in PoPS o However differences in Criteria may be warranted  ICPM – Health of Program from Contractor view  PoPS – Health of Program from Government view  DAPS and DoN PoPS o DAPS is a mix of Executive and Management Level information o DAPS factors covered by  DoN Gate Reviews Core Criteria and PoPS (Exec / Management Level)  Systems Engineering Technical Reviews (SETRs) (Management / Detail Level)  DoN PoPS o Executive Level with input from Management / Detail Reviews (SETR) o Used continuously for life of Program o As input for Major Decision Reviews (Gate and Milestone) o Applicable to All ACAT Programs

DA HIEF YSTEMS NGINEER 12 Back-up

DA HIEF YSTEMS NGINEER 13 Details of Core Criteria and Templates by Gate - URL: ents/Forms/AllItems.aspx?RootFolder=%2fasnrda%2fasn%5frda%5fpops%2fpops20de v%2fStep%202%20Draft%20Documents%2f0%20Draft%20Core%20and%20Briefing% 20Templates%20in%20Exec%20Staffing&FolderCTID=&View=%7b7AEE306D%2d7B7 1%2d4326%2d999E%2dDEEC9A1DF241%7d Details of PoPS Criteria and Templates by Gate - URL: cuments/Forms/AllItems.aspx?RootFolder=%2fasnrda%2fasn%5frda%5fpops%2f pops20dev%2fStep%202%20Draft%20Documents%2fProposed%20PoPS%20v2% 20Criteria%20and%20Templates%20in%20Exec%20Staffing&FolderCTID=&View= %7b7AEE306D%2d7B71%2d4326%2d999E%2dDEEC9A1DF241%7d cuments/Forms/AllItems.aspx?RootFolder=%2fasnrda%2fasn%5frda%5fpops%2f pops20dev%2fStep%202%20Draft%20Documents%2fProposed%20PoPS%20v2% 20Criteria%20and%20Templates%20in%20Exec%20Staffing&FolderCTID=&View= %7b7AEE306D%2d7B71%2d4326%2d999E%2dDEEC9A1DF241%7d

DA HIEF YSTEMS NGINEER 14 ICPM Requirements Stability and Completeness Category DoN POPS Program Requirements Factor Metric CONOPs Requirements Volatility Scope Evolution TBD/TBR Burn down Requirements TraceabilityParameter Status Requirements Assessment ICPM versus PoPS

DA HIEF YSTEMS NGINEER 15 ICPM Funding Stability & Contract Health Category DoN POPS Program Planning & Execution Factor Metric Program Funding PlanTotal Ownership Cost Estimating Contract Change VolumeContract Execution Program Funding Status Program Resources Factor - Budget Resources Assessment ICPM versus PoPS

DA HIEF YSTEMS NGINEER 16 ICPM Technical Performance & Productivity Variance Category DoN POPS Program Planning & Execution Factor Metric TPM (Summary quick look) Technical Maturity TPM Progress/Regress Burn Down Chart Software Productivity Software Resources Assessment ICPM versus PoPS

DA HIEF YSTEMS NGINEER 17 ICPM Risk & Opportunity Management Category DoN POPS Program Planning & Execution Factor Metric Risk / Opportunity Summary Government Program Office Performance Risk / Opportunity relative to MR Risk Mitigation Plan Resources Assessment ICPM versus PoPS

DA HIEF YSTEMS NGINEER 18 ICPM Program Cost & Schedule Category DoN POPS Program Planning & Execution Factor Metric Cost Performance Index Contract Planning and Execution Schedule Performance Index Statistical EACs Cum CPI vs. TCPIEAC Baseline Execution Index Late/On-time Starts Critical Path Acquisition Management Resources Assessment ICPM versus PoPS

DA HIEF YSTEMS NGINEER 19 ICPM Staffing & Critical Skills Category DoN POPS Program Resources Factor Metric Critical SkillsManning Staffing Profile Resources Assessment ICPM versus PoPS

DA HIEF YSTEMS NGINEER 20 ICPM Supply Chain Performance Category DoN POPS Program Planning & Execution Factor Metric Parts Demand Fulfillment Contract Planning & Execution Supplier Acceptance Rate Supplier Late Starts Resources Assessment ICPM versus PoPS

DA HIEF YSTEMS NGINEER 21 ICPM Resources Category DoN POPS Program Planning & Execution Factor Metric Top 8 IT Readiness IndicatorsSoftware Physical Inventory, Maintenance, Records, Relief of Stewardship Sustainment Government Program Office Performance Resources Assessment ICPM versus PoPS

DA HIEF YSTEMS NGINEER 22 ICPM Issues/Help Needed Category DoN POPS Metric Action Tracking / Burndown Resources Assessment ICPM versus PoPS Assessed at DoN Gate Reviews and SETRs

DA HIEF YSTEMS NGINEER 23 ICPM Customer Satisfaction Category DoN POPS Program Planning & Execution Factor Metric CPARIndustry and Company Assessment Award FeeContract Planning & Execution Product Performance Technical Maturity Test and Evaluation Program Requirements Factor - Parameter Status Resources Assessment ICPM versus PoPS

DA HIEF YSTEMS NGINEER 24 OSD DAPS Mission Capabilities / Requirements Generation Area DoN POPS Program Requirements Factor Sub AreaFactorMetric CONOPs Mission Description CONOPs FoS / SoS Dependencies / Interfaces Analysis of Alternatives Validity and Currency Scope Evolution Linkages and Traceability Capabilities Reasonableness, Stability, and Testability Parameter Status KPP / KSA Requirements Assessment DAPS versus PoPS

DA HIEF YSTEMS NGINEER 25 OSD DAPS Resources Area DoN POPS Program Resources Factor Sub AreaFactorMetric Budget Sufficiency & Phasing Program Funding and Allocation Budget Continuity and Stability Program Planning & Execution Factor - Total Ownership Cost Estimating Staffing Level Sufficiency of Numbers and Qualifications Manning Continuity and Stability Program Schedule Overview Viability Program Planning & Execution Factor - Acquisition Management Constraints and Dependencies Resources Assessment DAPS versus PoPS

DA HIEF YSTEMS NGINEER 26 OSD DAPS Management Area DoN POPS Program Planning & Execution Factor Sub AreaFactorMetric Acquisition Strategy Credibility Acquisition Management Acceptability External Influences - Program Advocacy External Influences - Fit in Vision Knowledge-Based Decisions & Milestones Statuary & Regulatory Compliance & Guidance Entrance & Exit/Success Criteria Certifications Program & Project Management Program/ Plan/Schedule Government Program Office Performance WBS Management Structure & Communications Management Methods, Metrics, and Techniques Information Management (IDE, IT, Data Rights) Management of Dependencies & External Interfaces (FoS/SoS)) External Influences - Interdependencies Contracting Prime Contractor Management Contract Planning & Execution Subcontractor Management Value Engineering Industry / Company Assessment Management Assessment DAPS versus PoPS Assessed at DoN Gate Reviews and SETRs

DA HIEF YSTEMS NGINEER 27 OSD DAPS Technical Process Area DoN POPS Program Planning & Execution Factor Sub AreaFactorMetric Design Considerations System Assurance Technology Protection Modular Open Systems Approach Architecture Environment, Safety, and Occupational Health Spectrum Management Sustainment as a Design Consideration Corrosion HSI Requirements Development Analysis and Decomposition Management of Requirements Technology and Maturity Integration Technical Maturity Trade Studies and Approaches Technical Baseline Technical Review Planning Configuration Management Baseline Stability Technical Process Assessment DAPS versus PoPS Assessed at DoN SETRs Assessed at DoN Gate Reviews and SETRs

DA HIEF YSTEMS NGINEER 28 OSD DAPS Technical Process Area DoN POPS Program Planning & Execution Factor Sub AreaFactorMetric Engineering Tools Systems and Software Engineering (SSE) Tools M&S Tools Producibility and Production Planning Tools Software Software Development Plan Software Estimation Design Verification Test and Evaluation Plan Test and Evaluation Verification Correlation Supportability Planning Acquisition Logistics Sustainment Performance-Based Logistics (PBL) Sustainment Technical Process Assessment DAPS versus PoPS, Con’t Assessed at DoN Gate Reviews and SETRs

DA HIEF YSTEMS NGINEER 29 OSD DAPS Program Performance Area DoN POPS Program Planning & Execution Factor Sub AreaFactorMetric Effectiveness Design Capabilities Assessment Test and Evaluation Technical Maturity Special Interest Areas - Readiness Level Suitability Reliability Assessment Sustainment Availability Assessment Maintainability Assessment Survivability Live Fire Test and Evaluation Assessment Test and Evaluation Production Assessed Manufacturing Industry / Company Assessment Assessed Quality Program Performance Assessment DAPS versus PoPS