1. 1 In H/ACAT Programs The Application Of Risk In Metrics Used For Department of Navy Weapon Systems Acquisition (May 9, 2002) Larry Szutenbach OASN(RD&A) PP&R SZUTENBACH.LARRY@HQ.NAVY.MIL

2. 2 In H/ACAT Programs This Presentation Will Briefly Touch on the Use of Risk in Our: Program Summary Documents (PSDs) Program Performance Metrics Defense Acquisition Executive Summary (DAES) Integrated Baseline Reviews Consideration of Scorecards

3. 3 In H/ACAT Programs Risk as used in Defense: The potential for a Two characteristics: oProbability of occurrence oConsequences Use of risk in our metrics: Often intentionally mix performance with risk, but it can be important to recognize the difference. Sometimes risk is indirectly addressed, such as in the EVM “Bullseye” chart negative future reality.

4. 4 In H/ACAT Programs Program Assessments and PSDs Program Assessment Indicators/Program Manager’s Assessment (indicate Green/Yellow/Red, Program managers comments, an assessment based on current funding profile and current approved Acquisition Program Baseline) Program Assessment Indicators Performance CharacteristicsGreenScheduleGreen Test & EvaluationGreenContractsGreen Logistics Reqts & Readiness Obj GreenProductionGreen CostGreenManagement StructureGreen FundingGreenInteroperability Green Managers’ Assessment (for each Assessment Indicator rated Green Advisory, Yellow, Yellow Advisory, or Red). 1.Performance Characteristics. 2. Test and Evaluation. 3. Logistics Requirements and Readiness Objectives. 4. Cost Performance. 5. Funding. 6.Schedule Performance. 7.Contracts. 8.Production. 9.Management Structure. DAES Report

5. 5 In H/ACAT Programs ACATPM CODEPROGRAM TITLEAPB DATECOSTSCHPERFOVERALL III PMA201GBU-24 E/BOCT 99 IIIPMA208IMPROVED TACTICAL AIR LAUNCHED DECOYDEC 99 IIIPMA242HARM BLOCK VI (IHUP)JUL 99 IIIPMA281AFLOAT PLANNING SYS (APS) SEP 99 IIIPMA281JOINT SERV IMAGING PROC SYS NAVY (JSIPS) SEP 98 IIIPMA282TACTICAL TOMAHAWK WEAPONS CONT SYS - IIIPMA282ADV TOMAHAWK WPN CONTROL SYS (ATWCS) DEC 99 IVMPMA208SUPERSONIC SEA SKIMMING TRGT (SSST) JUN 99 IVMPMA20821 ST CENTURY AERIAL TARGET (TARGET 21) DEC 99 IVMPMA208BQM-74E MOBILE SUBSONIC AERIAL TGTDEC 99 IVMPMA208VANDAL EXTENDED RANGE (EER) MAY 97 IVMPMA208AQM-37C SUPERSONIC TARGET SYSTEMDEC 99 IVMPMA208QF-4S FULL SCALE A/C TRGT SYS DEC 99 IVTPMA258AN / AWW-13 ADVANCED DATA LINK POD AUG 91 ACAT III and IV Program Status Mr. Randel H. Stone Date of Review: 16 Nov 2000 PEO (W) ACAT III & I V

6. 6 In H/ACAT Programs Color Coded Rating System For DAES And DoN PSDs and Program Performance Metrics The DAES rating system [adapted for use in PSDs and the summary charts in program performance metrics] is intended to be used as an early warning report of both potential and actual problems. In this regard, the system depends on the Program Manager exercising sound judgment in assessing the program’s status. Rating an indicator as "on-track," solely because the APB might be "on-track" is counterproductive and leads to downstream problems. If the problem is a potential one, the PM should clearly note this fact, so there is no doubt that this is an advisory and that the situation is being properly managed. Early reporting of potential problems and that corrective action plans are underway is essential.

7. 7 In H/ACAT Programs Color Coded Rating System For DAES, and DoN PSDs and Program Performance Metrics Summary Charts GREEN : On-Track: All aspects of the program are progressing satisfactorily. Some minor problem(s) may exist, but solutions are available. Costs are expected not to exceed approved funding levels and not to exceed contract target costs by more than 5%. YELLOW: Potential or Actual Problem: Some event, action or delay has occurred that impairs progress against major objectives in one or more segments of the program. In the case of a potential risk to a major program objective or acquisition program baseline, the Program Manager should state this distinction in Section 3 (Program Manager’s Comments). Early reporting is encouraged. RED: Major Weakness (Red): Some event, action, or delay has occurred that seriously impedes successful accomplishment of one or more major program objectives. ADVISORY (GREEN, YELLOW OR RED): Advisory indicates the program is either assessed to be between ratings, or is moving from one rating to another, thus this provides advance notification of shifting status. Mixing performance and risk

8. 8 In H/ACAT Programs Program Risk Assessment This is one of the most commonly used charts for depicting risk. 5 Likelihood 5 4 3 2 1 Consequence 43 2 1 High Medium Low Risk Advisory Board Mr. X, PMA xxx Ms. Y, XYZ Corp Ms. Z, DASN YY etc A brief description of Issue # 5 and rationale for its rating. Approach to remedy/mitigation A brief description of Issue # 5 and rationale for its rating. Approach to remedy/mitigation A brief description of Issue # 1 and rationale for its rating. Approach to remedy/mitigation A brief description of Issue # 1 and rationale for its rating. Approach to remedy/mitigation A brief description of Issue # 3 and rationale for its rating. Approach to remedy/mitigation A brief description of Issue # 3 and rationale for its rating. Approach to remedy/mitigation A brief description of Issue # 4 and rationale for its rating. Approach to remedy/mitigation A brief description of Issue # 4 and rationale for its rating. Approach to remedy/mitigation A brief description of Issue # 2 and rationale for its rating. Approach to remedy/mitigation A brief description of Issue # 2 and rationale for its rating. Approach to remedy/mitigation A brief description of Issue # 6 and rationale for its rating. Approach to remedy/mitigation A brief description of Issue # 6 and rationale for its rating. Approach to remedy/mitigation Shows the risks Shows the plan to manage risk

9. 9 In H/ACAT Programs : Overall Assessment 1: Training 2: Support Equipment 3: Publications 4: Facilities 5: Maintenance Concept 6: Supply Support 7: MTBF Logistics Areas Logistics Areas (examples) Logistics Risk Assessment Consequence Likelihood 5 4 3 2 1 543 2 1 4 1 2 6 5 3 7 Low Risk Medium RiskHigh Risk Program Acronym ACAT XX Date of Review: dd mmm yyCAPT J. Doe, PMS-499 PEO XXX RISK # 4 Brief description of Issue and rationale for its rating. Approach to remedy/mitigation. Risk mitigation funding. RISK #5 Brief description of Issue and rationale for its rating. Approach to remedy/mitigation. Risk mitigation funding. RISK # 6 Brief description of Issue and rationale for its rating. Approach to remedy/mitigation. Risk mitigation funding.

10. 10 In H/ACAT Programs Risk Assessment: Address each issue which might affect the success of the program. Likelihood –Negligible - One can reasonably assume no occurrence (<10%) –Unlikely - Occurrence possible but less than likely (10-40%) –Likely - significant chance of occurrence (40-65%) –Highly Probable - Very high chances of occurrence (65-90%) –Near Certainty - Assume and anticipate occurrence (>90%) Consequence –Marginal - Remedy will cause disruption to the program –Significant - Shorts a significant mission need –Serious - Shorts a critical mission need but expect no breech –Very Serious - Potentially fails a KPP or OPEVAL. –Catastrophic - Jeopardizes an exit criterion of current Consequence  Likelihood  5 4 3 2 1 543 2 1 4 1 2 6 5 3 7 Low Risk Medium RiskHigh Risk Definitions Of “Likelihood” And “Consequence” When Constructing This Chart

11. 11 In H/ACAT Programs Budget Program Manager/Office______________ Date of Review_________ Programmed (%Obligated) Activity FY99 (%) FY00 (%) FY 01 FY 02 FY 03 FY 04 FY05 FY06 Display each significant task G Solvency This addresses whether there enough money budgeted/planned over the course of the life of the program to accomplish the program goals and objectives

12. 12 In H/ACAT Programs This “Solvency” addresses how well the PM is able to cover the cost of the PROGRAM within funds already allocated to the program. The color ratings used generally follow these themes: GREEN-can handle within existing resources without descoping or affecting KPPs YELLOW-can resolve within existing budget by out-year transfer of funds between appropriations or descoping, and will request changing KPPs RED-shortfall in the current execution or next fiscal year with no available offsets or workarounds and will request Congressional approval be sought for a current year reprogramming; in short, program requires outside help. G Program Solvency

13. 13 In H/ACAT Programs $100 111% 56%$50 100% $90 122%$110 00% 04/00 04/0208/02 04/98 October 2000 Guidance – Example Master Chart Contract Performance for [give short contract title] Briefed: FEB 02 YYMMDD Nxxxxx-YY-CxxxxContractor Name [Prime or Significant Sub] PEO and Program Manager/PMS-XXX Program Acronym ACAT XX Date of Last Rebaselining: JAN00 Number of Rebaselinings: 1 Date of Next Rebaselining: MMM YY Y Solvency KTR’s EAC: 104M Date of Last Award Fee: MMM YY Date of Next Award Fee: MMM YY 1.18 PM’s EAC Total Spent Total Calendar Schedule $M 0 % TAB BAC ACWP EAC EV % Spent 50% [TCPI EAC = 0.76] CV = $2.0 M SV = $2.9 M 100%108% 01/00 SPI 1.18 Ahead of Schedule and Underspent Behind Schedule and Underspent Ahead of Schedule and OverspentBehind Schedule and Overspent 0.940 0.960 0.82 0.86 0.90 0.94 0.98 1.02 1.06 1.10 1.14 0.820.860.900.940.981.021.061.101.14 CPI 01/98 10/97 07/97 04/97 04/00 03/00 02/00 01/00 10/99 07/99 04/99 1/99 10/98 07/98 04/98 01/00 42% PM’s Projected Performance at Completion for CPI and Duration.

14. 14 In H/ACAT Programs This “Solvency” addresses how well the PM is able to cover the cost of the CONTRACT within funds already allocated to the program. The color ratings used generally follow these themes: GREEN-can handle within existing resources without descoping or affecting KPPs YELLOW-can resolve within existing budget by out-year transfer of funds between appropriations or descoping, and changing KPPS will be considered. RED-shortfall in the current execution or next fiscal year with no available offsets or workarounds. the program office has no way to resolve the funding shortfall and needs assistance from the sponsor, DASN, ASN, PEO or others. Y Contract Solvency

15. 15 In H/ACAT Programs This Is Not a Metrics Chart, It Is an Illustration of Aggressive Bidding and A Changing Contract Original Contract - APR 00 [Same status at OCT 00] $68 Contract Target Cost PM’s Est Cost $100 Contractor’s Est Cost $85 PM Allocation for Contract $112 Contractor’s Est Cost $109 $100 Current Status APR 01 Contract Target Cost $112 PM Allocation for Contract PM’S Est Cost $130 Y Solvency Contract performance is poor, but yields little budget risk Rebaselined/Renegotiated Contract - JAN 01 Contract Target Cost PM’s Est Cost $115 Contractor’s Est Cost $100 $100 $112 PM Allocation for Contract Y Solvency G OCT 00 Contract performance is poor, but yields little budget risk

16. 16 In H/ACAT Programs CAPT J. Doe, PMS-499 Program Acronym ACAT XX CPARS/IPARS/Award Fee Matrix PEO XXX Date of Review: ddmmmyy At times, an interesting phenomenon occurs: Contract past performance cost ratings closely follow the “Solvency” rather than actual performance.

17. 17 In H/ACAT Programs Major Weakness Some event seriously delaying major program objective; breach of program baseline or unit cost. Potential or Actual Problem Some event has occurred that impairs progress against major objectives of the program. “On-Track.” [CPI >0.95] Program progressing satisfactorily. Contract costs not expected to exceed contract target by > 5%, thus not an issue. NOT USED DAES Not for release to contractors. Brings up issues within Defense. Applies to Numerous Indicators * ACAT I & II Program Contracts (EVM) 5 OCT 2000 Applies to Cost and Schedule * PDUSD(A&T) CPARS/IPARS 24 AUG 1999 In DoN CPARS/IPARS Guidance Applies to Numerous Indicators Color Rating CPI and SPI < 0.90 Unsatisfactory Does not meet most requirements and timely recovery unlikely. Red CPI and SPI > 0.90 Marginal Does not meet some requirements; serious problem with no corrective action identified. Yellow CPI and SPI > 1.0 Satisfactory Meets requirements; some minor problems, which were corrected Green CPI and SPI > 1.1 Very Good - Exceeds some requirements; some minor problems, which were corrected. Purple CPI and SPI > 1.15 and exceed many requirements, resolve all problems. Exceptional Exceed many requirements; resolve all problems. Blue For Reference: Comparison of Color Ratings (We have attempted to normalize these as far as we can)

18. 18 In H/ACAT Programs Risk Assessments In Integrated Baseline Reviews For Contracts Managed At The Program Manager Level No Consistent Form And Format For The IBRs. Defense/Industry Developing Joint IBR Guide Joint Strike Fighter IBR Offers A Good Example to Emulate

19. 19 In H/ACAT Programs Extracts of Templates Used In The Integrated Baseline Review For The Joint Strike Fighter

20. 20 In H/ACAT Programs (Sample) WBS 1346 Fuel Sys Interview Summary The Team believes that this cost account can be successfully accomplished within cost and schedule at acceptable risk Strengths –good discussion; CAM well-prepared –comprehensive plan linkages well defined Weakness –Resource availability Overall Assessment Dialogue Highlights Concerns –Staffing availability –Hardware fabrication cycle with new vendors (different than Phase ll) –Communication between IPT’s - late data / data updates could cause redos Risks –Resource availability/timing –Design maturity / hardware schedule WBS 1300 Assessment Area Score Technical Schedule Cost Resources Manageme nt Processes

21. 21 In H/ACAT Programs IBR Discussion Evaluation Summary TECHSCHEDCOST RESOURCE MGT PROCESS Comments [Including Risk Costs] (WBS)(Comments) (WBS)(Comments) (WBS)(Comments) (WBS)(Comments) (WBS)(Comments) (WBS)(Comments) (WBS)(Comments) (WBS)(Comments) (WBS)(Comments) (Level 3 IPT)

22. 22 In H/ACAT Programs Our Guidance in Defense for Risk Systems Engineering risk management calls for risk: oAvoidance oControl oAssumption22 oTransfer None of these imply an opportunity. All of these focus on possible negative future occurrences. The basic objective of risk management in systems engineering is to reduce all risk to an acceptable level.

23. 23 In H/ACAT Programs Industry Takes Risk a Step Farther Industry sees the potential Compensaton of Industry managers is often tied to the risk of success, which is tracked in scorecards. In DASN(PP&R) we are reviewing industry use of scorecards and seeing if we can reasonably attach our metrics to a scorecard appropriate for the government and reaching for the risk of success. risk of success.