1. The Application Of Risk In Metrics Used For Department of Navy Weapon Systems Acquisition (May 9, 2002)
Larry Szutenbach
OASN(RD&A) PP&R

2. This Presentation Will Briefly Touch on the Use of Risk in Our:
Program Summary Documents (PSDs)
Program Performance Metrics
Defense Acquisition Executive Summary (DAES)
Integrated Baseline Reviews
Consideration of Scorecards

3. negative future reality. Risk as used in Defense: The potential for a
Two characteristics:
Probability of occurrence
Consequences
Use of risk in our metrics:
Often intentionally mix performance with risk, but it can be important to recognize the difference.
Sometimes risk is indirectly addressed, such as in the EVM “Bullseye” chart
negative
future
reality.

4. Program Assessments and PSDs
Program Assessment Indicators/Program Manager’s Assessment (indicate Green/Yellow/Red, Program managers comments, an assessment based on current funding profile and current approved Acquisition Program Baseline)
Program Assessment Indicators
Performance Characteristics Green
Schedule Green
Test & Evaluation Green
Contracts Green
Logistics Reqts & Readiness Obj Green
Production Green
Cost Green
Management Structure Green
Funding Green
Interoperability Green
Managers’ Assessment (for each Assessment Indicator rated Green Advisory, Yellow, Yellow Advisory, or Red).
DAES Report
Schedule Performance.
Contracts.
Production.
Management Structure.
Performance Characteristics.
 Test and Evaluation.
 Logistics Requirements and Readiness Objectives.  
Cost Performance.
Funding.

5. ACAT III and IV Program Status
Mr. Randel H. Stone Date of Review: 16 Nov 2000
PEO (W)
ACAT III & I V
ACAT
PM CODE
PROGRAM TITLE
APB DATE
COST
SCH
PERF
OVERALL
III PMA201 GBU-24 E/B OCT 99
III PMA208 IMPROVED TACTICAL AIR LAUNCHED DECOY DEC 99
III PMA242 HARM BLOCK VI (IHUP) JUL 99
III PMA281 AFLOAT PLANNING SYS (APS) SEP 99
III PMA281 JOINT SERV IMAGING PROC SYS NAVY (JSIPS) SEP 98
III PMA282 TACTICAL TOMAHAWK WEAPONS CONT SYS -
III PMA282 ADV TOMAHAWK WPN CONTROL SYS (ATWCS) DEC 99
IVM PMA208 SUPERSONIC SEA SKIMMING TRGT (SSST) JUN 99
IVM PMA208 21ST CENTURY AERIAL TARGET (TARGET 21) DEC 99
IVM PMA208 BQM-74E MOBILE SUBSONIC AERIAL TGT DEC 99
IVM PMA208 VANDAL EXTENDED RANGE (EER) MAY 97
IVM PMA208 AQM-37C SUPERSONIC TARGET SYSTEM DEC 99
IVM PMA208 QF-4S FULL SCALE A/C TRGT SYS DEC 99
IVT PMA258 AN / AWW-13 ADVANCED DATA LINK POD AUG 91

6. Color Coded Rating System For DAES
And DoN PSDs and Program Performance Metrics
The DAES rating system [adapted for use in PSDs and the summary charts in program performance metrics] is intended to be used as an early warning report of both potential and actual problems.
In this regard, the system depends on the Program Manager exercising sound judgment in assessing the program’s status.
Rating an indicator as "on-track," solely because the APB might be "on-track" is counterproductive and leads to downstream problems.
If the problem is a potential one, the PM should clearly note this fact, so there is no doubt that this is an advisory and that the situation is being properly managed.
Early reporting of potential problems and that corrective action plans are underway is essential.

7. DoN PSDs and Program Performance Metrics Summary Charts
Color Coded Rating System For DAES, and
DoN PSDs and Program Performance Metrics Summary Charts 
GREEN:
On-Track: All aspects of the program are progressing satisfactorily. Some minor problem(s) may exist, but solutions are available. Costs are expected not to exceed approved funding levels and not to exceed contract target costs by more than 5%.
YELLOW:
Potential or Actual Problem: Some event, action or delay has occurred that impairs progress against major objectives in one or more segments of the program.
In the case of a potential risk to a major program objective or acquisition program baseline, the Program Manager should state this distinction in Section 3 (Program Manager’s Comments). Early reporting is encouraged.
RED:
Major Weakness (Red): Some event, action, or delay has occurred that seriously impedes successful accomplishment of one or more major program objectives.
 ADVISORY (GREEN, YELLOW OR RED):
Advisory indicates the program is either assessed to be between ratings, or is moving from one rating to another, thus this provides advance notification of shifting status.
Mixing performance and risk
Mixing performance and risk

8. Program Risk Assessment
This is one of the most commonly used charts for depicting risk.
Program Risk Assessment 5
Likelihood 4 3 2 1
Consequence
High
Medium
Low
Risk Advisory Board
Mr. X, PMA xxx
Ms. Y, XYZ Corp
Ms. Z, DASN YY
etc
A brief description of Issue # 5 and rationale for its rating.
Approach to remedy/mitigation
A brief description of Issue # 1 and rationale for its rating.
A brief description of Issue # 3 and rationale for its rating.
A brief description of Issue # 4 and rationale for its rating.
A brief description of Issue # 2 and rationale for its rating.
A brief description of Issue # 6 and rationale for its rating.
Risk Assessment: Each issue which might affect the success of the program (technical, schedule, fiscal, etc) needs to be identified and assessed as to likelihood and consequences (performance or financial) of occurrence. The following is a rough key to scoring:
Likelihood (1)Negligible - One can reasonably assume no occurrence (<10%)
(2)Unlikely - Occurrence possible but less than likely (10-40%)
(3)Likely - significant chance of occurrence (40-65%)
(4)Highly Probable - Very high changes of occurrence (65-90%)
(5)Near Certainty - Assume and anticipate occurrence (>90%)
Consequences (1)Marginal - Remedy will cause disruption to the program
(2)Significant -.Shorts an significant mission need
(3)Serious - Shorts a critical mission need but expect no breech
(4)Very Serious - Potentially fails a KPP or OPEVAL.
(5)Catastrophic - Jeopardizes an exit criterion of current Phase
If the assessment is done formally by a standing advisory board (good program management) then please list the members and their affiliations. Each issue box should contain a brief statement of intended approach. Presenter should be prepared for more detailed discussion on these issues and alternative courses of action.
Shows the plan to manage risk
Shows the risks

9. Logistics Risk Assessment
PEO XXX
Program Acronym ACAT XX
Logistics Risk Assessment
CAPT J. Doe, PMS-499
Date of Review: dd mmm yy
Consequence
Likelihood 5 4 3 2 1 6 7
Low Risk
Medium Risk
High Risk
: Overall Assessment
1: Training
2: Support Equipment
3: Publications
4: Facilities
5: Maintenance Concept
6: Supply Support
7: MTBF
Logistics Areas (examples)
Logistics Risks
Indicate data points for the major logistics planning areas and provide a brief description/mitigation plan for those items in the RED and YELLOW blocks. Logistics planning areas include, but are not limited to: supply support, training (including training equipment), support equipment (including test equipment), facilities and publications. Indicate your overall logistics assessment with a triangle. Consider system reliability (Mean Time Between Failures) and maintainability (Maintenance Man Hours per Operating Hour) in positioning the triangle. If the system is not on track to achieve reliability and maintainability targets, especially reliability, logistics support will be negatively impacted.
REMINDER: refer to Dr. Buchanan’s definitions WRT “likelihood” and “consequence” when constructing this chart - see below:
Likelihood
(1) Negligible - One can reasonably assume no occurrence (<10%)
(2) Unlikely - Occurrence possible but less than likely (10-40%)
(3) Likely - significant chance of occurrence (40-65%)
(4) Highly Probable - Very high chances of occurrence (65-90%)
(5) Near Certainty - Assume and anticipate occurrence (>90%)
Consequence
(1) Marginal - Remedy will cause disruption to the program
(2) Significant - Shorts a significant mission need
(3) Serious - Shorts a critical mission need but expect no breech
(4) Very Serious - Potentially fails a KPP or OPEVAL.
(5) Catastrophic - Jeopardizes an exit criterion of current phase
RISK # 4
Brief description of Issue and rationale for its rating.
Approach to remedy/mitigation.
Risk mitigation funding.
RISK #5
Brief description of Issue and rationale for its rating.
Approach to remedy/mitigation.
Risk mitigation funding.
RISK # 6
Brief description of Issue and rationale for its rating.
Approach to remedy/mitigation.
Risk mitigation funding.

10. “Likelihood” And “Consequence” When Constructing This Chart
Definitions Of
“Likelihood” And “Consequence” When Constructing This Chart
Consequence 
Likelihood  5 4 3 2 1 6 7
Low Risk
Medium Risk
High Risk
Risk Assessment: Address each issue which might
affect the success of the program.
Likelihood
Negligible - One can reasonably assume no occurrence (<10%)
Unlikely - Occurrence possible but less than likely (10-40%)
Likely - significant chance of occurrence (40-65%)
Highly Probable - Very high chances of occurrence (65-90%)
Near Certainty - Assume and anticipate occurrence (>90%)
Consequence
Marginal - Remedy will cause disruption to the program
Significant - Shorts a significant mission need
Serious - Shorts a critical mission need but expect no breech
Very Serious - Potentially fails a KPP or OPEVAL.
Catastrophic - Jeopardizes an exit criterion of current

11. G
Solvency
Budget
Program Manager/Office______________ Date of Review_________
Programmed (%Obligated)
Activity FY99 (%) FY00 (%) FY FY FY FY FY05 FY06
Display each significant task
This addresses whether there enough money budgeted/planned over the course of the life of the program to accomplish the program goals and objectives
Budget: Show the entire budget for the program broken down in the way that you manage it, not the way you budget for it! I am not looking for a replication of the DAES report. Instead I am looking for a breakout of the functional distribution of spending. For instance, a good program manager will have a management reserve built into the budget. Please display it or be prepared to say why no reserve is necessary. Please also break out SETA support and SYSCOM support as separate and explicit categories. At a minimum, each separate contract should be displayed. With large contracts, each significant task above $1M a year should be displayed. For FY 1998 and 1999 please also show the current (as of the date of the report) percentage of obligation in each line.

12. G
Program Solvency
This “Solvency” addresses how well the PM is able to cover the cost of the PROGRAM within funds already allocated to the program.
The color ratings used generally follow these themes:
GREEN-can handle within existing resources without descoping or affecting KPPs
YELLOW-can resolve within existing budget by out-year transfer of funds between appropriations or descoping, and will request changing KPPs
RED-shortfall in the current execution or next fiscal year with no available offsets or workarounds and will request Congressional approval be sought for a current year reprogramming; in short, program requires outside help.

13. Program Acronym ACAT XX Ahead of Schedule and Underspent
October 2000 Guidance – Example Master Chart Contract Performance for [give short contract title] Y
PEO and Program Manager/PMS-XXX
Briefed: FEB 02
Solvency
Nxxxxx-YY-Cxxxx
Contractor Name [Prime or Significant Sub]
[TCPIEAC = 0.76]
CV = $2.0 M
SV = $2.9 M
04/98
01/00
04/00
04/02
08/02
Total Calendar Schedule
PM’s EAC $M
0 %
42%
50%
100%
108%
122%
$110
EAC
1.18
Behind Schedule and Underspent
Ahead of Schedule and Underspent
111%
$100
TAB
1.14
100%
$90
BAC
04/97
1.10
07/97
10/97
04/98
1.06
PM’s Projected
Performance at Completion
for CPI and Duration.
01/98
07/98
10/98
1/99
1.02
01/00
04/99
CPI
02/00
56%
$50
0.98
See following slides for discussion that elaborates on the information provided in this example.
04/00
07/99
ACWP
0.960
03/00
0.94
EV % Spent
10/99
01/00
0.90
Total Spent
0.86
0.940
Behind Schedule and Overspent
Ahead of Schedule and Overspent
0.82 0%
0.82
0.86
0.90
0.94
0.98
1.02
1.06
1.10
1.14
1.18
1.18
KTR’s EAC: 104M
SPI
Date of Last Award Fee: MMM YY Date of Next Award Fee: MMM YY
Date of Last Rebaselining: JAN00 Number of Rebaselinings: 1 Date of Next Rebaselining: MMM YY
YYMMDD

14. Y
Contract Solvency
This “Solvency” addresses how well the PM is able to cover the cost of the CONTRACT within funds already allocated to the program.
The color ratings used generally follow these themes:
GREEN-can handle within existing resources without descoping or affecting KPPs
YELLOW-can resolve within existing budget by out-year transfer of funds between appropriations or descoping, and changing KPPS will be considered.
RED-shortfall in the current execution or next fiscal year with no available offsets or workarounds. the program office has no way to resolve the funding shortfall and needs assistance from the sponsor, DASN, ASN, PEO or others.

15. This Is Not a Metrics Chart, It Is an Illustration of Aggressive Bidding and A Changing Contract
Original Contract - APR 00
[Same status at OCT 00]
PM’s Est Cost $100 G
Solvency
OCT 00
Contractor’s Est Cost $85
Contract Target Cost
PM Allocation for Contract
$68
$112
Contract performance is poor, but yields little budget risk
Contract performance is poor, but yields little budget risk
Contract performance is poor, but yields little budget risk
Rebaselined/Renegotiated Contract - JAN 01
Contract Target Cost
PM’s Est Cost
$115
Contractor’s Est Cost $100
$100
$112
PM Allocation for Contract Y
Solvency
This slide is used if there has been a contract rebaselining.
Provide in this notes page the current contract ceiling, and contract incentives and cost share ratios: (For instance, ceiling is [$XXX M] and contract calls for a 50/50 split for underruns and a 60/40 split for overruns).
Acronyms/terminology are taken generally from definitions for DAES reporting or ASN(RD&A) Memorandum of 5 OCT 2001.
“KTR” means “contractor”.
The “Allocation for Contract Cost” in this slide is the amount of funding dedicated by the Program Manager, within the existing program budget, for the contract price. This includes profit and fees.
Current Status APR 01
PM’S Est Cost
Contractor’s Est Cost $109 Y
Solvency
$130
Contract Target Cost
$112
PM Allocation for Contract
$100

16. CPARS/IPARS/Award Fee Matrix
PEO XXX
Program Acronym ACAT XX
CPARS/IPARS/Award Fee Matrix
CAPT J. Doe, PMS-499
Date of Review: ddmmmyy
At times, an interesting phenomenon occurs: Contract past performance cost ratings closely follow the “Solvency” rather than actual performance.
At times, an interesting phenomenon occurs: Contract past performance cost ratings closely follow the “Solvency” rather than actual performance.
At times, an interesting phenomenon occurs: Contract past performance cost ratings closely follow the “Solvency” rather than actual performance.
CPAR/IPAR/AF Chart Guidance
Prepare one chart for each contract addressed in the “Performance Overview”, or earned value, chart as applicable.
Cover all CPARs and IPARs through the full period of performance for the contract.
Be prepared to address any disconnects between award fee and CPAR/IPAR ratings, e.g. an award fee of 90%, and a number of YELLOWs on CPAR/IPAR.

17. For Reference: Comparison of Color Ratings
(We have attempted to normalize these as far as we can)
Major Weakness
Some event seriously delaying major program objective; breach of program baseline or unit cost.
Potential or Actual Problem
Some event has occurred that impairs progress against major objectives of the program.
“On-Track.” [CPI >0.95]
Program progressing satisfactorily. Contract costs not expected to exceed contract target by > 5%, thus not an issue.
NOT USED
DAES
Not for release to contractors. Brings up issues within Defense.
Applies to Numerous Indicators
* ACAT I & II Program Contracts (EVM)
5 OCT 2000
Applies to Cost and Schedule
* PDUSD(A&T) CPARS/IPARS
24 AUG 1999
In DoN CPARS/IPARS Guidance
Color
Rating
CPI and SPI < 0.90
Unsatisfactory
Does not meet most requirements and timely recovery unlikely.
Red
CPI and SPI > 0.90
Marginal
Does not meet some requirements; serious problem with no corrective action identified.
Yellow
CPI and SPI > 1.0
Satisfactory
Meets requirements; some minor problems, which were corrected
Green
CPI and SPI > 1.1
Very Good - Exceeds some requirements; some minor problems, which were corrected.
Purple
CPI and SPI > 1.15 and exceed many requirements, resolve all problems.
Exceptional
Exceed many requirements; resolve all problems.
Blue

18. Integrated Baseline Reviews
Risk Assessments In
Integrated Baseline Reviews
For Contracts
Managed At The Program Manager Level
No Consistent Form And Format For The IBRs.
Defense/Industry Developing Joint IBR Guide
Joint Strike Fighter IBR Offers A Good Example to Emulate

19. Extracts of Templates Used In The Integrated Baseline Review
For The
Joint Strike Fighter

20. (Sample) WBS 1346 Fuel Sys Interview Summary
Assessment Area
Score
Technical
Schedule
Cost
Resources
Management Processes
Overall Assessment
The Team believes that this cost account can be successfully accomplished within cost and schedule at acceptable risk
Dialogue Highlights
Strengths
good discussion; CAM well-prepared
comprehensive plan
linkages well defined
Weakness
Resource availability
Concerns
Staffing availability
Hardware fabrication cycle with new vendors (different than Phase ll)
Communication between IPT’s - late
data / data updates could cause redos
Risks
Resource availability/timing
Design maturity / hardware schedule

21. IBR Discussion Evaluation Summary
(Level 3 IPT)
TECH
SCHED
COST
RESOURCE
MGT
PROCESS
Comments [Including Risk Costs]
(WBS)
(Comments)

22. Our Guidance in Defense for Risk
Systems Engineering risk management calls for risk:
Avoidance
Control
Assumption22
Transfer
None of these imply an opportunity.
All of these focus on possible negative future occurrences.
The basic objective of risk management in systems engineering is to reduce all risk to an acceptable level.

23. Industry Takes Risk a Step Farther
Industry sees the potential
Compensaton of Industry managers is often tied to the risk of success, which is tracked in scorecards.
In DASN(PP&R) we are reviewing industry use of scorecards and seeing if we can reasonably attach our metrics to a scorecard appropriate for the government and reaching for the risk of success.
success. of
risk