1. 0 0 Exploring a Range of Decision Making Techniques Nov 18,2009 John Bates ( bates_john@bah.com ) bates_john@bah.com

2. 1 1 The Range of Decision Making Techniques Considerations when selecting a technique… How important is the Decision; who will review decision? How much data/information do you have or can you get? How much time and $ do you have? Discussion Group (BOGSAT) Poorly Done Decision Analysis Decision Analysis Incorporating Mod & Sim Decision Analysis Incorporating SMEs Coin Toss or Guess Least Desirable - Most Desirable Advantages/Disadvantages to each technique. Fielded system; record data throughout lifecycle. Operational Exercise and Testing

3. 2 2 Symptoms of poorly done decision analysis Lack of creative and significant Alternatives Solving the Wrong Problem/Objective Not involving the real decision makers Competing Objectives Avoiding Uncertainty Success measured on weight and thickness of final brief Not sure about the definition of the attributes Provided the solution at the beginning

4. 3 3 How important is your decision?

5. 4 4 Recommend a structured process that includes all stakeholders and is traceable/defendable Scoring scales can be nominal, ordinal, interval, or ratios Utility curves transform scoring scales into a common measurable output (Utility) Weighting of the tree structure determines relative importance There are various methods to determine importance AHP Relative Weighting Overall Goal: Determine best solution to fill need Sub-Goals: Maximize Technical Performance, Measures of Effectiveness Maximize Operational Suitability Minimize Risks, etc. Criteria must be independent, discriminatory, meaningful, and measurable Assess each alternative relative to the criteria in the decision tree Utility curves then transform the scoring scales into measurable data ranging from 0 to 1 with 1 being the most preferred Rank ordering of alternatives can be further decomposed to determine major contributors to the overall decision Conduct estimates for each alternative Can use cost objectives and thresholds Cost estimates and be treated separate or added into the evaluation model Sensitivity analysis can be conducted to analyze the results due to: Changes in weights Changes in scores Utility scores can be can be graphed versus cost to determine the "Best Value" alternative Determine Requirements Identify/Clarify Goal Define Criteria 1 Determine Priorities Develop Utility Curves Develop Scoring Scales 2 Conduct Sensitivity Analysis Determine Best Value 5 Assess Alternatives Rank Order Alternatives 4 Generate Alternatives Develop Estimates Incorporate Estimates into Model 3 Understand Problem/VisionModelingDecide and Communicate

6. 5 5 Prioritization (Weighting) Techniques Analytic Hierarchy Process Weightings To Prioritize User Needs Calculation: 1/ 2.667= 0.375 Sum RowsCalculate Row Average (Called the Row Average of the Normalized Columns (RANC) method) Assign Scores Using a Scale Sum Column Developed by Thomas L. Saaty in the 1970s. The AHP provides a comprehensive and rational framework for structuring a decision problem, for representing and quantifying its elements, for relating those elements to overall goals, and for evaluating alternative solutions.Thomas L. Saaty Determine Priorities Develop Utility Curves Develop Scoring Scales 2 Importance 1Equal 3Moderate 5Strong 7Very Strong 9Extreme

7. 6 6 Capturing SME input Participants will be asked to evaluate criteria through a series of pairwise comparisons, which calculates each factors relative importance. To capture these comparisons, each participant uses a remote control to assign a number (1- 9) that reflects the relative weight or importance of each decision factor. Determine Priorities Develop Utility Curves Develop Scoring Scales 2 # Inputs = (N * N-1)/2

8. 7 7 Methodology for collecting group input Pairwise Comparisons ensure all combinations are considered independently The Analytic Hierarchy Process is used to determine the criteria priority. Pairwise comparisons are made for each unique combination. During the session, the facilitator will focus on areas of disagreement by identifying divergent points of view and encouraging participants to elaborate. Participants are encouraged to state their opinions and change their minds – and votes – if they so desire. During facilitated discussion, participants may continue to disagree on the relative importance of different decision factors but they will have a better understanding of different views within the group. Determine Priorities Develop Utility Curves Develop Scoring Scales 2

9. 8 8 Decision Analysis at multiple levels? Mission Areas UIC (ASSIGNED TO RESPORG) Capability Area 1 Capability Area 2 Capability Area 3 PESTO Resource Elements Capability 2 Capability 3 Capability 1 O T S E P NTA2NTA1 Joint Capability Areas (Future) NUMBER OF INPUTS REQUIRED ~10 ~50 ~250 ~3000 Determine Priorities Develop Utility Curves Develop Scoring Scales 2

10. 9 9 Relative Scoring Example SolutionMission AFormula% Importance Car1=1/425% Tank Plane3=3/475% Boat TOTAL4100% CriteriaScale Supports Mission Success1 Important to Mission Success3 Critical to Mission Success9 Determine Priorities Develop Utility Curves Develop Scoring Scales 2

11. 10 Expert Choice – Great discussion/consensus using clickers, takes a lot of time Group Systems – Great at capturing multiple user input simultaneously, can be performed remotely (surveys), requires extensive setup. Excel AHP – Used to Prioritize Resource Categories within a level - Simple to setup, easy to facilitate potential for a lot of intersections ((n * n-1)/2), lose input variation and correlation checking Excel (0,1,3,9) Scale – Simple, quick, needs to be strongly facilitated, lose input variation and consistency check The choice of techniques is primarily based on number of intersections and time needed to perform the analysis TIME TO COLLECT Summary of Different Prioritization Techniques

12. 11 Scaling Examples for Alternative Scoring Utility Signature Improvement in undetectable range (meters) 1 0 01000 Utility Range (meters to Refuel) 1 0 21426000 The range for the Measure of Merit for Signature is based on the improvement in detection range the vehicle realizes through IR signature reduction. The range of the Measure of Merit for Range is based on the ORD Requirement on meters to recognition. Determine Priorities Develop Utility Curves Develop Scoring Scales 2

13. 12 Alternative Comparison Technique DIV A DIV B DIV C Capability/MOE Definition: The ability to transport passengers 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 …. T O T O T O Number of Passengers Determine Priorities Develop Utility Curves Develop Scoring Scales 2

14. 13 Functional Analysis Systems Engineering-based Functional Decomposition provides structured process to define Required Mission Capabilities to assess Alternatives

15. 14 Modeling and Simulation Techniques Discrete Event Simulation Behavior/Functional Modeling Physical or Prototype Modeling Wargaming/Simulations Sensitivity Analysis Linear Programming Reliability,Failure Modes and Effects Analysis Mathematical/Monte Carlo or Probability Modeling

16. 15 Method to Communicate Analysis Results of Analysis 1. System B 2. System A 3. System C

17. 16 Method to Communicate Analysis TRL LEVELS 9-10 7-8 <6

18. 17 Another Method of Communicating Decisions This quad chart represents the capability areas that should be pursued first. Quad 1 High Priority Poor Capabilities Quad 3 Low Priority Poor Capabilities Quad 4 Low Priority Good Capabilities Quad 2 High Priority Good Capabilities

19. 18 The Range of Decision Making Techniques So what are the key considerations when selecting a technique… How important is the Decision; who will review decision? How much data/information do you have or can you get? How much time and $ do you have?

20. 19 Questions ?