Measuring the Quality of Decisionmaking and Planning Framed in the Context of IBC Experimentation February 9, 2007 Evidence Based Research, Inc.

2 Today’s Ground Rules IBC provides the framework for the discussion on measuring decisionmaking. This is not an IBC brief. The briefer does not have the answers! The purpose is to explore how one measures the quality of decisionmaking in complex environments – this is a discussion session.

3 Background Regarding IBC Focus: Visualization, operational assessment, and planning tools for EBO. Multiphased effort: Phase 0: Proof of concept using ad hoc suite of tools. Phase I: Demonstrated improvement potential of federated tool sets developed by two contractors. Phase II: Evolve tool set to expanded capabilities for transition to JFCOM & COCOMS.

4 Objectives of Phase II Experimentation Establish the operational utility of the tools. Learn which tools can be helpful under what circumstances and for what purposes. Explore novel approaches throughout the spiral development process. Measure progress in increasing readiness of the technology. Facilitate early transition to operational users.

5 Challenges Complex operational problem: –Wide range of situations, –Multiple knowledge domains, and –Poorly understood relationships. Major technical challenges: –Variation in maturity of underlying models and tools, –Limitations in availability of relevant data, and –Composability of federated capability from continually expanding set of tools. Transition hurdles: –Operational risk due to high potential for limitations in fidelity and possible misapplication.

6 Option Exploration Tool (1) Comprehensive family of models. Action-to-effect and effect-to-action. Generates the distribution of plausible outcomes. Visualization: assist leaders to understand and act.

7 DIME Dimensions PMESII Dimensions Conceptual Models Modeling Paradigm Object Models Causal Models Statistical Models Proposed Action Desired Effect Agent Based Simulation Concept Map Event-based Simulation Differential Equations Bayesian Network Social Network Petri net Influence Diagram Spread Sheet Political/ Religious Model Social/ Culture Model Social Information Model Political/Economic Model Economic/Infrastructure Model Enables “What-if” Analysis, Answers “Why” Undesired Effect Desired Effect Military Model Religious Model Distribution authorized only to US Government Agencies and their contractors involved in Phase 2 of the Integrated Battle Command Program Option Exploration Tool (2)

8 Campaign Planning Tools (1) Supports modular development of multiple lines of effort. Automatically detects and displays interdependencies, assumptions, resources, actions, duration of effects, metrics, and next state. Ability to modify plans, actions, interdependencies, models, and next states based on measured performance.

9 Political/ Strategy Security Strategy Political-Economic Strategy Next State Actions Assumptions Resources Measured Performance Interdependencies Allows the leader to visualize complex plans Distribution authorized only to US Government Agencies and their contractors involved in Phase 2 of the Integrated Battle Command Program Campaign Planning Tools (2)

10 Network Centric Operations

11 Decisionmaking Quality Two approaches: –Quality of the process: How the decision was made, and Elements in the process. –Quality of the decision: Was it “right” or at least “not wrong”? Did it have the desired effect?

12 Quality of the Process Correct sequence of actions. Complete set of choices. Assumes an observable process. Repeatability. Outcome is irrelevant.

13 Quality of the Decision Applicability to the solution space. Compatibility with other SMEs. Desired outcome/effects. Outcome is relevant.

14 Measuring Quality The field needs a way to measure the quality of a product: –What does it mean to do good? –What does it mean to make a good decision? A simple problem: –Has a correct answer. A complex problem: –Has no right answer, –May have “not wrong” answers, and –Will have multiple perspectives.

15 Discussion Time Thoughts?