INTRODUCTION TO EXERCISE DESIGN & EVALUATION

INTRODUCTIONS  Name  Agency  Exercise Experience  Course Goals

OBJECTIVES  Familiarize participants with and/or review the exercise design and evaluation process  Introduce MTDES evaluation documentation  Examine and analyze HSEEP Toolkit documents and systems

REASONS TO EXERCISE  Establish a comprehensive exercise program  Reveal planning weaknesses  Reveal gaps in resources  Satisfy regulatory requirements  Clarify roles and responsibilities  Test and evaluate plans, policies, and procedures

BENEFITS OF EXERCISING  Individual Training  System Improvement

REASONS TO USE HSEEP  Common exercise policy and program guidance  Consistent terminology  Provides tools for exercise design, conduct, and evaluation

PREPAREDNESS CYCLE

CAPABILITIES-BASED PLANNING  Reviewing current plan  Conducting needs assessment  Assessing jurisdiction’s capability to conduct an exercise

EXERCISE PROGRAM MANAGEMENT  Project Management  Multi-Year Planning  Budgeting  Grant Management  Staff Hiring  Funding Allocation  Expenditure Tracking

MULTI-YEAR TRAINING & EXERCISE PLAN  Goals  Components  Capabilities-Based

PROGRESSIVE EXERCISES  Component of a comprehensive exercise program  Goal:  Large-scale functional or full-scale exercise  Small-scale discussion-based and operations-based exercises in preparation for large-scale exercise

EXERCISE TYPES

DISCUSSION-BASED EXERCISES

SEMINAR  Overview/Introduction  Purpose  Leadership  Participants  Applications

WORKSHOP  Purpose  Characteristics  Participants  Applications

TABLETOP  Purpose  Characteristics  Participants  Applications

GAME  Purpose  Characteristics  Applications

OPERATIONS-BASED EXERCISES

DRILL  Purpose  Characteristics  Applications

FUNCTIONAL EXERCISE  Purpose  Characteristics  Applications

FULL-SCALE  Purpose  Characteristics  Participants  Applications

EXERCISE PROJECT MANAGEMENT  Foundation  Design and Development  Conduct  Evaluation  Improvement Planning

FOUNDATION

 Base of Support  Exercise Planning Timelines  Exercise Planning Team  Exercise Planning Conferences

BASE OF SUPPORT  Senior Officials  Buy-In  Purpose

EXERCISE PLANNING TIMELINES  Target timeframes  Multi-Year Training and Exercise Plan  Origin & Composition

EXERCISE PLANNING TEAM  Purpose  Responsibilities  Composition  ICS Exercise Planning Team Leader Operations Safety (Ops only) Planning Admin Finance Logistics Site Liaison Resources Exercise Documentation Evaluation Props Actors Reporting Budgeting

EXERCISE PLANNING CONFERENCES  Concept and Objectives Meeting  Initial Planning Conference  Mid-Term Planning Conference  Master Scenario Events List Conference  Final Planning Conference

DESIGN & DEVELOPMENT

EXERCISE DESIGN STEPS  Assess Needs  Define Scope  Write a Statement of Purpose  Define Objectives  Compose a Narrative  Write Major and Detailed Events  List Expected Actions  Prepare Injects Assess Needs Define Scope Statement of Purpose Define Objectives Narrative Major & Detailed Events Expected Actions Injects

ASSESS NEEDS  Purpose  Needs Assessment Process  Evaluations from Past Exercises  Needs Assessment Product

TARGET CAPABILITIES  Capabilities-based planning  Capabilities-based exercises  Tasks & Objectives

DEFINE THE SCOPE  Purpose  Determining Functions  Key Elements

WRITE A STATEMENT OF PURPOSE  Purpose  Goal  Content

DEFINE OBJECTIVES  Purpose  Origin  Number  “Good” Objectives  SMART Objectives S imple M easurable A chievable R ealistic T ask Oriented SMART System

SCENARIO  Purpose  Characteristics  Narrative Outline

WRITE MAJOR AND DETAILED EVENTS Major EventsSample  Purpose  Goal  Strategy  Origin  Sample 1. Fuselage breaks apart and hits buildings below 2. Jet fuel ignites several homes in the area 3. About 60 survivors are thought to be trapped in the front section of the aircraft 4. Estimates of passenger casualties rise between 200 & 220 deaths and at least 70 severe burn victims

WRITE MAJOR AND DETAILED EVENTS (CONT.) Detailed EventsSample  Purpose  Goal  Strategy  Origin  Sample 1. The mortuary is unable to accept the large numbers of remains resulting from the crash. 2. Local hospitals lack specialized facilities and personnel to treat large numbers of severe burn victims. 3. The American Red Cross has agreed to set up a family information center to link victims and their families

LIST EXPECTED ACTIONS  Purpose  Goals  Types of Actions  Guidelines  Relationship to Objectives  Example

DOCUMENTATION  Situation Manual  Exercise Plan  Controller & Evaluator Handbook  Master Scenario Events List  Player Handout  Exercise Evaluation Guides  Exercise Policies

LOGISTICS  Discussion-based Exercises  Operations-based Exercises

EXERCISE CONDUCT

 Setup  Briefings  Management of Facilitators, Controllers, Evaluators, Players, and Actors  Wrap-Up Activities

SETUP  Site Visits  Pre-Exercise Check  Discussion-Based Exercises  Operations-Based Exercises

PRESENTATIONS/BRIEFINGS  Discussion-Based Exercises  Operations-Based Exercises

PERSONNEL  Facilitators  Controllers  Evaluators  Players  Actors  Simulators

EXERCISE EVALUATION

EVALUATION GOALS  Identify successes and shortfalls of:  Response organizations  Planning documents  Facilities  Emergency Management System  Training/Staffing deficiencies  Equipment  Improve Emergency Management Capabilities

EVALUATION STEPS  Plan and Organize the Evaluation  Observe the Exercise and Collect Data  Analyze Data  Develop the Draft After Action Report & Improvement Plan

PLAN & ORGANIZE THE EVALUATION

EVALUATION TEAM LEADER  Qualifications  Responsibilities  Designation

EVALUATION REQUIREMENTS & METHODOLOGY  Purpose  Defining the Team Structure  Criteria  Strategies  Evaluation Plan

EVALUATION TEAM  Team Structure  Responsibilities  Strategies

RECRUIT, ASSIGN, AND TRAIN EVALUATORS  Recruiting  Experience & Expertise  Time Commitment  Establishing Base of Evaluators  Assigning  Strategy  When to Assign  Training  Purpose  Materials  Instructions

FINALIZE EVALUATION PLAN  Exercise-Specific Information  Evaluator Team Organization, Assignments, and Location  Evaluator Instructions  Evaluation Tools

CONTROLLER AND EVALUATOR BRIEFING  Purpose  Composition

OBSERVE THE EXERCISE AND COLLECT DATA

EXERCISE EVALUATION GUIDES  Purpose  Use  Composition

DISCUSSION-BASED EXERCISES  Higher Level Capability Issues  Types of Information Collected  Debrief

OPERATIONS BASED EXERCISES  Task & Activity Level Observations  Types of Information Collected  Use Events to Gauge Success  Evaluator Guidelines

HOT WASH  Purpose  Roles and Responsibilities  Guidelines

COLLECTING SUPPLEMENTAL DATA  Delegation  Purpose of Data  Examples

ANALYZE DATA

IDENTIFYING ROOT CAUSE AND DEVELOPING RECOMMENDATIONS  What Happened  Why It Happened  Root Cause Analysis

DATA ANALYSIS Discussion-Based ExercisesOperations-Based Exercises  Controller & Evaluator Debrief  Draft AAR Content  Preliminary Analysis Content  Controller & Evaluator Debrief  Draft AAR Content  Exercise Event Timeline  Preliminary Analysis Content

DEVELOP THE DRAFT AFTER ACTION REPORT & IMPROVEMENT PLAN

AFTER ACTION REPORT  Purpose  Who is responsible for it?  Form  Format

IMPROVEMENT PLANNING

CONDUCT AFTER ACTION CONFERENCE  Purpose  Content  Format

IDENTIFY CORRECTIVE ACTION TO BE IMPLEMENTED  Corrective Actions  Purpose  Content  Realistic Goals

IMPROVEMENT PLAN  Purpose  Components  Capability  Observation  Recommendations  Corrective Action Description  Responsible Party/Agency  Agency POC  Start Date  End Date

CAPABILITY ELEMENTS  Planning  Organization and Leadership  Personnel  Equipment and Systems  Training  Exercises, Evaluations, and Corrective Actions

FINALIZE AAR/IP  Purpose  Composition  Goals

TRACK IMPLEMENTATION  Event Points of Contact  Participating Entity Points of Contact  Action Officers  Continual Improvement

QUESTIONS?