1. INTRODUCTION TO EXERCISE DESIGN & EVALUATION

2. INTRODUCTIONS  Name  Agency  Exercise Experience  Course Goals

3. OBJECTIVES  Familiarize participants with and/or review the exercise design and evaluation process  Introduce MTDES evaluation documentation  Examine and analyze HSEEP Toolkit documents and systems

4. REASONS TO EXERCISE  Establish a comprehensive exercise program  Reveal planning weaknesses  Reveal gaps in resources  Satisfy regulatory requirements  Clarify roles and responsibilities  Test and evaluate plans, policies, and procedures

5. BENEFITS OF EXERCISING  Individual Training  System Improvement

6. REASONS TO USE HSEEP  Common exercise policy and program guidance  Consistent terminology  Provides tools for exercise design, conduct, and evaluation

7. PREPAREDNESS CYCLE

8. CAPABILITIES-BASED PLANNING  Reviewing current plan  Conducting needs assessment  Assessing jurisdiction’s capability to conduct an exercise

9. EXERCISE PROGRAM MANAGEMENT  Project Management  Multi-Year Planning  Budgeting  Grant Management  Staff Hiring  Funding Allocation  Expenditure Tracking

10. MULTI-YEAR TRAINING & EXERCISE PLAN  Goals  Components  Capabilities-Based

11. PROGRESSIVE EXERCISES  Component of a comprehensive exercise program  Goal:  Large-scale functional or full-scale exercise  Small-scale discussion-based and operations-based exercises in preparation for large-scale exercise

12. EXERCISE TYPES

13. DISCUSSION-BASED EXERCISES

14. SEMINAR  Overview/Introduction  Purpose  Leadership  Participants  Applications

15. WORKSHOP  Purpose  Characteristics  Participants  Applications

16. TABLETOP  Purpose  Characteristics  Participants  Applications

17. GAME  Purpose  Characteristics  Applications

18. OPERATIONS-BASED EXERCISES

19. DRILL  Purpose  Characteristics  Applications

20. FUNCTIONAL EXERCISE  Purpose  Characteristics  Applications

21. FULL-SCALE  Purpose  Characteristics  Participants  Applications

22. EXERCISE PROJECT MANAGEMENT  Foundation  Design and Development  Conduct  Evaluation  Improvement Planning

23. FOUNDATION

24.  Base of Support  Exercise Planning Timelines  Exercise Planning Team  Exercise Planning Conferences

25. BASE OF SUPPORT  Senior Officials  Buy-In  Purpose

26. EXERCISE PLANNING TIMELINES  Target timeframes  Multi-Year Training and Exercise Plan  Origin & Composition

27. EXERCISE PLANNING TEAM  Purpose  Responsibilities  Composition  ICS Exercise Planning Team Leader Operations Safety (Ops only) Planning Admin Finance Logistics Site Liaison Resources Exercise Documentation Evaluation Props Actors Reporting Budgeting

28. EXERCISE PLANNING CONFERENCES  Concept and Objectives Meeting  Initial Planning Conference  Mid-Term Planning Conference  Master Scenario Events List Conference  Final Planning Conference

29. DESIGN & DEVELOPMENT

30. EXERCISE DESIGN STEPS  Assess Needs  Define Scope  Write a Statement of Purpose  Define Objectives  Compose a Narrative  Write Major and Detailed Events  List Expected Actions  Prepare Injects Assess Needs Define Scope Statement of Purpose Define Objectives Narrative Major & Detailed Events Expected Actions Injects

31. ASSESS NEEDS  Purpose  Needs Assessment Process  Evaluations from Past Exercises  Needs Assessment Product

32. TARGET CAPABILITIES  Capabilities-based planning  Capabilities-based exercises  Tasks & Objectives

33. DEFINE THE SCOPE  Purpose  Determining Functions  Key Elements

34. WRITE A STATEMENT OF PURPOSE  Purpose  Goal  Content

35. DEFINE OBJECTIVES  Purpose  Origin  Number  “Good” Objectives  SMART Objectives S imple M easurable A chievable R ealistic T ask Oriented SMART System

36. SCENARIO  Purpose  Characteristics  Narrative Outline

37. WRITE MAJOR AND DETAILED EVENTS Major EventsSample  Purpose  Goal  Strategy  Origin  Sample 1. Fuselage breaks apart and hits buildings below 2. Jet fuel ignites several homes in the area 3. About 60 survivors are thought to be trapped in the front section of the aircraft 4. Estimates of passenger casualties rise between 200 & 220 deaths and at least 70 severe burn victims

38. WRITE MAJOR AND DETAILED EVENTS (CONT.) Detailed EventsSample  Purpose  Goal  Strategy  Origin  Sample 1. The mortuary is unable to accept the large numbers of remains resulting from the crash. 2. Local hospitals lack specialized facilities and personnel to treat large numbers of severe burn victims. 3. The American Red Cross has agreed to set up a family information center to link victims and their families

39. LIST EXPECTED ACTIONS  Purpose  Goals  Types of Actions  Guidelines  Relationship to Objectives  Example

40. DOCUMENTATION  Situation Manual  Exercise Plan  Controller & Evaluator Handbook  Master Scenario Events List  Player Handout  Exercise Evaluation Guides  Exercise Policies

41. LOGISTICS  Discussion-based Exercises  Operations-based Exercises

42. EXERCISE CONDUCT

43.  Setup  Briefings  Management of Facilitators, Controllers, Evaluators, Players, and Actors  Wrap-Up Activities

44. SETUP  Site Visits  Pre-Exercise Check  Discussion-Based Exercises  Operations-Based Exercises

45. PRESENTATIONS/BRIEFINGS  Discussion-Based Exercises  Operations-Based Exercises

46. PERSONNEL  Facilitators  Controllers  Evaluators  Players  Actors  Simulators

47. EXERCISE EVALUATION

48. EVALUATION GOALS  Identify successes and shortfalls of:  Response organizations  Planning documents  Facilities  Emergency Management System  Training/Staffing deficiencies  Equipment  Improve Emergency Management Capabilities

49. EVALUATION STEPS  Plan and Organize the Evaluation  Observe the Exercise and Collect Data  Analyze Data  Develop the Draft After Action Report & Improvement Plan

50. PLAN & ORGANIZE THE EVALUATION

51. EVALUATION TEAM LEADER  Qualifications  Responsibilities  Designation

52. EVALUATION REQUIREMENTS & METHODOLOGY  Purpose  Defining the Team Structure  Criteria  Strategies  Evaluation Plan

53. EVALUATION TEAM  Team Structure  Responsibilities  Strategies

54. RECRUIT, ASSIGN, AND TRAIN EVALUATORS  Recruiting  Experience & Expertise  Time Commitment  Establishing Base of Evaluators  Assigning  Strategy  When to Assign  Training  Purpose  Materials  Instructions

55. FINALIZE EVALUATION PLAN  Exercise-Specific Information  Evaluator Team Organization, Assignments, and Location  Evaluator Instructions  Evaluation Tools

56. CONTROLLER AND EVALUATOR BRIEFING  Purpose  Composition

57. OBSERVE THE EXERCISE AND COLLECT DATA

58. EXERCISE EVALUATION GUIDES  Purpose  Use  Composition

59. DISCUSSION-BASED EXERCISES  Higher Level Capability Issues  Types of Information Collected  Debrief

60. OPERATIONS BASED EXERCISES  Task & Activity Level Observations  Types of Information Collected  Use Events to Gauge Success  Evaluator Guidelines

61. HOT WASH  Purpose  Roles and Responsibilities  Guidelines

62. COLLECTING SUPPLEMENTAL DATA  Delegation  Purpose of Data  Examples

63. ANALYZE DATA

64. IDENTIFYING ROOT CAUSE AND DEVELOPING RECOMMENDATIONS  What Happened  Why It Happened  Root Cause Analysis

65. DATA ANALYSIS Discussion-Based ExercisesOperations-Based Exercises  Controller & Evaluator Debrief  Draft AAR Content  Preliminary Analysis Content  Controller & Evaluator Debrief  Draft AAR Content  Exercise Event Timeline  Preliminary Analysis Content

66. DEVELOP THE DRAFT AFTER ACTION REPORT & IMPROVEMENT PLAN

67. AFTER ACTION REPORT  Purpose  Who is responsible for it?  Form  Format

68. IMPROVEMENT PLANNING

69. CONDUCT AFTER ACTION CONFERENCE  Purpose  Content  Format

70. IDENTIFY CORRECTIVE ACTION TO BE IMPLEMENTED  Corrective Actions  Purpose  Content  Realistic Goals

71. IMPROVEMENT PLAN  Purpose  Components  Capability  Observation  Recommendations  Corrective Action Description  Responsible Party/Agency  Agency POC  Start Date  End Date

72. CAPABILITY ELEMENTS  Planning  Organization and Leadership  Personnel  Equipment and Systems  Training  Exercises, Evaluations, and Corrective Actions

73. FINALIZE AAR/IP  Purpose  Composition  Goals

74. TRACK IMPLEMENTATION  Event Points of Contact  Participating Entity Points of Contact  Action Officers  Continual Improvement

75. QUESTIONS?