Facilitator/Evaluator Orientation/Briefing MARC Mass Fatality TTX May 21,

2 Roles and Responsibilities Evaluation Methodology Overview –Exercise Evaluation Guide (EEG) Concept Evaluation Process –Evaluator Responsibilities Assessment, Review, and Analysis of the Exercise Briefing Topics

Roles and Responsibilities Facilitators provide situation updates and moderate discussions. They also provide additional information or resolve questions as required. Evaluators use EEGs to measure and assess performance, capture unresolved issues, and analyze exercise results. Evaluators passively assess and document players’ performance against established emergency plans and exercise evaluation criteria, in accordance with HSEEP standards. Evaluators have a passive role in the exercise and only note the actions/decisions of players without interfering with exercise flow. For this exercise, evaluators may provide a limited coaching role of stimulating thought process. 3

FacilitatorsEvaluators Chris Kates – UC John Linstrom- MF Justin Sorg – EM/Logs Joan Harding– Health and Medical Dwight Graves PIO Dana Hall VIC 4 UC – Joe Lynch MF – Chuck Smith EM Logs – Ricky Carroll Health & Medical – Erin Lynch PIO Dwight Graves VIC – Lisa Ladue

5 Validate strengths, and identify improvement opportunities Validate plans, procedures, and protocols Improve processes and plans Determine level of capability in regard to the exercise’s core capabilities Evaluation Goals

The Homeland Security Exercise and Evaluation Program (HSEEP) Volume III provides evaluation and improvement planning policy. Evaluations should be objective-based to remain in line with: –National Preparedness Goal –Core Capabilities List –Jurisdiction and agency plans HSEEP EEG templates provide streamlined process for standardized evaluation concept, not necessarily content. EEG Concept 6

Allows evaluators to review validation of plan elements –Plans used will depend on participating agencies Includes references to plan sections, tables, and charts –Allows evaluators to better prepare for exercise –Provides detail for what will be expected of players Remains focused on exercise objectives as they relate to the selected core capability EEG Function 7

Evaluation team should use EEGs in conjunction with full plans to prepare Use EEGs as guides for observation, but do not focus on completing them during the exercise  Use EEGs for data analysis after exercise and in preparation for the evaluator debriefing  Use measured results to make actionable improvement recommendations within the After Action Report (AAR)/ Improvement Plan (IP) Use of EEGs 8

HSEEP Methodology Steps Evaluation Tools Timeline Exercise objectives, actual plans and procedures Evaluator logEEG analysis sheet AAR data input form Through Final Planning Conference (FPC) planning During exercise Controller/ Evaluator (C/E) Debriefing Post exercise, before AAR draft development Use of Evaluation Tools 9

Evaluator Notes and Data Evaluators should capture player discussions, decisions, actions, and activity during exercise play. Use back of EEGs for notes The EEGs provided should serve as guides for documentation of exercise activity 10

EEG Analysis Form Each evaluator will be provided with EEGs that will give specific guidance on what data to collect during the exercise, how to record it, and how to analyze it These forms should be completed at the conclusion of the exercise, using data gathered during the exercise 11

EEG AAR Data Input Form Each EEG will include a second form to be completed following the C/E Debriefing This form will serve as an expansion to the notes captured in the EEG analysis form, including an area to provide recommendations. Evaluators should complete one AAR input form for each identified strength or area for improvement 12

After Action Report (AAR)/Improvement Plan (IP ) Data from the EEGs and debriefing will be compiled into the AAR/IP. IP 13

Be on time Wear appropriate attire Make sure you have the tools necessary to perform task Understand the exercise concept and scenario Read the sit man Facilitator Evaluator Responsibilities 14

Be familiar with the assigned EEGs Collect information Make observations on player discussions, knowledge, and intended impact of decisions Analyze actions and develop post exercise AAR/IP input Evaluator Responsibilities 15

Review appropriate plans, procedures, and protocols Participate in C/E Briefing Review appropriate exercise materials to be familiar with scenario, objectives, and core capabilities Review EEGs and other supporting materials for your area of responsibility Pre-exercise Responsibilities 16

Monitor and document player performance Use EEGs to confirm objectives are being met Focus on critical tasks, as specified in the EEGs and plans being exercised Do not prompt players or interfere with exercise play Limited coaching is allowed Exercise Responsibilities 17

At the start of the exercise, evaluators should: Position themselves to appropriately hear discussions Be prepared to move between break out groups if necessary to follow various discussions Evaluators should not: Focus solely on individuals Interfere with exercise discussion or action Evaluator Guidelines 18

Evaluators must keep an accurate written record of important issues that take place during the exercise:  Were participants knowledgeable of plans and procedures to meet exercise objectives?  Did discussions validate that operations would occur in accordance with plans and procedures?  Did participants validate that sufficient personnel are equipped, trained, and assigned to perform tasks? Observing the Exercise 19

Evaluators should record events such as: –Discussed actions of players –Key decisions –Discussions that denote deviations from plans and procedures –Player discussion of key movements and activities –Module responses on easel pad –Hotwash responses Recording Important Events 20

Postexercise Analysis Responsibilities Participate in the Hot Wash, and take notes on findings identified by players Summarize your notes and prepare for the C/E Debriefing Have your summary ready for the lead evaluator, including bulleted strengths and areas for improvement from activity analysis Use data to determine whether the exercise capabilities and plans were effectively implemented –Identify observations as strengths or areas for improvement 21

Allows players to participate in a self- assessment of the exercise play Allows evaluators to clarify additional points and collect missing information Identify three sustains and three improves Make sure sustains and improves are listed on the easel pad Player Hot Wash 22

Work to complete EEG analysis forms for submission Discuss degree to which each objective was completed Provide strengths, best practices, lessons learned, and possible IP recommendations Begin outlining issues to be included in AAR/IP on AAR data input forms Controller and Evaluator (C/E) Debriefing 23

AAR/IP Input Use the EEG AAR data input form Complete one form for each strength and area for improvement. –Observation. Write a one-sentence statement that identifies the strength or area for improvement as noted in the activity analysis tables. –References. Reference relevant plans, policies, procedures, laws, or regulations that could be used to facilitate implementing recommendations for improvement or capturing strengths. –Analysis. Document detailed data on what you observed during the exercise specific to this individual observation. –Recommendations. Suggest steps to allow the participants to further improve the plans or share strengths. All EEG input is due to Joe or Lou no later than departure from the exercise site on the exercise day. 24

Evaluation Matrix Objective MetObjective Not Met Players followed plan procedures Plan works Document best practices and strengths Capture why plan didn’t work Document how to improve relevant plan elements Players did not follow plan procedures Capture how objective was met Recommend successful actions and procedures be included in plan rewrite Capture why plan was not used May require plan training and familiarization 25

Evaluator Deliverables Before you leave –Completed EEGs –Notes –Pictures of easel pads in your section (Joe has camera and will do this but easel sheets are important to retain) 26

Questions/Discussion 27