IAEA Training in Emergency Preparedness and Response Exercise Evaluation Lecture Module L-056

L-056: Exercise Evaluation2 Objectives Understand the importance of evaluation for the success of the exercise Understand performance-based evaluation Know the role of evaluators and their procedures Be able to develop evaluation criteria Know how to grade and report evaluations

L-056: Exercise Evaluation3 Principles of Evaluation Performance required by plans and procedures is being tested Evaluations are results-oriented to enhance the plan and the supporting elements, such as proper organization, training, and equipment

L-056: Exercise Evaluation4 Aim of Evaluations The aim of an evaluation is to identify strengths and weakness in the emergency response capability

L-056: Exercise Evaluation5 What is a Performance-based Evaluation? Focus on results, not on process Based on response objectives It answers the question: was the result achieved?

L-056: Exercise Evaluation6 How to Evaluate Based on Performance (1) Define the response or exercise objective e.g. Protection of facility personnel Define the criteria e.g. The personnel are promptly alerted e.g. The dose to personnel is kept within established limits

L-056: Exercise Evaluation7 How to Evaluate Based on Performance (2) For each performance objective, define the evaluation criteria e.g. The personnel are promptly alerted Was a public address message broadcast immediately upon discovery of an emergency situation? Did all personnel hear the message? e.g. The dose to personnel is kept within established limits Were personnel relocated to safe areas? Was the dose rate monitored at personnel assembly areas?

L-056: Exercise Evaluation8 How to Evaluate Based on Performance (3) After the exercise, based on the FACTS, answer the question: was the performance objective met? It is not absolutely necessary that all criteria be satisfied The actual response will depend on the situation

L-056: Exercise Evaluation9 How to Evaluate Based on Performance (4) If the performance objective was met Was the procedure followed? Was the result achieved in good faith or by emergency? If the performance objective was not met, why? Was the procedure followed? Was training lacking? Was equipment lacking? Was the exercise at fault?

L-056: Exercise Evaluation10 Evaluation Team Organization Chief evaluator Evaluators for all key groups or locations

L-056: Exercise Evaluation11 Evaluator Role and Responsibilities Chief evaluator Coordinate the evaluation team Instruct evaluators on the schedule, method and criteria Ensure consistency between evaluators Instruct the evaluators on the format and schedule of the evaluation reports Produce the final evaluation report

L-056: Exercise Evaluation12 Evaluator Role and Responsibilities Evaluators Observe and record Do not evaluate during the exercise Stick to facts Evaluate after the exercise Provide a report to the chief evaluator

L-056: Exercise Evaluation13 Choice of Evaluators Evaluators must be qualified for the function they evaluate Evaluators must be trained Evaluation techniques Procedures of the groups being evaluated Evaluators do not need to know the detailed scenario

L-056: Exercise Evaluation14 Preparation of the Evaluation Team Brief the evaluation team Provide clear written instructions Organize a site orientation tour

L-056: Exercise Evaluation15 Evaluator Instructions Before the exercise: Review the scenario timeline and objectives Review the procedures used by the participants Review the applicable evaluation criteria Understand the actions required to demonstrate a performance objective

L-056: Exercise Evaluation16 Evaluator Instructions (Contd) During the exercise Arrive before an event Locate a telephone Wear Evaluators identification Do not interfere with the participants Never criticize the participants Minimize communications

L-056: Exercise Evaluation17 Evaluator Instructions Following the exercise Attend debriefing and critique Observe the effectiveness of the debriefing and critique Obtain exercise documents Ensure exercise location is in a safe state Finalize your chronology notes

L-056: Exercise Evaluation18 Observation Techniques Record arrival times Record names and functions Record times of major events Record repetitive interventions Listen to commands and instructions Listen and record Controller inputs Record exercise observations

L-056: Exercise Evaluation19 Guide for Evaluators After the exercise, based on the FACTS, answer the question: was the performance objective met It is not absolutely necessary that all criteria be satisfied: The actual response will depend on the situation

L-056: Exercise Evaluation20 Drillmanship Drillmanship is the art of exercise delivery Record facts on: Simulation The exercise scenario Safety Participant feedback

L-056: Exercise Evaluation21 On the Use of Evaluation Checklists... Several organizations use checklists for each group being evaluated This focuses the evaluation on each group in isolation rather than on the overall coordinated response performance Focus on recording important facts rather than checking a list Evaluate the overall performance on the basis of exercise objectives using all the facts collected by the evaluators

L-056: Exercise Evaluation22 Debriefings Hot Wash-up Exercise Team Debriefing Executive Debriefing Brief the management staff Give credit to the players for identifying deficiencies and give them a grace period to fix the problems before forcing compliance or imposing penalties

L-056: Exercise Evaluation23 Safety Responsibilities Evaluators are also responsible for monitoring exercise play to ensure a safe environment

L-056: Exercise Evaluation24 Grading Criteria Excellent Satisfactory Unsatisfactory

L-056: Exercise Evaluation25 Grading Criteria (contd) Grading must be supported by facts (if it was not observed, it didnt happen) Trivial mistakes remain trivial even when they are observed

L-056: Exercise Evaluation26 Classification of Unsatisfactory Grades Critical Deficiency Major Deficiency Minor Deficiency

L-056: Exercise Evaluation27 Deficiency Classification Planning Training Resources

L-056: Exercise Evaluation28 Exercise Report A useful report is most difficult to write Based on facts, not impressions Focus on key areas Use the exercise objectives as the basis for the report Prepare and publish in a timely fashion Not everybody will agree with the findings

L-056: Exercise Evaluation29 Post Exercise Actions Report clarification Ensure there are no inaccuracies in the report Critical deficiencies Take corrective actions as soon as possible, e.g. one month Major deficiencies Take corrective actions within a reasonable time frame, e.g. months Minor deficiencies Take corrective actions during the annual revision of the plan or procedures

L-056: Exercise Evaluation30 Summary The evaluation should be performance- based Evaluators and controllers are not the same persons An evaluator does NOT evaluate DURING an exercise, but after Base the evaluation report on the exercise objectives Look for corrective actions