1. IAEA International Atomic Energy Agency TOOLS AND METHODS FOR ENHANCING IRRS EFFECTIVENESS AND EFFICIENCY Presentation to the Workshop on Lessons Learned from IRRS Missions Moscow, Russian Federation 9-11 December 2014

2. IAEA Outline On the IRRS missions in general Continuous improvement of the IRRS programme Conclusions IRRS enhancements2

3. IAEA IRRS BASICS IRRS enhancements3

4. IAEA IRRS Process Prep-meeting about 6 months before Self-assessment by host First mission: 12 days Report finalization about 3 months Follow-up missions after 2-3 years, about 7 days IRRS enhancements4

5. IAEA IRRS Guidelines IRRS enhancements5 Published in May 2013 A series of Targeted Consultancies was held for collecting experience and advice for a thorough revision Novel edition will be compiled based on the consultancies

6. IAEA IAEA Safety Standards used in IRRS IRRS enhancements6

7. IAEA Basis for findings: GSR Part 1 Requirements IRRS enhancements7 IRRS Modules + other IAEA Safety Standards

8. IAEA Missions in 2006-2014 IRRS enhancements8

9. IAEA Participation in the missions (2006-2014) * No. of experts (including IAEA): 373 No. of participations: 707 From 59 countries IRRS enhancements9 * by October 2014

10. IAEA CONTINUOUS IMPROVEMENT OF THE IRRS PROGRAMME IRRS enhancements10

11. IAEA IAEA Nuclear Safety Action Plan IAEA is to “Strengthen IAEA peer reviews in order to maximize the benefits to Member States”. In specific: 2.4.1 IAEA Secretariat to review the effectiveness of their peer reviews: IRRS, and therein 2.4.1.2 Define effectiveness criteria, apply the criteria to a representative number of selected IRRS missions and prepare a report IRRS enhancements11

12. IAEA Ways to improve IRRS Training IRRS Mission Standards Feedbacks Measuring effectiveness and efficiency Prompt Evaluation Report IRRS enhancements12

13. IAEA First Basic IRRS Training Course First held in October 2013 62 participants from 38 MSs + EC In the framework of the EC-IAEA cooperation in the field of Nuclear Safety Covered all aspects of the IRRS programme, 16 Training Lectures, 3 Information Lectures Test Questionnaires for all Lectures IRRS enhancements13

14. IAEA Feedback on the course IRRS enhancements14

15. IAEA Feedback from the Test Questionnaires IRRS enhancements15

16. IAEA 2 nd Edition Held in October 2014 63 participants from 39 MSs + EC Even more favourable feedback IRRS enhancements16

17. IAEA On-site training By invitation from MSs Minimum 10 competent participants First held at ONR (Bootle, UK) 15 participants Excellent acceptance IRRS enhancements17

18. IAEA Refresher Training On the 0 th day of the missions Covers details important for the given missions: Overview of the IRRS Process Methodology of the review Evaluation and documentation of Observations Report writing IRRS Modules to review IRRS Mission Schedule Held in 2014 in the Netherlands and France IRRS enhancements18

19. IAEA Feedback IRRS enhancements19

20. IAEA Ways to improve IRRS Training IRRS Mission Standards Feedbacks Measuring effectiveness and efficiency Prompt Evaluation Report IRRS enhancements20

21. IAEA Mission Standards – ARM Report Template Under development / discussion IRRS enhancements21

22. IAEA Mission Standards – Team Assignment IRRS enhancements22

23. IAEA Mission Standards – Report Template Cross-contribution table Instructions for reviewers Issues to cover Format of observations IRRS enhancements23

24. IAEA Mission Standards – Schedule IRRS enhancements24

25. IAEA Ways to improve IRRS Training IRRS Mission Standards Feedbacks Measuring effectiveness and efficiency Prompt Evaluation Report IRRS enhancements25

26. IAEA Feedbacks on the IRRS process Questionnaire on the quality of the ARM – reviewers at mission start Questionnaire on effectiveness of the mission – reviewers at end of mission Round-table conclusions and suggestions – reviewers at end of mission Questionnaire on effectiveness of the mission – host after mission Questionnaire on effectiveness of the initial mission – host before follow-up mission Workshop for team leaders (January 2013) Workshop for experienced reviewers (April 2015) IRRS enhancements26

27. IAEA Ways to improve IRRS Training IRRS Mission Standards Feedbacks Measuring effectiveness and efficiency Prompt Evaluation Report IRRS enhancements27

28. IAEA Effectiveness and efficiency of IRRS NSAP requires establishing criteria for efficiency of IRRS missions 16 Performance Indicators introduced to characterize the E&E of a mission, such as: Team size; team experience; Time available to prepare for the mission; Feedback from team members and from hosts; Coverage of the Action Plan by findings; Balance of findings; Conciseness of mission report; etc. IRRS enhancements28 http://gnssn.iaea.org/regnet/ irrs/Pages/IRRS_pub_docs. aspx

29. IAEA Examples of Performance Indicators (1) IRRS enhancements29

30. IAEA Examples of Performance Indicators (2) IRRS enhancements30 Balance of findings: R*GP, R/S  const. Expectations: a)R and GP are negatively correlated b)R and S are positively correlated 1)The expected correlations are not present in all missions 2)The reasons in case of R and GP are: reviewers feel obliged to compensate by GP some topics (with possible issues) are more frequently considered than others (with GP) 3)The reasons in case of R and S are: R -> S: compromise between the team & host S -> R: the team tends to be stricter or the specificities of the given mission

31. IAEA E & E of past IRRS missions Note: The Performance Indicator values presented here characterize the effectiveness and efficiency of the IRRS process and represent no indication whatsoever on the safety performance of the host countries of the missions IRRS enhancements31 The distance from optimum tends to decrease, i.e. the E&E tends to increase

32. IAEA Ways to improve IRRS Training IRRS Mission Standards Feedbacks Measuring effectiveness and efficiency Prompt Evaluation Report IRRS enhancements32

33. IAEA Prompt evaluation report CONTENTS Basic mission data Feedback from the team Feedback from the hosts Overall evaluation of effectiveness and efficiency Discussion, conclusions Summary IRRS enhancements33

34. IAEA Evaluation of a mission via PI’s IRRS enhancements34

35. IAEA Reports on nuclear missions IRRS enhancements35

36. IAEA Reports on nuclear missions IRRS enhancements36

37. IAEA Reports on nuclear missions IRRS enhancements37

38. IAEA IN SUMMARY IRRS enhancements38

39. IAEA Conclusions IRRS is an effective means of reviewing compliance of RB status and activities with IAEA safety standards Regular invitation of IRRS missions by all IAEA MS is highly recommended The IRRS process is mature and effective – yet it is under continuous development by using novel standardized tools and feedback from participants Lessons learned from past missions as well as mission effectiveness and efficiency analysed Prompt evaluations of IRRS missions to nuclear countries were performed in a pilot phase IRRS enhancements39

40. IAEA IRRS enhancements40 THANK YOU FOR YOUR ATTENTION Photo: V. Friedrich This activity is conducted by the IAEA, with funding by the European Union. The views expressed in this presentation do not necessarily reflect the views of the European Commission