1. IAEA International Atomic Energy Agency

2. IAEA Outline Refreshing IRRS Training - Evaluation of Observations2 Learning Objectives Definitions Formulation of Observations Experience from Past Missions

3. IAEA Learning objectives After finishing this module the trainee will: be aware of the observation types to be offered in IRRS missions; be able to select the observation-type adequate to the review findings; know the process of formulating observations in a mission; have some idea about the statistical properties of the observations. Refreshing IRRS Training - Evaluation of Observations3

4. IAEA DEFINITIONS Refreshing IRRS Training - Evaluation of Observations4

5. IAEA Observations and their types Observations lead to: Recommendation Suggestion Good Practice (Remarkable Performance) Refreshing IRRS Training - Evaluation of Observations5 Findings Observation: focused results of the review process, based on facts and related to possible ways of improvement or achievements to recognise, with reference to requirements in IAEA Safety Standards In brief: observations

6. IAEA Recommendation (1) Recommendations are proposed where aspects relative to the IAEA Safety Requirements are missing, incomplete, or inadequately implemented. Recommendations should be Specific Realistic Designed to result in tangible improvements to regulatory effectiveness (within the individual context of the Host Country) Refreshing IRRS Training - Evaluation of Observations6

7. IAEA Recommendation (2) Recommendations should also be based on IAEA Safety Requirements, with the basis documented (Safety Guide may not be a basis); be formulated in such a way that they are succinct and self-explanatory; clearly specify the responsible party; use “should”-language. Refreshing IRRS Training - Evaluation of Observations7

8. IAEA Recommendation (3) When a recommendation is formulated it should be based on checked and evidenced factual information advise on what improvement in the regulatory regime should be made; not advise on how it should be completed; address the root causes, rather than the symptoms Refreshing IRRS Training - Evaluation of Observations8

9. IAEA Suggestion (1) Suggestions are proposed when opportunities for improvement not directly related to inadequate conformance with IAEA Safety Requirements are identified. Suggestions may contribute to improvements in national regulatory arrangements but are primarily intended to make the regulatory body’s performance more effective or efficient Refreshing IRRS Training - Evaluation of Observations9

10. IAEA Suggestion (2) Suggestions should stimulate the RB’s management and staff to consider new approaches; may be proposed in conjunction with a recommendation; should be based either on IAEA Safety Requirements or Safety Guides, with the basis documented; formulated in such a way that they are succinct and self-explanatory; clearly specify the responsible party; use “should consider”-language. Refreshing IRRS Training - Evaluation of Observations10

11. IAEA Good Practice (1) A good practice is identified in recognition of an outstanding organization, arrangement, programme or performance superior to those generally observed elsewhere A good practice goes beyond the fulfilment of current requirements or expectations; is worthy of the attention of other regulatory bodies as a model might be applied elsewhere; shall reference a basis in IAEA Safety Requirements or Safety Guides; shall be clearly documented in the mission report. Refreshing IRRS Training - Evaluation of Observations11

12. IAEA Good Practice (2) Thus a good practice should have the following characteristics: Has proven benefit; Be replicable and applicable elsewhere; Be effective and creative; Produce good results; Does not contradicts a finding; Be novel or unique. Refreshing IRRS Training - Evaluation of Observations12

13. IAEA FORMULATION OF OBSERVATIONS Refreshing IRRS Training - Evaluation of Observations13

14. IAEA Format of an Observation Basic IRRS Training - Evaluation of Observations14 A description of what the reviewer observed, that has lead to the present finding or good practice Quotation from the relevant IAEA Safety Standard that is to be / has been followed. (There may be more than one Bases) Recommendation / suggestion / good practice offered by the reviewer

15. IAEA Recording Observations During interviews and direct observations, reviewers should make detailed notes of all relevant information gained. On the techniques and content of taking notes see Training Module No. 5 (Methodology of the review) Note here too that the number of observations in an IRRS mission does not represent a measure of the current performance of the reviewed RB relative to any other RB. Refreshing IRRS Training - Evaluation of Observations15

16. IAEA Formulation of Observations The reviewer is expected to formulate in writing his/her observations daily Observations of the day need to be presented on the Daily Team Meeting. Be aware that the first set of findings (R+S) needs to be ready as early as on the third day of the mission!! (For more details see Training Module No. 8 on Scheduling IRRS Missions) Good practices may be formulated later Refreshing IRRS Training - Evaluation of Observations16

17. IAEA Finalization of an Observation Each observation is discussed and agreed upon by the team in Daily Team Meetings Agreed observations are finalized and inserted into the mission report by the reviewer who formulated them Observations agreed by the team should be discussed by the reviewers with the host counterparts for ensuring factual correctness of the observation No agreement on the findings with the host counterpart is needed or sought! Refreshing IRRS Training - Evaluation of Observations17

18. IAEA Use of Action Plans Hosts provide Action Plan(s) based on their Self- Assessment Actions in the AP do not imply that they may not be IRRS findings (think of the follow-up m.) Please consult the AP’s for input! Refreshing IRRS Training - Evaluation of Observations 18

19. IAEA EXPERTS SAY Refreshing IRRS Training - Evaluation of Observations19

20. IAEA R,S & GP should be adequately justified in the text The text should be based on fact and evidence (not assumptions) and the conclusions justified against IAEA’s Standards. The text should be succinct and self-explanatory – not an exhaustive explanation but should include some examples. The focus for balance, harmonization and consistency of R, S & GPs should reflect the main purpose of the mission which is to improve the regulation of nuclear safety for the Host Country and the International Regulatory Community Some of the observations may fit in principle with the concept of Suggestions but are considered by the team to be beneath the threshold for inclusion as formal Suggestions – these should be considered for inclusion as observations in the main text only In the formulation of the mission findings the team leaders should consider the application of the principle of graded approach Refreshing IRRS Training - Evaluation of Observations20 Recommendation of Experienced Reviewers Give particular attention to:

21. IAEA EXPERIENCE FORM PAST MISSIONS Refreshing IRRS Training - Evaluation of Observations21

22. IAEA Collections of Observations Refreshing IRRS Training - Evaluation of Observations22

23. IAEA Statistics of Observations (1) Refreshing IRRS Training - Evaluation of Observations23 All missions Without outliers Follow-up missions

24. IAEA Refreshing IRRS Training - Evaluation of Observations24 Statistics of Observations (2) R + S is negatively correlated with GP In most missions the no. of findings is in the range of 40 - 60 Follow- ups

25. IAEA Statistics of Observations (3) The number of recommendations and that of good practices are strongly negatively correlated Refreshing IRRS Training - Evaluation of Observations25 The number of recommendations and that of suggestions are strongly positively correlated

26. IAEA Performance Indicators (1) Balance of findings: Many Rs implies few GPs and vice versa  N R *N GP  const The No of Rs and that of Ss are nearly proportional  N R /N S  const Refreshing IRRS Training - Evaluation of Observations26

27. IAEA Performance Indicators (2) Action Plan Coverage: No. of findings also in the AP No. of findings not in the AP Refreshing IRRS Training - Evaluation of Observations27

28. IAEA Warning Note once more: The number of observations have certain definite statistical properties, as such their relative values may have orientating merits within a given mission, yet their absolute values should not be used for any comparison of separate missions. Refreshing IRRS Training - Evaluation of Observations28

29. IAEA Summary (1) Three types of observations have been defined: Recommendation Suggestion Good Practice Observations need to refer to IAEA Safety Standards (Requirements or Guides) The various observations have specific features, yet all need to be based on factual information and not on assumptions Inputs: ARM (including SA), Action Plans, interviews, direct obervations Refreshing IRRS Training - Evaluation of Observations29

30. IAEA Summary (2) Reviewers have to take notes and formulate their observations daily The first set of observations is expected to be ready at the end of the third day of the mission Observations are inserted into the mission report in a standard table format The number of recommendations, suggestions and good practices follow certain correlated tendencies. Yet comparison of such numbers between IRRS missions to various countries should not be attempted Refreshing IRRS Training - Evaluation of Observations 30

31. IAEA THANK YOU FOR YOUR ATTENTION Refreshing IRRS Training - Evaluation of Observations31 Photo: V. Friedrich This activity is conducted by the IAEA, with funding by the European Union. The views expressed in this presentation do not necessarily reflect the views of the European Commission

32. IAEA Remarkable Performance* Remarkable performance is identified in recognition of an organization, arrangement, programme or performance that – although is not unique and thus does not qualify for a good practice – is substantially superior to the general international practice Remarkable performance is meant to appreciate high level performance or practice of the RB; in all other attributes is similar to good practices. Refreshing IRRS Training - Evaluation of Observations32 * Under consideration, not yet official