BASIC IRRS TRAINING Lecture 6 Evaluation and Documentation of Observations

Outline Learning Objectives Definitions Formulation of Observations Experience from Past Missions Basic IRRS Training - Evaluation of Observations

Learning objectives After finishing this lecture the trainee will: be aware of the observation types to be offered in IRRS missions; be able to select the observation-type adequate to the review findings; know the process of formulating observations in a mission; have some idea about the statistical properties of the observations. Basic IRRS Training - Evaluation of Observations

definitions Basic IRRS Training - Evaluation of Observations

Observations and their types Observation: focused results of the review process, based on facts and related to possible ways of improvement or achievements to recognise, with reference to requirements in IAEA Safety Standards Observations lead to: Recommendation Suggestion Good Practice (Commendable Performance) Findings In brief: observations Basic IRRS Training - Evaluation of Observations

Recommendation (1) Recommendations are proposed where aspects relative to the IAEA Safety Requirements are missing, incomplete, or inadequately implemented. Recommendations should be Specific Realistic Designed to result in tangible improvements to regulatory effectiveness (within the individual context of the Host Country) Basic IRRS Training - Evaluation of Observations

Recommendation (2) Recommendations should also be based on IAEA Safety Requirements, with the basis documented (Safety Guide may not be a basis); be formulated in such a way that they are succinct and self-explanatory; clearly specify the responsible party; use “should”-language. Basic IRRS Training - Evaluation of Observations

Recommendation (3) When a recommendation is formulated it should be based on checked and evidenced factual information advise on what improvement in the regulatory regime should be made; not advise on how it should be completed; address the root causes, rather than the symptoms Basic IRRS Training - Evaluation of Observations

Suggestion (1) Suggestions are proposed when opportunities for improvement not directly related to inadequate conformance with IAEA Safety Requirements are identified. Suggestions may contribute to improvements in national regulatory arrangements but are primarily intended to make the regulatory body’s performance more effective or efficient Basic IRRS Training - Evaluation of Observations

Suggestion (2) Suggestions should stimulate the RB’s management and staff to consider new approaches; may be proposed in conjunction with a recommendation; should be based either on IAEA Safety Requirements or Safety Guides, with the basis documented; formulated in such a way that they are succinct and self-explanatory; clearly specify the responsible party; use “should consider”-language. Basic IRRS Training - Evaluation of Observations

Good Practice (1) A good practice is identified in recognition of an outstanding organization, arrangement, programme or performance superior to those generally observed elsewhere A good practice goes beyond the fulfilment of current requirements or expectations; is worthy of the attention of other regulatory bodies as a model might be applied elsewhere; shall reference a basis in IAEA Safety Requirements or Safety Guides; shall be clearly documented in the mission report. Basic IRRS Training - Evaluation of Observations

Thus a good practice should have the following characteristics: Has proven benefit; Be replicable and applicable elsewhere; Be effective and creative; Produce good results; Does not contradicts a finding; Be novel or unique. Basic IRRS Training - Evaluation of Observations

Commendable Performance Where an innovative activity is observed which clearly strengthens regulatory effectiveness and/or organisational efficiency but is not definably ‘superior to those generally observed elsewhere’ it may be referred to as commendable performance (CP).   CP will be worthy of the attention of other regulatory bodies as a model in the general drive for excellence Not yet officially introduced Basic IRRS Training - Evaluation of Observations

Important warning The number of Recommendations, Suggestions and Good Practices is no way a measure of the status of the regulatory body. Comparison of such numbers between IRRS missions to different countries should not be attempted! Basic IRRS Training - Evaluation of Observations

Formulation of observations Basic IRRS Training - Evaluation of Observations

Recording Observations During interviews and direct observations, reviewers should make detailed notes of all relevant information gained. On the techniques and content of taking notes see Training Lecture No. 5 (Methodology of the review) Note here too that the number of observations in an IRRS mission does not represent a measure of the current performance of the reviewed RB relative to any other RB. Basic IRRS Training - Evaluation of Observations

Formulation of Observations The reviewer is expected to formulate in writing his/her observations daily Observations of the day need to be presented on the Daily Team Meeting. Be aware that the first set of findings (R+S) needs to be ready as early as on the third day of the mission!! (For more details see Training Lecture No. 8 on Scheduling IRRS Missions) Good practices may be formulated later Basic IRRS Training - Evaluation of Observations

Format of an Observation A description of what the reviewer observed, that has lead to the present finding or good practice Quotation from the relevant IAEA Safety Standard that is to be / has been followed. (There may be more than one Bases) Recommendation / suggestion / good practice offered by the reviewer Basic IRRS Training - Evaluation of Observations

Finalization of an Observation Each observation is discussed and agreed upon by the team in Daily Team Meetings Agreed observations are finalized and inserted into the mission report by the reviewer who formulated them Observations agreed by the team should be discussed by the reviewers with the host counterparts for ensuring factual correctness of the observation No agreement on the findings with the host counterpart is needed or sought! Basic IRRS Training - Evaluation of Observations

Experts say Basic IRRS Training - Evaluation of Observations

Recommendation of Experienced Reviewers Give particular attention to: R,S & GP should be adequately justified in the text The text should be based on fact and evidence (not assumptions) and the conclusions justified against IAEA’s Standards. The text should be succinct and self-explanatory – not an exhaustive explanation but should include some examples. The focus for balance, harmonization and consistency of R, S & GPs should reflect the main purpose of the mission which is to improve the regulation of nuclear safety for the Host Country and the International Regulatory Community Some of the observations may fit in principle with the concept of Suggestions but are considered by the team to be beneath the threshold for inclusion as formal Suggestions – these should be considered for inclusion as observations in the main text only In the formulation of the mission findings the team leaders should consider the application of the principle of graded approach Basic IRRS Training - Evaluation of Observations

Experience from past missions to nuclear countries Basic IRRS Training - Evaluation of Observations

Collections of Observations Basic IRRS Training - Evaluation of Observations

Statistics of Observations (1) Basic IRRS Training - Evaluation of Observations

Statistics of Observations (2) The number of findings is negatively correlated with that of Good Practices Basic IRRS Training - Evaluation of Observations

Statistics of Observations (3) The number of recommendations and that of suggestions are positively correlated (with exceptions) The number of recommendations and that of good practices are strongly negatively correlated Basic IRRS Training - Evaluation of Observations

Warning Note once more: The number of observations have certain definite statistical properties, as such their relative values may have orientating merits within a given mission, yet their absolute values should not be used for any comparison of separate missions. Basic IRRS Training - Evaluation of Observations

Summary (1) Three types of observations have been defined: Recommendation Suggestion Good Practice Observations need to refer to IAEA Safety Standards (Requirements or Guides) The various observations have specific features, yet all need to be based on factual information and not on assumptions Basic IRRS Training - Evaluation of Observations

Summary (2) Reviewers have to take notes and formulate their observations daily The first set of observations is expected to be ready at the end of the third day of the mission Observations are inserted into the mission report in a standard table format The number of recommendations, suggestions and good practices is in no way a measure of the status of the regulatory body. Comparison of such numbers between IRRS missions to various countries should not be attempted Basic IRRS Training - Evaluation of Observations

THANK YOU FOR YOUR ATTENTION This activity is conducted by the IAEA, with funding by the European Union. The views expressed in this presentation do not necessarily reflect the views of the European Commission Photo: V. Friedrich Basic IRRS Training - Evaluation of Observations

Discussion What is the difference between a Recommendation and a Suggestion? When will you offer a Good Practice? How can the numbers of various observations be used as some kind of measures of performance? Basic IRRS Training - Evaluation of Observations