1. BASIC IRRS TRAINING Lecture 6
Evaluation and Documentation of Observations

2. Outline Learning Objectives Definitions Formulation of Observations
Experience from Past Missions
Basic IRRS Training - Evaluation of Observations

3. Learning objectives After finishing this lecture the trainee will:
be aware of the observation types to be offered in IRRS missions;
be able to select the observation-type adequate to the review findings;
know the process of formulating observations in a mission;
have some idea about the statistical properties of the observations.
Basic IRRS Training - Evaluation of Observations

4. definitions
Basic IRRS Training - Evaluation of Observations

5. Observations and their types
Observation: focused results of the review process, based on facts and related to possible ways of improvement or achievements to recognise, with reference to requirements in IAEA Safety Standards
Observations lead to:
Recommendation
Suggestion
Good Practice
(Commendable Performance)
Findings
In brief: observations
Basic IRRS Training - Evaluation of Observations

6. Recommendation (1)
Recommendations are proposed where aspects relative to the IAEA Safety Requirements are missing, incomplete, or inadequately implemented.
Recommendations should be
Specific
Realistic
Designed to result in tangible improvements to regulatory effectiveness (within the individual context of the Host Country)
Basic IRRS Training - Evaluation of Observations

7. Recommendation (2) Recommendations should also
be based on IAEA Safety Requirements, with the basis documented (Safety Guide may not be a basis);
be formulated in such a way that they are succinct and self-explanatory;
clearly specify the responsible party;
use “should”-language.
Basic IRRS Training - Evaluation of Observations

8. Recommendation (3) When a recommendation is formulated it should
be based on checked and evidenced factual information
advise on what improvement in the regulatory regime should be made;
not advise on how it should be completed;
address the root causes, rather than the symptoms
Basic IRRS Training - Evaluation of Observations

9. Suggestion (1)
Suggestions are proposed when opportunities for improvement not directly related to inadequate conformance with IAEA Safety Requirements are identified.
Suggestions may contribute to improvements in national regulatory arrangements but are primarily intended to make the regulatory body’s performance more effective or efficient
Basic IRRS Training - Evaluation of Observations

10. Suggestion (2) Suggestions
should stimulate the RB’s management and staff to consider new approaches;
may be proposed in conjunction with a recommendation;
should be based either on IAEA Safety Requirements or Safety Guides, with the basis documented;
formulated in such a way that they are succinct and self-explanatory;
clearly specify the responsible party;
use “should consider”-language.
Basic IRRS Training - Evaluation of Observations

11. Good Practice (1)
A good practice is identified in recognition of an outstanding organization, arrangement, programme or performance superior to those generally observed elsewhere
A good practice
goes beyond the fulfilment of current requirements or expectations;
is worthy of the attention of other regulatory bodies as a model
might be applied elsewhere;
shall reference a basis in IAEA Safety Requirements or Safety Guides;
shall be clearly documented in the mission report.
Basic IRRS Training - Evaluation of Observations

12. Thus a good practice should have the following characteristics:
Has proven benefit;
Be replicable and applicable elsewhere;
Be effective and creative;
Produce good results;
Does not contradicts a finding;
Be novel or unique.
Basic IRRS Training - Evaluation of Observations

13. Commendable Performance
Where an innovative activity is observed which clearly strengthens regulatory effectiveness and/or organisational efficiency but is not definably ‘superior to those generally observed elsewhere’ it may be referred to as commendable performance (CP).  
CP will be worthy of the attention of other regulatory bodies as a model in the general drive for excellence
Not yet officially introduced
Basic IRRS Training - Evaluation of Observations

14. Important warning
The number of Recommendations, Suggestions and Good Practices is no way a measure of the status of the regulatory body. Comparison of such numbers between IRRS missions to different countries should not be attempted!
Basic IRRS Training - Evaluation of Observations

15. Formulation of observations
Basic IRRS Training - Evaluation of Observations

16. Recording Observations
During interviews and direct observations, reviewers should make detailed notes of all relevant information gained.
On the techniques and content of taking notes see Training Lecture No. 5 (Methodology of the review)
Note here too that the number of observations in an IRRS mission does not represent a measure of the current performance of the reviewed RB relative to any other RB.
Basic IRRS Training - Evaluation of Observations

17. Formulation of Observations
The reviewer is expected to formulate in writing his/her observations daily
Observations of the day need to be presented on the Daily Team Meeting.
Be aware that the first set of findings (R+S) needs to be ready as early as on the third day of the mission!! (For more details see Training Lecture No. 8 on Scheduling IRRS Missions)
Good practices may be formulated later
Basic IRRS Training - Evaluation of Observations

18. Format of an Observation
A description of what the reviewer observed, that has lead to the present finding or good practice
Quotation from the relevant IAEA Safety Standard that is to be / has been followed.
(There may be more than one Bases)
Recommendation / suggestion / good practice offered by the reviewer
Basic IRRS Training - Evaluation of Observations

19. Finalization of an Observation
Each observation is discussed and agreed upon by the team in Daily Team Meetings
Agreed observations are finalized and inserted into the mission report by the reviewer who formulated them
Observations agreed by the team should be discussed by the reviewers with the host counterparts for ensuring factual correctness of the observation
No agreement on the findings with the host counterpart is needed or sought!
Basic IRRS Training - Evaluation of Observations

20. Experts say
Basic IRRS Training - Evaluation of Observations

21. Recommendation of Experienced Reviewers Give particular attention to:
R,S & GP should be adequately justified in the text The text should be based on fact and evidence (not assumptions) and the conclusions justified against IAEA’s Standards. The text should be succinct and self-explanatory – not an exhaustive explanation but should include some examples.
The focus for balance, harmonization and consistency of R, S & GPs should reflect the main purpose of the mission which is to improve the regulation of nuclear safety for the Host Country and the International Regulatory Community
Some of the observations may fit in principle with the concept of Suggestions but are considered by the team to be beneath the threshold for inclusion as formal Suggestions – these should be considered for inclusion as observations in the main text only
In the formulation of the mission findings the team leaders should consider the application of the principle of graded approach
Basic IRRS Training - Evaluation of Observations

22. Experience from past missions to nuclear countries
Basic IRRS Training - Evaluation of Observations

23. Collections of Observations
Basic IRRS Training - Evaluation of Observations

24. Statistics of Observations (1)
Basic IRRS Training - Evaluation of Observations

25. Statistics of Observations (2)
The number of findings is negatively correlated with that of Good Practices
Basic IRRS Training - Evaluation of Observations

26. Statistics of Observations (3)
The number of recommendations and that of suggestions are positively correlated (with exceptions)
The number of recommendations and that of good practices are strongly negatively correlated
Basic IRRS Training - Evaluation of Observations

27. Warning Note once more:
The number of observations have certain definite statistical properties,
as such their relative values may have orientating merits within a given mission,
yet their absolute values should not be used for any comparison of separate missions.
Basic IRRS Training - Evaluation of Observations

28. Summary (1) Three types of observations have been defined:
Recommendation
Suggestion
Good Practice
Observations need to refer to IAEA Safety Standards (Requirements or Guides)
The various observations have specific features, yet all need to be based on factual information and not on assumptions
Basic IRRS Training - Evaluation of Observations

29. Summary (2)
Reviewers have to take notes and formulate their observations daily
The first set of observations is expected to be ready at the end of the third day of the mission
Observations are inserted into the mission report in a standard table format
The number of recommendations, suggestions and good practices is in no way a measure of the status of the regulatory body. Comparison of such numbers between IRRS missions to various countries should not be attempted
Basic IRRS Training - Evaluation of Observations

30. THANK YOU FOR YOUR ATTENTION
This activity is conducted by the IAEA, with funding by the European Union. The views expressed in this presentation do not necessarily reflect the views of the European Commission
Photo: V. Friedrich
Basic IRRS Training - Evaluation of Observations

31. Discussion
What is the difference between a Recommendation and a Suggestion?
When will you offer a Good Practice?
How can the numbers of various observations be used as some kind of measures of performance?
Basic IRRS Training - Evaluation of Observations