1. IRRS REFRESHER TRAINING Lecture 2
Methodology of the review

2. Outline PART I PART II General review principles Team work
Daily team meetings
Counterpart and reviewer involvement in data collection
PART II
Reviewing additional written material
Taking notes
Interviews
Direct observations
Site visits
Refresher IRRS Training - Methodology of the Review

3. Learning objectives
After finishing this lecture the trainee will understand:
The general review principles behind an IRRS mission
How team work is organized and conducted in IRRS
The purpose, conduct and output of Daily Team Meetings
The roles of reviewers and counterparts in data collection
The role, methodology and contribution of different data collection methods in IRRS: review of additional documentation, note taking, interviews, direct observation, site visits
Refresher IRRS Training - Methodology of the Review

4. The Great Challenge
The IRRS team has, in a short time and in a new environment, to:
look for issues
identify findings
agree findings with counterpart
write a report
This Training Lecture provides recommendations on how to work in order to accomplish these tasks in the best way
Refresher IRRS Training - Methodology of the Review

5. Contents of Part I General review principles IRRS Team Goal Team work
Daily team meetings
Counterpart and reviewer involvement in data collection
Refresher IRRS Training - Methodology of the Review

6. General review principles
Refresher IRRS Training - Methodology of the Review

7. General review principles (1)
The reviewer should be
Well prepared:
Use the IRRS Guidelines, IAEA Safety Standards, ARM including the self-assessment and the action plan
Reviewers should not refer to their national practice!
Efficient:
Open minded, rely on facts, look for examples
Take notes
Professional
Team member
Refresher IRRS Training - Methodology of the Review

8. General review principles (2)
Remember:
As a reviewer you always have to be flexible and receptive
Recognize that there are several ways to accomplish a certain objective or task
Do not try to impose your own solutions on the counterpart but give examples if necessary to illustrate an issue
A relaxed interview situation is best for all parties
You and your counterpart are colleagues, not competitors or adversaries, and should share the same interest in having a constructive peer-review
Refresher IRRS Training - Methodology of the Review

9. Team Work
Refresher IRRS Training - Methodology of the Review

10. IRRS Team Goal What are we here to do?
A joint search for improvement of nuclear regulation through providing recommendations and suggestions, and to illustrate good practices
Review against the IAEA Safety Standards and best international practice by
reviewing documents
interviewing
observing and
Write a report consistent with the IRRS Template, and all in a sprit of good teamwork
Refresher IRRS Training - Methodology of the Review

11. Team work, methodology (1)
An IRRS mission can not succeed without good team work. Good team workers:
Support and help other team members
Seek guidance or assistance from other team members
Use available good English speakers for language review
Accept and use critical comments
Do not be defensive
Critical comments from the TL/DTL should be understood as a challenge to the issue, not a criticism of the team member´s work
Refresher IRRS Training - Methodology of the Review

12. Team work, methodology (2)
Seek and accept assistance or guidance from the TL or DTL early with respect to:
Problems with counterparts
Problems with interpreters
Difficulties in writing working notes
Personal problems
Problems with team activities or relationships
Let the team leader know if your counterpart invites you home or out
Refresher IRRS Training - Methodology of the Review

13. Team work, expectations (1)
Be open minded and critical in thought, objective in discussions and writing
Work hard:
Evenings are work time
Write and update working notes
Define your issues
Prepare for the next day
Relax on free Sunday and enjoy the social programme
Prioritize your effort
You can not cover everything in IRRS Guidelines to the same depth
Use discussions and observations to identify performance and importance
Refresher IRRS Training - Methodology of the Review

14. Team work, expectations (2)
Use
IAEA Safety Standards
The summary reports of the IRRS findings as reference material
Prepare well for team meetings
Focus on potential issues
Respect the time of others
Stay on schedule
Review work - daily
Working notes writing - daily
Remember: you represent IAEA
Be a team worker, help other reviewers!
The Team Leaders are there to support you all!
Refresher IRRS Training - Methodology of the Review

15. Daily team meetings
Refresher IRRS Training - Methodology of the Review

16. Daily team meetings, purpose
Meeting of the whole team and the host country Liaison Officer at the end of each working day, until the first draft of the mission report has been completed. Its purpose is
To discuss the main observations of the day
To share findings and insights regarding technical and policy issues related to the different review areas
Refresher IRRS Training - Methodology of the Review

17. … are not meant for general activity reporting by reviewers
Daily team meetings
… are not meant for general activity reporting by reviewers
The team leader has to establish an effective style and conduct of these meetings to prevent wasting reviewer´s time meant to work on their reports
Refresher IRRS Training - Methodology of the Review

18. Daily team meetings, conduct
Common mistakes:
The meeting has no standardized structure
The TL asks around the table for the day´s impressions
The team members have not prepared to present conclusions and concerns only, but provide a general report about the activities of the day
The TL does not limit the time for presentations and discussions, the meeting can take several hours
The TL does not provide clear conclusions and directions for the further work
The Liaison Officer interferes in discussions in a defensive way rather than answering questions and providing clarifications
Refresher IRRS Training - Methodology of the Review

19. Daily team meetings, conduct (3)
Summary of Daily Team Meetings
Is meant to summarize daily observations and cross-cutting issues, it is not an activity report
Team members need to thoroughly prepare the report to the meeting Group-wise
Both the Group-reports and the entire meeting need to be limited in time
Should be closed with conclusions and instructions for the next day
Refresher IRRS Training - Methodology of the Review

20. Counterpart and reviewer involvement in data collection
Refresher IRRS Training - Methodology of the Review

21. Counterpart involvement
The reviewer should encourage the counterpart to:
Actively contribute
Help obtain correct and complete information
Assist in the search for improvement
Develop a sense of ownership
Keep an open minded approach
Not to be defensive
Refresher IRRS Training - Methodology of the Review

22. Reviewer involvement The reviewer needs to Make preparations
Control the length of host organization´s presentations
Ask questions for clarification
Concentrate on important aspects
Keep copious notes
Check related documentation
Check processes or records
Check any performance statistics/indicators
Refresher IRRS Training - Methodology of the Review

23. END OF PART I
Refresher IRRS Training - Methodology of the Review

24. Data collection methods
Contents of Part II
Data collection methods
Reviewing additional written material
Interviews
Good interviewing practices
Direct observations
Site visits
Refresher IRRS Training - Methodology of the Review

25. additional written material
Refresher IRRS Training - Methodology of the Review

26. Requesting and reviewing additional written material (1)
During the mission, additional material will need to be requested
Typical examples are:
Lower level Management System documents, such as procedures and work instructions
Annual activity planning and follow up documents
Training plans and programmes for the regulatory staff
Inspection reports
Event reports
Safety analysis reports or parts of such reports
MoU with other national bodies
Refresher IRRS Training - Methodology of the Review

27. Requesting and reviewing additional written material (2)
For some documents full translations can often not be provided, then
Ask for a translated summary
Ask the counterpart to explain the contents of the document
Use the Google translator, if possible
The regulatory body´s intranet may contain the required documents
Refresher IRRS Training - Methodology of the Review

28. interviews
Refresher IRRS Training - Methodology of the Review

29. Purpose of interviews
to gather information not covered by the written material
to seek clarification of written information provided
to provide an opportunity for host regulatory body staff to discuss their practices and professional opinions with the reviewers
Refresher IRRS Training - Methodology of the Review

30. Good interviewing practices (1)
Prepare properly for each interview:
Review all documents and activities available prior to the interview
Develop several questions to cover the area
Formulate to avoid simple answers like ”yes” and ”no”
Ask questions only to persons who are familiar with the area under review
Refresher IRRS Training - Methodology of the Review

31. Good interviewing practices (2)
Make sure your words are clear and easy to understand
Keep the right distance between you and the interviewee, consider cultural aspects
Control your body language and try to interpret the body language of your interviewee
Maintain a professional attitude at all times
Refresher IRRS Training - Methodology of the Review

32. Good interviewing practices (3)
At the very beginning
Let the interviewee calm down and show a positive attitude
Explain review targets*
Ask for a short presentation, if needed, just to explain main activities, roles and responsibilities in the reviewing area
* For nervous interviewees, emphasise that you ask because you would like to know how the organization works, not in order to inspect or to assess individual performance
Refresher IRRS Training - Methodology of the Review

33. Good interviewing practices (4)
How to ask?
Keep questions simple and focussed on facts
Use open ended questions
Use a funnel approach: from broad leading to specific questions
Avoid devious or trick questions
Refresher IRRS Training - Methodology of the Review

34. Good interviewing practices (5)
How to listen?
Be interested - show enthusiasm
Maintain eye contact
Listen to the answer before asking next question
If you don´t understand the answer, ask for clarification
Do not let note taking interfere with your listening
Request copies of documents for later study
Don´t assume, ask questions
Refresher IRRS Training - Methodology of the Review

35. Direct observations
Refresher IRRS Training - Methodology of the Review

36. Direct observations (1)
Opportunities to observe work activities should be taken both at the regulatory body offices and during site visit
Activities to observe may be
meetings,
audits,
inspections,
exercises, training sessions
Use of electronic documentation
Refresher IRRS Training - Methodology of the Review

37. Direct observations (2)
Remember:
A precondition for using observation as a data source for conclusions is that the context of the observed activity is understood by the reviewer
Prepare by checking relevant procedures and if necessary ask the counterpart to explain the significance of the activity and to translate (discretely) during the activity
Be aware that presence of an observer could affect the observed activity
Observations should be properly recorded in technical notes and reflected in the mission report.
Refresher IRRS Training - Methodology of the Review

38. Site visits
Refresher IRRS Training - Methodology of the Review

39. Site visits, purpose
Certain interviews and direct observations of regulatory practices and activities (typically inspections) should be carried out on the sites of authorized organizations
Such visits will enable a better understanding of how the regulatory body discharges its responsibility and give an opportunity for reviewers to meet with licensees
Refresher IRRS Training - Methodology of the Review

40. Site visits, types of information
Through observations of work activities and interviews with the regulatory body staff and the licensee staff types of information to be gathered may include:
Perceived and actual roles and responsibilities of the regulator at the visited site
Resources to fulfil those responsibilities, including facilities, equipment and staffing
Knowledge, skills and abilities of the regulatory inspectors to perform inspections
Effectiveness of the regulatory processes
Elements relevant to policy issues
Relations between the regulatory body and the licensee
Refresher IRRS Training - Methodology of the Review

41. Site visits, conduct
The visit should start with an opening statement to the site host informing about the scope of the IRRS mission, the purpose of the visit and questions to be asked
It is important that staff of the visited facility understand in advance that the visit is not an IAEA review of their facility but is conducted solely to observe practices and actions of the regulatory staff
Refresher IRRS Training - Methodology of the Review

42. Site visits, conduct (2)
On site the IRRS reviewers should concentrate on observing the site inspector conducting a normal inspection or other routine regulatory tasks
Before the walk around, the inspector should brief the reviewers about the planning process for the inspection
During the walk around the reviewers should act as observers and not interfere in the inspection activities
Reviewers may ask the inspector to explain what he is doing if this does not disturb his/her activities
After the inspection, the inspector should inform the reviewers about his/her observations and conclusions and what the next steps are
Refresher IRRS Training - Methodology of the Review

43. Site visits, conduct (3)
Such information will provide a valuable complement to the information provided by the regulatory body itself
During interviews with licensees, the preference is that host regulatory body staff is not present, in order to facilitate open and frank discussions
After the visit, reviewers should discuss their findings with the full IRRS team at the next Daily Team Meeting
Refresher IRRS Training - Methodology of the Review

44. END OF PART II
Refresher IRRS Training - Methodology of the Review

45. Summary (1) An IRRS is not possible without good team work
Every reviewer needs to prepare well and be efficient, professional and skilled
Reviewers are assigned to review groups (A-F)
Good team workers are flexible and open minded
The daily team meeting is the most important instrument for discussing conclusions and agreeing recommendations, suggestions and good practices
Refresher IRRS Training - Methodology of the Review

46. Summary (2) To collect data during the mission:
Additional documentation needs to be reviewed
Copious notes should be taken at the earliest opportunity
Lists of questions should be prepared for interviews
Interviews should be conducted in a friendly and professional manner and encourage the counterpart to provide substantial information, don´t assume – ask
Direct observation of work activities could provide valuable additional information
Site visits are vital for understanding of the regulatory effectiveness and the relations with the licensees
Refresher IRRS Training - Methodology of the Review

47. THANK YOU FOR YOUR ATTENTION
This activity is conducted by the IAEA, with funding by the European Union. The views expressed in this presentation do not necessarily reflect the views of the European Commission
Photo: V. Friedrich
Refresher IRRS Training - Methodology of the Review