Readiness Review Course

Readiness Review Course
Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Timelines, More Roles/Responsibilities, and Meetings Document Evaluation Process Interviewing Techniques Observation Process Finding Categorization Process Final Report Format and Administration Nov 2006

Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Timelines, More Roles/Responsibilities, and Meetings Nov 2006

Readiness Review Process
Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Initial Team Mtg Pre-Site Visit, IP Drafted) 6 Months Prior to RR 1 Year Prior to RR Generally 2 weeks Nov 2006 Mod 12

Learning Objectives Understand Readiness Review Timelines
Identify Team Leader and Member Responsibilities Plan and Organize Opening, Daily, and Closeout Meetings Nov 2006

Development of Implementation Plan,
ORR Time Line POA Approval Initial Pre- review visit Site Introduction meeting Conduct Review Close-out meeting Follow-up Review Team meeting ORR Report Development of Implementation Plan, Generation & CRADs, & Checklists Review Nov 2006 Mod 12

Typical RR Timeline SUN MON TUE WED THUR FRI SAT ARRIVE Opening
Records Operations Operations Interviews Report Meeting Review Writing Drills Drills Records Mangmt.. Review Interviews Mangmt.. Interviews SUN MON TUE WED THUR FRI SAT Off Finish Finish Closeout Forms Forms Continue Thursday and Friday, as needed; Report is signed and Submitted . . . Catch Up Peer Rev. Peer Rev. Report Report Sections Sections Brief Prep Brief Prep Nov 2006 Mod 12

Team Leader Responsibilities During the Review
Ensure all team members have completed all access requirements prior to the review; Ensure all team members have completed all required reading prior to the review; Conducts the following meetings: Opening (kickoff); Conducts the Daily (team and site personnel); Conducts Daily Site Manager Briefings (as needed); Conducts Final Team; Conducts Final Out Brief. Nov 2006 Mod 12

Team Members Responsibilities During the Review
Developing and Completing Form 1s and 2s: Provides draft forms to the Team Leader as soon as possible; Updates forms on a daily basis; Organizing and coordinating factual accuracy reviews; Completed Form 1s and 2s means that they have been: Completed, signed, and submitted to the Team Leader; Factual accuracy of the individual forms has been confirmed with the counterparts; Accepted by the Team Leader before the closeout meeting and before the team member leaves the site; Summary evaluations completed, submitted to, and signed-off by Team Leader prior to the closeout meeting. Nov 2006 Mod 12

Maintains accurate and complete notes of activities performed; Updates schedule of activities on a daily basis; Provides support in developing a final conclusion for following subject areas: Intergraded Safety Management; Lessons Learned; Adequacy of contractors review; and As assigned by Team Leader. Supports and attends ALL meeting as requested by Team Leader; Nov 2006 Mod 12

Work with other ORR/RA members: to resolve and communicate issues; to prepare the review documentation and report; to eliminate redundant reviews. Submit completed review documents for review and approval. Concur with determination of operational readiness and conclusions of ORR/RA report (in their area of assessment). Nov 2006 Mod 12

An Effective ORR/RA Requires Effective Coordination
Avoid duplication of: Interviews, procedure reviews; Drills, exercise observations . . . Know time line of the review Schedule driven; Defined sequences; Stay on point, avoid distractions; Preparation of report forms critical and critical path . . . Nov 2006 Mod 12

Meetings Opening (Kickoff) Meeting;
Daily Team and Site Personnel Meeting; Daily Site Management Briefing, if needed; Final Team Meeting; Final Out Brief Meeting. Nov 2006 Mod 12

Opening (Kickoff) Meeting
Team Leaders Meeting Make introductions. Allow for a series of briefs by the site explaining the site's management and organization, its functions, operations and interfaces. Brief the audience on the ORR process and the ORR schedule. Brief site audience and team on the rules for interaction between team members and site personnel. Nov 2006 Mod 12

Opening (Kickoff) Meeting (Cont)…
Rules for Interaction Between Team Members and Site Personnel: The team will strive to be invisible so as to not influence the performance of the facility; Request that all operators be briefed on these rules and in particular that no action should be taken in response to questions or discussion by the team; The team will initiate action, preferable through a supervisor or escort, if an immediate unsafe condition is encountered. Nov 2006 Mod 12

Rules for Interaction Between Team Members and Site Personnel: The team will not act as safety monitors or direct operations; The team will refrain from distracting the operators; The team will not violate site rules, cross boundaries without authority, violate control room/area boundaries or in any other way become an initiating event during the ORR; The team will not act as evolution or drill observers; Nov 2006 Mod 12

Daily Team and Site Personnel Meeting Meeting
Performed at the end of each day Team Meeting First – Site Personnel Meeting Second; All Team Members Present; Site Personnel Meeting includes Management and all POCs; Remind site personnel that they are observers not participators. Nov 2006 Mod 12

Daily Team and Site Personnel Meeting
Identification of any past, new, or unclear issues; Re-emphasize who is responsible for what area; Discuss and restate scope of review; Discuss process for going beyond the scope of the review; Discuss process for making contact outside of their assigned areas (e.g., interviews with senior management); Discuss validation of facts; Keep a running record of all the issues; Nov 2006 Mod 12

Daily Site Management Briefing
Held only if requested by management (if management cannot attend daily meetings) Presents current and potential findings; Presents status of the review and support; Discusses any changes if needed. Nov 2006 Mod 12

Final Team Meeting Last opportunity to work and clarify any questions with Team Members; Due prior to this meeting are all: Final Forms 1 and 2 Draft summary evaluations (ISM, Lessons Learned, Contractor Eval, Summary Conclusion) Last opportunity to discover/address dissenting opinions and conflicts Any such issues must be addressed at this time to avoid any confusion at the closeout meeting. Meeting for TEAM Members ONLY, since no new information will be discussed or presented. Nov 2006 Mod 12

Final Out Brief Meeting
Team Leader informs the site management of the major issues identified during the ORR; The closeout meeting kept short; Invite Site Management and POCs; No surprises at this meeting – no new issues or findings; As a goal, provide a draft copy of the final report to the senior managers; At a minimum, prepare the executive summary of the draft report as a handout to everyone attending the briefing; Nov 2006 Mod 12

Team Leader informs the site management of the major issues identified during the ORR; Discuss the final report schedule and deliverables; Teams consensus will be presented: Their final conclusion, Any major issues, and Overall themes, if any will be presented. Exit briefing based on the Forms 1, Forms 2, and Team’s conclusions; Nov 2006 Mod 12

Questions Nov 2006 23

Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Document Evaluation Process Nov 2006

Learning Objectives Describe the preparation for onsite document review; Discuss the ORR/RA principle of sampling versus auditing; Discuss the complete trail sampling technique; Discuss good Team Member record / note keeping practices related to records review; Nov 2006

Document Evaluation Process
Preparation Execution Follow-up Reporting Nov 2006 Mod 12

Document Evaluation Process Preparation
Request documents before the Review Request those which address or describe the following Regulatory Drivers (DOE Orders, Federal Law) Safety Basis Drivers (DSA, TSR, etc.) Facility Procedures Process Procedures Nov 2006 Mod 12

Example of Documents to Request- Hierarchy
Requirements Objective Core Orders, Codes, Requirements Standards, Regulations Criteria Identification of Required Program & Processes Identification of important implementing Procedures / Policies Identification of important Systems, Structures, Components Identification of important Job Factors Nov 2006 Mod 12

Performance Review Flow
Top Tier Requirement Performance Review Flow DOE Site Office Requirements Requirement Specific Contract (what) Requirements Company Policy / Procedure Manual Project Procedures Implementation Specific (how) Line Instructions Result Project Records Nov 2006 Mod 12

Request pertinent documents; Sampling - Not a complete document audit; Communicate with Contractor counterpart to determine appropriate documents; Communicate with the Team; Avoid duplication of review, if possible Ensure that Team Members are aware of documents that may affect their review Prioritize the selected target documents for review. Nov 2006 Mod 12

Document Evaluation Process Execution
Read and understand the documents Determine if document contains required information to address applicable Criteria, Review and Approach Document; Determine if regulatory requirements are met or described; Determine how document is controlled and approved; Determine if procedures have been written correctly. Nov 2006 Mod 12

Do procedures accurately document and implement the following: Regulatory Drivers . . . Programs and Processes . . . Commitments . . . DSA/SAR / TSR / Accident assumptions; BIO / JCO / FHA, and EIS; List A/B, WSS, S/RIDs; Nov 2006 Mod 12

Do procedures accurately define and communicate Roles and Responsibilities for individuals as they relate to specific requirements.. Are they understood by those identified individuals . . . Do they comply with “writing guides”.. Nov 2006 Mod 12

Document Reviews do not have to be line by line copies of Order or regulatory driver, but must: Detail the mechanisms to fulfill the intent and requirements of the regulatory/contract driver . . . Detail the mechanisms and requirements to adequately control and document the activity . . . Focus on implementation . . . Nov 2006 Mod 12

Document Evaluation Process Follow-up
Follow-up is needed if you are “Pulling the String”; Request additional documents if needed; Check beyond the document evaluation Training Controls to ensure procedures are followed Document configuration control Nov 2006 Mod 12

Document Evaluation Process Reporting
Record the information relating to the acceptance and review criteria Check reference information Integrate information gained into observation and interview processes Discuss conclusions with Team Leader and Team Members as applicable Take Accurate Notes Nov 2006 Mod 12

Notes and Records Your personal permanent record of the review . . .
Thorough enough to reconstruct: Names; Titles; Dates, Times; Specific Documents; Equipment identification; Events; Quotations or paraphrasing; Transfer to Forms daily – “Write Every Night”! Nov 2006 Mod 12

Summary… Does the documentation accurately and completely implement the requirements: Contractual? Authorization Basis? Programmatic? Performance? Is the documentation being implemented and requirements met? Is “Flow-Down” consistent, complete, and accurate? Nov 2006 Mod 12

Conduct of the Review – Mod 12 Interviewing Techniques
Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Interviewing Techniques Nov 2006

Learning Objectives Describe the two major types of interviews;
Describe how to prepare for an interview; Compare the strengths and weaknesses of various questioning techniques; Discuss the timing / scheduling of interviews during an ORR/RA. Nov 2006

Two Types of Interviews
Management Learn how they do business Address issues discovered during review of their processes and procedure documentation Level of Knowledge Systems Safety Basis Processes Precautions/Hazards Principles Nov 2006 Mod 12

Interviewing Fundamentals of Good Interviewing Planning Opening
Questioning Closing Post Interview Nov 2006 Mod 12

Fundamentals of Good Interviewing - Planning
Plan questions in advance Define the purpose of the interview Keep the interview on track Focus on one issue at a time Cover important subject matters Gather data that will provide knowledge Review previous findings related to the topic Know the interviewee’s functions and responsibilities Control the interview setting (if possible) Nov 2006 Mod 12

Fundamentals of Good Interviewing - Opening
Set the tone State the purpose of the interview Start with general topic and move to more detail as the interview progresses Obtain the title of the interviewee Establish and keep control of the interview Put the interviewee at ease State you will be taking notes and why Nov 2006 Mod 12

Fundamentals of Good Interviewing - Questioning
Ask clear and concise questions Avoid leading questions Allow interviewee time to think through the response Keep interview on track Restate questions Ask follow-up questions Ask for clarification or basis of opinion Get complete answer to one question before progressing to the next Ask for illustrations or examples of documents alluded to by the interviewee Limit participation to two team members per interviewee ABOVE ALL - LISTEN CAREFULLY Nov 2006 Mod 12

Example Interview Questions
Leading “How often do you discuss policy with a worker?” Neutral “How do the workers gain knowledge of company policy?” Primary “What tools are required by the Preventative Maintenance Procedure?” Secondary “What tools are actually used?” “Why are all the other tools taken to the work site?” Nov 2006 Mod 12

Open “Can you describe your relationship with your subordinates?” “Could you describe how preventative maintenance is scheduled over a month?” Closed “How often do you see a worker in a typical week?” “How far in advance does a worker know what work he will do?” Nov 2006 Mod 12

Fundamentals of Good Interviewing - Closing
Closing should be a natural part of the interview Conclude the interview when the requested information is obtained You are not obligated to fill the scheduled time Provide interviewee with a summary and what will happen next Leave the door open for follow-up time if necessary Nov 2006 Mod 12

Fundamentals of Good Interviewing – Post Interview
Immediately summarize information or conclusions! Check information or documentation referred to during the interview . . . Discuss conclusions with Team as appropriate . . . Conduct self critique of the interview: Determine if your technique created problems . . . Modify your approach as necessary! Nov 2006 Mod 12

Interviewing Short pertinent notes
Necessary, but can be distracting to the interview Pre-written questions are helpful Provide time for yourself immediately after the interview to clarify your notes Nov 2006 Mod 12

Interviewing Listening Process
Active Listening: 70% of an interview should be listening Hearing the message Processing the message Providing feedback to ensure you understand Nov 2006 Mod 12

Interviewing Barriers to Effective Listening Preconceived ideas
Competing demands Excess mental time Emotionally charged words Viewing listening as a passive process Detrimental actions Nov 2006 Mod 12

Interviewing Don’ts Interviewer Talk too much
Communicate desired response Fail to follow-up leads Interrupt Manage time poorly Write too much Form premature conclusions Nov 2006 Mod 12

Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Observation Process Nov 2006

Learning Objectives Describe how to prepare for an observation;
Explain the principle of observation vice participation (Heisenberg Principle); Describe requirements based observation techniques; List several types of ORR/RA evolutions; Explain the need for a facility to run a drill vice only respond to a drill. Nov 2006

Purpose of Observations
To perform, to the maximum extent possible, a representative sample of anticipated operations and evolutions, and responses to credible casualties and “off-normal events” in order to demonstrate the facility operational readiness . . . Single most important step in the process: Serves as a foundation; Team must select representative operations; Surrogate material use . . . Nov 2006 Mod 12

Observation Process Preparation Perform Observation Follow-up
“You can see a lot by watching” - Berra’s Law - Lawrence Peter (Yogi) Berra Preparation Perform Observation Follow-up Reporting Nov 2006 Mod 12

Fundamentals of Good Observations - Preparation
Decide what is needed Is it important to the facility, personnel, procedures, management systems Is it a known or suspected problem area Are there corrective actions to rectify previous problems Nov 2006 Mod 12

Select the Activity Targets of Opportunity Drills/Dry runs System Operability Checks Maintenance Work In-Progress Importance to safety and reliability Problems or potential problem areas Performance of new people Effectiveness of new or revised procedures and/or controls Existing condition of equipment Nov 2006 Mod 12

Types of Observations Exercise Briefings Drills Training session
Operations Maintenance Surveillances Tours System startup Briefings Training session Critique Meetings Turnovers Walk downs Alarm response Nov 2006 Mod 12

Important Inter-relationships
Nov 2006 Mod 12

Fundamentals of Good Observations - Observation
Interaction with observed personnel: Establish rapport; Explain purpose of observation; Outline your role as an observer: Questioning techniques; Will not interfere with work; Be Transparent. Nov 2006 Mod 12

Take thorough notes in real time: Times actions occurred; Procedure details- have procedure in hand; Nomenclature and physical details; Be quantitative, record pertinent data; Include questions and follow-up items; Follow procedure as conducted. Nov 2006 Mod 12

Clearly understand what actually happened: Don't assume . . . Ask clarifying questions without interfering; Seek additional details as needed; Full understanding is imperative to accuracy of the observations; Coordinate with other Team Members who observed. Nov 2006 Mod 12

Start at the true beginning: Observe Pre-job briefings; Observe system alignment checks; Observe daily checks; Note the peripheral activity: People; Environment; Preparation/set-up activities. Become creatively inquisitive . . . Keep attention to important items . . . Nov 2006 Mod 12

Heisenberg Uncertainty Principle
Difficult to make an accurate measurement without affecting what is being measured . . Difficult to measure people performance without influencing their performance . . Do not ask prompting type questions that could change performance . . . Nov 2006 Mod 12

Rules for Observations
Do not act as a Safety Observer! Do not violate rules! Do not interfere with operations! Do not impose conditions, simulations, or anomalies! Be transparent . . . Request Facility to communicate roles and rules! Nov 2006 Mod 12

Fundamentals of Good Observations – Follow-up
Valid conclusions require good follow-up Resolve open questions . . . Primary purpose of follow-up: What you observed are indicators . . . Follow-up will define real problem: need for improvement; underlying causes. Clarify any open questions Pull the Strings Nov 2006 Mod 12

Fundamentals of Good Observations – Reporting
Observations provide the basis for: Findings; Observations; Correlation with Interviews/Document Reviews. Combine several observations if appropriate to draw conclusions concerning program status: Rarely should a single observations result in a finding; Adequacy is the standard, perfection is the goal. Record immediately after observation, Form 1 and 2 Nov 2006 Mod 12

Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Finding Categorization Process Nov 2006

Learning Objectives Distinguish between the various types of findings;
Understand the methodology of finding categorization; Clearly document a finding. Nov 2006

Finding Identification and Categorization
A single issue, finding, or group of related issues, which has been documented on a Form 2, may constitute a finding. Team members in the evaluation of specific Functional Areas should group similar issues to a single finding that concisely characterize the systemic deficiency. The team leader, in consultation with the team member, has the responsibility for making the determination of whether a finding is a Pre-or Post-start Finding. Nov 2006 Mod 12

Pre-start Finding – A finding that must be corrected before an activity can be started. Post-start Finding – A finding that must be corrected, but may be corrected after the start of the activity. Post-start findings are addressed by a corrective action plan, which includes any compensatory measures taken. Observation – An issue of concern identified by the RA team that should be evaluated for possible corrective action in terms of opportunities for improvement. An observation is an issue that does not reach the level of being considered by the team as a finding but warrants evaluation. Nov 2006 Mod 12

Pre-Start / Post-start Categorization
Prior to being published, each finding should be identified as to whether or not it is a Pre-Start or Post-Start. Guidance is in the Order, Standard, Handbook and may also be included in the POA and IP. Lets SEE The Example… EXAMPLE Nov 2006 Mod 12

A non-compliance issue does not necessarily mean a Pre-Start finding! Does the non-compliance represent: significant risk? significant mission impact? significant political impact? significant lack of control? Nov 2006 Mod 12

Unacceptable Risk Higher Consequence Acceptable Risk Probability Post start Nov 2006 Mod 12

Does this finding represent a deficiency of such magnitude as to prevent startup until resolved? Clear failure to implement commitment or assumption of TSR or DSA? OR: Is it related to sustainability? Not a significant safety Issue? Limited impact to Activity within review scope? Nov 2006 Mod 12

Some Obvious Prestart Criteria
Causes a loss of operability that prevents safe shutdown or loss of monitoring ability of safety related systems (i.e. important to safety) . . . Indicates a lack of control that can have near term impact on operability or functionality of safety related systems . . . Requires operator action in short period of time to prevent or mitigate consequences . . . Violates assumption or controls by which DOE/NNSA accepts risk of operations . . . Nov 2006 Mod 12

Results in operation outside of limits of TSRs, OSRs, etc Results in an Unreviewed Safety Question: Increase in the probability or consequence of accident or failure mode described in SAR? Introduction of a new accident or failure mode not described in SAR? Reduction in the margin of safety as described in SAR? Results in failure to comply with Authorization Bases commitments (TSR, LCOs, Surveillances etc.) . . . Nov 2006 Mod 12

Operations/Activities that can lead to: Environmental release of hazardous material in excess of reporting guidelines, permit limits, etc. Exposure of off-site personnel in excess of radiological guidelines for off-site exposure or chemical exposure in excess of minimum reportable amounts. Worker / on-site exposures in excess of guidelines or requirements. Nov 2006 Mod 12

Finding - example Two LCOs ( and ) specified in the activity SAR are not covered by approved surveillance procedures. Nov 2006 Mod 12

Finding - example Emergency exits from this facility do not meet the requirements of NFPA 101 (Life Safety Code). The exits are required to have quick- open "crash bar" hardware. Nov 2006 Mod 12

Finding - example Posting of Radiological areas and entry control programs for these areas per the requirements of 10 CFR 835 were inadequate. No control device was used as required on each entrance to the hot cell 2 area (high radiation area). Nov 2006 Mod 12

Finding - example Although an adequate energy isolation procedure has been developed for this activity, maintenance personnel that service equipment in the area were not familiar with the procedure. No program has been established for root cause determination of maintenance failures, as required by DOE O B Nov 2006 Mod 12

Finding - example No valve identification was provided for the line emergency shutdown system. Although operators questioned were well aware of the shutdown procedure, new personnel may not be aware of these valves. Nov 2006 Mod 12

Finding - example The pipe content and direction flow labeling for this process has been limited to those lines with hazardous materials. Non-hazardous material lines have not yet been labeled. Nov 2006 Mod 12

Don't just identify the finding. . .
Understand the potential impact of the finding! The possible consequences of not resolving it . . . The potential that those consequences will be realized and impact: - personnel; environment; - facility; mission; - public . . . Nov 2006 Mod 12

Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Final Report Format and Administration Nov 2006

Learning Objectives Describe all ORR/RA onsite administration as it relates to a Team Member including: General Time Line Final Report Process Form 1 Form 2 Executive Summary Lessons Learned Recording Dissent Peer Review Nov 2006

Final Report Documentation
Team Leader (TL) will prescribe specifics of format . . . TL should provide examples, as early in the review as possible . . . IP will define content of final report . . . TL will assign report responsibilities . . . Nov 2006 Mod 12

Final Report Preparation Process
Team Member complete assessment forms Team Member validates assessment forms with contractor and/or DOE Team Leader reviews and approves the assessment forms Team Member provides an evaluation of the Contractor Readiness Review and finding closure packages for their functional area Team Member submits any input to the Lessons Learned Nov 2006 138 Mod 12

Final Report Preparation Process …
Team Leader composes executive summary and balance of report content Assessment Forms are included as appendix Team Leader and Team Members review and sign the report cover sheet Team Leader transmits report to the Authorization Authority with a Recommendation Nov 2006 Mod 12

Differing Opinions or Minority Opinions The Readiness Review Report SHALL provide an opportunity for Team Members to include: Differing professional opinions Non-judgmental general comments Observations Dissenting opinions of Team Members should be documented and attached to the report Nov 2006 Mod 12

Final Report Format EXAMPLE TITLE PAGE (Cover) TEAM SIGNATURE PAGE
OFFICIAL PAGE CHANGES AND EDITORIALS EXECUTIVE SUMMARY CONCLUSION INTRODUCTION NNSA ORR TEAM EVALUATION NNSA ORR TEAM INTEGRATED SAFETY MANAGEMENT FINDING CATEGORIZATION AND DOCUMENTATION APPENDIX A LESSONS LEARNED APPENDIX B FUNCTIONAL AREA FORMS 1 APPENDIX C FINDING FORM 2 APPENDIX D ACRONYM TABLE EXAMPLE Nov 2006 Mod 12

The Final Report … Documents the logic of the review
Conveys the results of the review Forms the basis for recommendations for The readiness of the activity to proceed with startup or restart The readiness of the management system (Contractor and DOE) to manage and oversee the activity Nov 2006 Mod 12

Documenting Review Methods Assessment - Form 1
Use Form 1 to document methods and actions taken by team members in evaluation process. Each Form 1: Covers specific objective; Documents team members review methodology; Complete enough to allow a reviewer of form to follow evaluation logic; Follows approach detailed by CRAD (document differences, if any, and reasons for change); States specifically whether criteria met or not; Lists issues (Form 2) associated with that objective. Nov 2006 Mod 12

Form 1 Objective: CRA / Rev Criteria Met Functional Area: Yes Date: No
Objective: (Core Requirement) Criteria (method of appraisal) Review Approach: Records Review - Personnel Interviewed Title (no names) Evolutions / Operations / Shift Performance Discussion of Results: Criteria by criteria to show basis for criteria met or not met Conclusion: Criteria met or not met. Paragraph discussion of overall results and criteria. Issues: Form 1 Inspected by: Approved by: Team Leader Nov 2006 Mod 12

Documentation of Deficiencies Deficiency Form 2
Used to document findings identified during criteria evaluation process . . . Requirement based = clear and specific description of the requirement and the condition which is deficient . . . Primary objective = clear communication . . . Nov 2006 Mod 12

ISSUE: the identified finding or observation
Form 2 Finding: Functional CRA no. / title Prestart: Issue no.: Area: Observation: Post-start: Date: ISSUE: the identified finding or observation REQUIREMENT: requirement statement(s) from reference REFERENCES: specific as possible, including section and paragraph DISCUSSION: Inspected by: Approved by: Team Leader Nov 2006 Mod 12

Finding Documentation
When documenting a finding the finding must include the following: The Criteria being assessed, or measured; The Review Approach used to derive a conclusion on the assessment of the criteria (document review, observation, or performance evaluation); A discussion regarding the Review Approach used, and; Identification of the Finding, with proper citation of the requirement. Nov 2006

Finding documentation begins with: A brief finding statement, and A citation of the requirement, Followed by the discussion paragraph (which discusses the finding by emphasizing the review approach used, as-found conditions, and relationship towards the requirement.) Nov 2006

Finding Documentation Example
Issue Statement: The Lightning Protection System (LPS), which is defined as a Safety Class system within the Documented Safety Analysis, does not meet the design specifications referenced by DOE O 420.1, Facility Safety, para Design Requirements. Nov 2006

Discussion: DOE O 420.1, , states that all Facility safety class electrical systems shall be designed to the basic approach outlined in Section (Electrical) of DOE G , Nonreactor Nuclear Safety Design Criteria and Explosives Safety Criteria Guide for Use with DOE O 420.1, Facility Safety, dated Referenced under Section is Table 5.5, which list hardware design requirements for both Safety Class and Safety Significant SSCs. As referenced, NFPA 780, Standard for the Installation of Lightning Protection Systems; requires that systems be bonded. A walk down of the LPS for the roof area identified several deficiencies, such as unbounded metallic penetrations and electrical conduit. Nov 2006

Other Sections of the Final Report
Summary: Brief overview of functional area results; Less than one page. Executive Summary: Summaries each of the findings, Provides Functional Area Conclusion Provides Team’s Overall Conclusion Lessons Learned, “Required by Order”: TL+TMs review lessons learned on EH web site; All asked to contribute, one TM coordinates for TL. Nov 2006 Mod 12

Review and Approval of Your Forms
Completed by Team Member; Approved by Team Leader; Senior Advisor/s involved and assist; Often an iterative process; Team Leader and Senior Advisor sign entire report; Team Member signs for Functional Area only; Dissenting opinion option, if no resolution. Nov 2006 Mod 12

Factual Accuracy Address Facts - Not Judgments . . .
Quick turn-around at the working level will speed the process . . . Limit distribution of all draft products and prior to final approval/submission. Nov 2006 Mod 12

If the Outcome is not Successful
Two Options: Suspend the Review. Interim Report including what was done and areas not done. Final report for completed Functional Areas (FA); High level discussion and findings for incomplete FAs. Terminate the Review. Report results with basis for repeat of entire ORR/RA. Clear discussion of what was and was not reviewed. What was and was not adequate! Nov 2006 Mod 12

Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Timelines, More Roles/Responsibilities, and Meetings Nov 2006

Learning Objectives Understand Readiness Review Timelines
Identify Team Leader and Member Responsibilities Plan and Organize Opening, Daily, and Closeout Meetings Nov 2006

Development of Implementation Plan,
ORR Time Line POA Approval Initial Pre- review visit Site Introduction meeting Conduct Review Close-out meeting Follow-up Review Team meeting ORR Report Development of Implementation Plan, Generation & CRADs, & Checklists Review Nov 2006 Mod 12

Team Leader Responsibilities During the Review
Ensure all team members have completed all access requirements prior to the review; Ensure all team members have completed all required reading prior to the review; Conducts the following meetings: Opening (kickoff); Conducts the Daily (team and site personnel); Conducts Daily Site Manager Briefings (as needed); Conducts Final Team; Conducts Final Out Brief. Nov 2006 Mod 12

Developing and Completing Form 1s and 2s: Provides draft forms to the Team Leader as soon as possible; Updates forms on a daily basis; Organizing and coordinating factual accuracy reviews; Completed Form 1s and 2s means that they have been: Completed, signed, and submitted to the Team Leader; Factual accuracy of the individual forms has been confirmed with the counterparts; Accepted by the Team Leader before the closeout meeting and before the team member leaves the site; Summary evaluations completed, submitted to, and signed-off by Team Leader prior to the closeout meeting. Nov 2006 Mod 12

Maintains accurate and complete notes of activities performed; Updates schedule of activities on a daily basis; Provides support in developing a final conclusion for following subject areas: Intergraded Safety Management; Lessons Learned; Adequacy of contractors review; and As assigned by Team Leader. Supports and attends ALL meeting as requested by Team Leader; Nov 2006 Mod 12

Work with other ORR/RA members: to resolve and communicate issues; to prepare the review documentation and report; to eliminate redundant reviews. Submit completed review documents for review and approval. Concur with determination of operational readiness and conclusions of ORR/RA report (in their area of assessment). Nov 2006 Mod 12

An Effective ORR/RA Requires Effective Coordination
Avoid duplication of: Interviews, procedure reviews; Drills, exercise observations . . . Know time line of the review Schedule driven; Defined sequences; Stay on point, avoid distractions; Preparation of report forms critical and critical path . . . Nov 2006 Mod 12

Meetings Opening (Kickoff) Meeting;
Daily Team and Site Personnel Meeting; Daily Site Management Briefing, if needed; Final Team Meeting; Final Out Brief Meeting. Nov 2006 Mod 12

Opening (Kickoff) Meeting
Team Leaders Meeting Make introductions. Allow for a series of briefs by the site explaining the site's management and organization, its functions, operations and interfaces. Brief the audience on the ORR process and the ORR schedule. Brief site audience and team on the rules for interaction between team members and site personnel. Nov 2006 Mod 12

Rules for Interaction Between Team Members and Site Personnel: The team will strive to be invisible so as to not influence the performance of the facility; Request that all operators be briefed on these rules and in particular that no action should be taken in response to questions or discussion by the team; The team will initiate action, preferable through a supervisor or escort, if an immediate unsafe condition is encountered. Nov 2006 Mod 12

Rules for Interaction Between Team Members and Site Personnel: The team will not act as safety monitors or direct operations; The team will refrain from distracting the operators; The team will not violate site rules, cross boundaries without authority, violate control room/area boundaries or in any other way become an initiating event during the ORR; The team will not act as evolution or drill observers; Nov 2006 Mod 12

Daily Team and Site Personnel Meeting Meeting
Performed at the end of each day Team Meeting First – Site Personnel Meeting Second; All Team Members Present; Site Personnel Meeting includes Management and all POCs; Remind site personnel that they are observers not participators. Nov 2006 Mod 12

Daily Team and Site Personnel Meeting
Identification of any past, new, or unclear issues; Re-emphasize who is responsible for what area; Discuss and restate scope of review; Discuss process for going beyond the scope of the review; Discuss process for making contact outside of their assigned areas (e.g., interviews with senior management); Discuss validation of facts; Keep a running record of all the issues; Nov 2006 Mod 12

Daily Site Management Briefing
Held only if requested by management (if management cannot attend daily meetings) Presents current and potential findings; Presents status of the review and support; Discusses any changes if needed. Nov 2006 Mod 12

Final Team Meeting Last opportunity to work and clarify any questions with Team Members; Due prior to this meeting are all: Final Forms 1 and 2 Draft summary evaluations (ISM, Lessons Learned, Contractor Eval, Summary Conclusion) Last opportunity to discover/address dissenting opinions and conflicts Any such issues must be addressed at this time to avoid any confusion at the closeout meeting. Meeting for TEAM Members ONLY, since no new information will be discussed or presented. Nov 2006 Mod 12

Team Leader informs the site management of the major issues identified during the ORR; The closeout meeting kept short; Invite Site Management and POCs; No surprises at this meeting – no new issues or findings; As a goal, provide a draft copy of the final report to the senior managers; At a minimum, prepare the executive summary of the draft report as a handout to everyone attending the briefing; Nov 2006 Mod 12

Team Leader informs the site management of the major issues identified during the ORR; Discuss the final report schedule and deliverables; Teams consensus will be presented: Their final conclusion, Any major issues, and Overall themes, if any will be presented. Exit briefing based on the Forms 1, Forms 2, and Team’s conclusions; Nov 2006 Mod 12

Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Document Evaluation Process Nov 2006

Learning Objectives Describe the preparation for onsite document review; Discuss the ORR/RA principle of sampling versus auditing; Discuss the complete trail sampling technique; Discuss good Team Member record / note keeping practices related to records review; Nov 2006

Document Evaluation Process
Preparation Execution Follow-up Reporting Nov 2006 Mod 12

Request documents before the Review Request those which address or describe the following Regulatory Drivers (DOE Orders, Federal Law) Safety Basis Drivers (DSA, TSR, etc.) Facility Procedures Process Procedures Nov 2006 Mod 12

Example of Documents to Request- Hierarchy
Requirements Objective Core Orders, Codes, Requirements Standards, Regulations Criteria Identification of Required Program & Processes Identification of important implementing Procedures / Policies Identification of important Systems, Structures, Components Identification of important Job Factors Nov 2006 Mod 12

Performance Review Flow
Top Tier Requirement Performance Review Flow DOE Site Office Requirements Requirement Specific Contract (what) Requirements Company Policy / Procedure Manual Project Procedures Implementation Specific (how) Line Instructions Result Project Records Nov 2006 Mod 12

Request pertinent documents; Sampling - Not a complete document audit; Communicate with Contractor counterpart to determine appropriate documents; Communicate with the Team; Avoid duplication of review, if possible Ensure that Team Members are aware of documents that may affect their review Prioritize the selected target documents for review. Nov 2006 Mod 12

Read and understand the documents Determine if document contains required information to address applicable Criteria, Review and Approach Document; Determine if regulatory requirements are met or described; Determine how document is controlled and approved; Determine if procedures have been written correctly. Nov 2006 Mod 12