Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Timelines, More Roles/Responsibilities, and Meetings Document Evaluation Process Interviewing Techniques Observation Process Finding Categorization Process Final Report Format and Administration Nov 2006
Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Timelines, More Roles/Responsibilities, and Meetings Document Evaluation Process Interviewing Techniques Observation Process Finding Categorization Process Final Report Format and Administration Nov 2006
Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Timelines, More Roles/Responsibilities, and Meetings Nov 2006
Readiness Review Process Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Initial Team Mtg Pre-Site Visit, IP Drafted) 6 Months Prior to RR 1 Year Prior to RR Generally 2 weeks Nov 2006 Mod 12
Learning Objectives Understand Readiness Review Timelines Identify Team Leader and Member Responsibilities Plan and Organize Opening, Daily, and Closeout Meetings Nov 2006
Development of Implementation Plan, ORR Time Line POA Approval Initial Pre- review visit Site Introduction meeting Conduct Review Close-out meeting Follow-up Review Team meeting 1 2 3 4 5 6 7 8 9 10 11 12 13 ORR Report Development of Implementation Plan, Generation & CRADs, & Checklists Review Nov 2006 Mod 12
Typical RR Timeline SUN MON TUE WED THUR FRI SAT ARRIVE Opening Records Operations Operations Interviews Report Meeting Review Writing Drills Drills Records Mangmt.. Review Interviews Mangmt.. Interviews SUN MON TUE WED THUR FRI SAT Off Finish Finish Closeout Forms Forms Continue Thursday and Friday, as needed; Report is signed and Submitted . . . Catch Up Peer Rev. Peer Rev. Report Report Sections Sections Brief Prep Brief Prep Nov 2006 Mod 12
Team Leader Responsibilities During the Review Ensure all team members have completed all access requirements prior to the review; Ensure all team members have completed all required reading prior to the review; Conducts the following meetings: Opening (kickoff); Conducts the Daily (team and site personnel); Conducts Daily Site Manager Briefings (as needed); Conducts Final Team; Conducts Final Out Brief. Nov 2006 Mod 12
Team Members Responsibilities During the Review Developing and Completing Form 1s and 2s: Provides draft forms to the Team Leader as soon as possible; Updates forms on a daily basis; Organizing and coordinating factual accuracy reviews; Completed Form 1s and 2s means that they have been: Completed, signed, and submitted to the Team Leader; Factual accuracy of the individual forms has been confirmed with the counterparts; Accepted by the Team Leader before the closeout meeting and before the team member leaves the site; Summary evaluations completed, submitted to, and signed-off by Team Leader prior to the closeout meeting. Nov 2006 Mod 12
Team Members Responsibilities During the Review Maintains accurate and complete notes of activities performed; Updates schedule of activities on a daily basis; Provides support in developing a final conclusion for following subject areas: Intergraded Safety Management; Lessons Learned; Adequacy of contractors review; and As assigned by Team Leader. Supports and attends ALL meeting as requested by Team Leader; Nov 2006 Mod 12
Team Members Responsibilities During the Review Work with other ORR/RA members: to resolve and communicate issues; to prepare the review documentation and report; to eliminate redundant reviews. Submit completed review documents for review and approval. Concur with determination of operational readiness and conclusions of ORR/RA report (in their area of assessment). Nov 2006 Mod 12
An Effective ORR/RA Requires Effective Coordination Avoid duplication of: Interviews, procedure reviews; Drills, exercise observations . . . Know time line of the review Schedule driven; Defined sequences; Stay on point, avoid distractions; Preparation of report forms critical and critical path . . . Nov 2006 Mod 12
Meetings Opening (Kickoff) Meeting; Daily Team and Site Personnel Meeting; Daily Site Management Briefing, if needed; Final Team Meeting; Final Out Brief Meeting. Nov 2006 Mod 12
Opening (Kickoff) Meeting Team Leaders Meeting Make introductions. Allow for a series of briefs by the site explaining the site's management and organization, its functions, operations and interfaces. Brief the audience on the ORR process and the ORR schedule. Brief site audience and team on the rules for interaction between team members and site personnel. Nov 2006 Mod 12
Opening (Kickoff) Meeting (Cont)… Rules for Interaction Between Team Members and Site Personnel: The team will strive to be invisible so as to not influence the performance of the facility; Request that all operators be briefed on these rules and in particular that no action should be taken in response to questions or discussion by the team; The team will initiate action, preferable through a supervisor or escort, if an immediate unsafe condition is encountered. Nov 2006 Mod 12
Opening (Kickoff) Meeting (Cont)… Rules for Interaction Between Team Members and Site Personnel: The team will not act as safety monitors or direct operations; The team will refrain from distracting the operators; The team will not violate site rules, cross boundaries without authority, violate control room/area boundaries or in any other way become an initiating event during the ORR; The team will not act as evolution or drill observers; Nov 2006 Mod 12
Daily Team and Site Personnel Meeting Meeting Performed at the end of each day Team Meeting First – Site Personnel Meeting Second; All Team Members Present; Site Personnel Meeting includes Management and all POCs; Remind site personnel that they are observers not participators. Nov 2006 Mod 12
Daily Team and Site Personnel Meeting Identification of any past, new, or unclear issues; Re-emphasize who is responsible for what area; Discuss and restate scope of review; Discuss process for going beyond the scope of the review; Discuss process for making contact outside of their assigned areas (e.g., interviews with senior management); Discuss validation of facts; Keep a running record of all the issues; Nov 2006 Mod 12
Daily Site Management Briefing Held only if requested by management (if management cannot attend daily meetings) Presents current and potential findings; Presents status of the review and support; Discusses any changes if needed. Nov 2006 Mod 12
Final Team Meeting Last opportunity to work and clarify any questions with Team Members; Due prior to this meeting are all: Final Forms 1 and 2 Draft summary evaluations (ISM, Lessons Learned, Contractor Eval, Summary Conclusion) Last opportunity to discover/address dissenting opinions and conflicts Any such issues must be addressed at this time to avoid any confusion at the closeout meeting. Meeting for TEAM Members ONLY, since no new information will be discussed or presented. Nov 2006 Mod 12
Final Out Brief Meeting Team Leader informs the site management of the major issues identified during the ORR; The closeout meeting kept short; Invite Site Management and POCs; No surprises at this meeting – no new issues or findings; As a goal, provide a draft copy of the final report to the senior managers; At a minimum, prepare the executive summary of the draft report as a handout to everyone attending the briefing; Nov 2006 Mod 12
Final Out Brief Meeting Team Leader informs the site management of the major issues identified during the ORR; Discuss the final report schedule and deliverables; Teams consensus will be presented: Their final conclusion, Any major issues, and Overall themes, if any will be presented. Exit briefing based on the Forms 1, Forms 2, and Team’s conclusions; Nov 2006 Mod 12
Questions Nov 2006 23
Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Document Evaluation Process Nov 2006
Readiness Review Process Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Initial Team Mtg Pre-Site Visit, IP Drafted) 6 Months Prior to RR 1 Year Prior to RR Generally 2 weeks Nov 2006 Mod 12
Learning Objectives Describe the preparation for onsite document review; Discuss the ORR/RA principle of sampling versus auditing; Discuss the complete trail sampling technique; Discuss good Team Member record / note keeping practices related to records review; Nov 2006
Document Evaluation Process Preparation Execution Follow-up Reporting Nov 2006 Mod 12
Document Evaluation Process Preparation Request documents before the Review Request those which address or describe the following Regulatory Drivers (DOE Orders, Federal Law) Safety Basis Drivers (DSA, TSR, etc.) Facility Procedures Process Procedures Nov 2006 Mod 12
Example of Documents to Request- Hierarchy Requirements Objective Core Orders, Codes, Requirements Standards, Regulations Criteria Identification of Required Program & Processes Identification of important implementing Procedures / Policies Identification of important Systems, Structures, Components Identification of important Job Factors Nov 2006 Mod 12
Performance Review Flow Top Tier Requirement Performance Review Flow DOE Site Office Requirements Requirement Specific Contract (what) Requirements Company Policy / Procedure Manual Project Procedures Implementation Specific (how) Line Instructions Result Project Records Nov 2006 Mod 12
Document Evaluation Process Preparation Request pertinent documents; Sampling - Not a complete document audit; Communicate with Contractor counterpart to determine appropriate documents; Communicate with the Team; Avoid duplication of review, if possible Ensure that Team Members are aware of documents that may affect their review Prioritize the selected target documents for review. Nov 2006 Mod 12
Document Evaluation Process Execution Read and understand the documents Determine if document contains required information to address applicable Criteria, Review and Approach Document; Determine if regulatory requirements are met or described; Determine how document is controlled and approved; Determine if procedures have been written correctly. Nov 2006 Mod 12
Document Evaluation Process Execution Do procedures accurately document and implement the following: Regulatory Drivers . . . Programs and Processes . . . Commitments . . . DSA/SAR / TSR / Accident assumptions; BIO / JCO / FHA, and EIS; List A/B, WSS, S/RIDs; Nov 2006 Mod 12
Document Evaluation Process Execution Do procedures accurately define and communicate Roles and Responsibilities for individuals as they relate to specific requirements.. Are they understood by those identified individuals . . . Do they comply with “writing guides”.. Nov 2006 Mod 12
Document Evaluation Process Execution Document Reviews do not have to be line by line copies of Order or regulatory driver, but must: Detail the mechanisms to fulfill the intent and requirements of the regulatory/contract driver . . . Detail the mechanisms and requirements to adequately control and document the activity . . . Focus on implementation . . . Nov 2006 Mod 12
Document Evaluation Process Follow-up Follow-up is needed if you are “Pulling the String”; Request additional documents if needed; Check beyond the document evaluation Training Controls to ensure procedures are followed Document configuration control Nov 2006 Mod 12
Document Evaluation Process Reporting Record the information relating to the acceptance and review criteria Check reference information Integrate information gained into observation and interview processes Discuss conclusions with Team Leader and Team Members as applicable Take Accurate Notes Nov 2006 Mod 12
Notes and Records Your personal permanent record of the review . . . Thorough enough to reconstruct: Names; Titles; Dates, Times; Specific Documents; Equipment identification; Events; Quotations or paraphrasing; Transfer to Forms daily – “Write Every Night”! Nov 2006 Mod 12
Summary… Does the documentation accurately and completely implement the requirements: Contractual? Authorization Basis? Programmatic? Performance? Is the documentation being implemented and requirements met? Is “Flow-Down” consistent, complete, and accurate? Nov 2006 Mod 12
Questions Nov 2006 23
Conduct of the Review – Mod 12 Interviewing Techniques Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Interviewing Techniques Nov 2006
Learning Objectives Describe the two major types of interviews; Describe how to prepare for an interview; Compare the strengths and weaknesses of various questioning techniques; Discuss the timing / scheduling of interviews during an ORR/RA. Nov 2006
Two Types of Interviews Management Learn how they do business Address issues discovered during review of their processes and procedure documentation Level of Knowledge Systems Safety Basis Processes Precautions/Hazards Principles Nov 2006 Mod 12
Interviewing Fundamentals of Good Interviewing Planning Opening Questioning Closing Post Interview Nov 2006 Mod 12
Fundamentals of Good Interviewing - Planning Plan questions in advance Define the purpose of the interview Keep the interview on track Focus on one issue at a time Cover important subject matters Gather data that will provide knowledge Review previous findings related to the topic Know the interviewee’s functions and responsibilities Control the interview setting (if possible) Nov 2006 Mod 12
Fundamentals of Good Interviewing - Opening Set the tone State the purpose of the interview Start with general topic and move to more detail as the interview progresses Obtain the title of the interviewee Establish and keep control of the interview Put the interviewee at ease State you will be taking notes and why Nov 2006 Mod 12
Fundamentals of Good Interviewing - Questioning Ask clear and concise questions Avoid leading questions Allow interviewee time to think through the response Keep interview on track Restate questions Ask follow-up questions Ask for clarification or basis of opinion Get complete answer to one question before progressing to the next Ask for illustrations or examples of documents alluded to by the interviewee Limit participation to two team members per interviewee ABOVE ALL - LISTEN CAREFULLY Nov 2006 Mod 12
Example Interview Questions Leading “How often do you discuss policy with a worker?” Neutral “How do the workers gain knowledge of company policy?” Primary “What tools are required by the Preventative Maintenance Procedure?” Secondary “What tools are actually used?” “Why are all the other tools taken to the work site?” Nov 2006 Mod 12
Example Interview Questions Open “Can you describe your relationship with your subordinates?” “Could you describe how preventative maintenance is scheduled over a month?” Closed “How often do you see a worker in a typical week?” “How far in advance does a worker know what work he will do?” Nov 2006 Mod 12
Fundamentals of Good Interviewing - Closing Closing should be a natural part of the interview Conclude the interview when the requested information is obtained You are not obligated to fill the scheduled time Provide interviewee with a summary and what will happen next Leave the door open for follow-up time if necessary Nov 2006 Mod 12
Fundamentals of Good Interviewing – Post Interview Immediately summarize information or conclusions! Check information or documentation referred to during the interview . . . Discuss conclusions with Team as appropriate . . . Conduct self critique of the interview: Determine if your technique created problems . . . Modify your approach as necessary! Nov 2006 Mod 12
Interviewing Short pertinent notes Necessary, but can be distracting to the interview Pre-written questions are helpful Provide time for yourself immediately after the interview to clarify your notes Nov 2006 Mod 12
Interviewing Listening Process Active Listening: 70% of an interview should be listening Hearing the message Processing the message Providing feedback to ensure you understand Nov 2006 Mod 12
Interviewing Barriers to Effective Listening Preconceived ideas Competing demands Excess mental time Emotionally charged words Viewing listening as a passive process Detrimental actions Nov 2006 Mod 12
Interviewing Don’ts Interviewer Talk too much Communicate desired response Fail to follow-up leads Interrupt Manage time poorly Write too much Form premature conclusions Nov 2006 Mod 12
Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Observation Process Nov 2006
Learning Objectives Describe how to prepare for an observation; Explain the principle of observation vice participation (Heisenberg Principle); Describe requirements based observation techniques; List several types of ORR/RA evolutions; Explain the need for a facility to run a drill vice only respond to a drill. Nov 2006
Purpose of Observations To perform, to the maximum extent possible, a representative sample of anticipated operations and evolutions, and responses to credible casualties and “off-normal events” in order to demonstrate the facility operational readiness . . . Single most important step in the process: Serves as a foundation; Team must select representative operations; Surrogate material use . . . Nov 2006 Mod 12
Observation Process Preparation Perform Observation Follow-up “You can see a lot by watching” - Berra’s Law - Lawrence Peter (Yogi) Berra Preparation Perform Observation Follow-up Reporting Nov 2006 Mod 12
Fundamentals of Good Observations - Preparation Decide what is needed Is it important to the facility, personnel, procedures, management systems Is it a known or suspected problem area Are there corrective actions to rectify previous problems Nov 2006 Mod 12
Fundamentals of Good Observations - Preparation Select the Activity Targets of Opportunity Drills/Dry runs System Operability Checks Maintenance Work In-Progress Importance to safety and reliability Problems or potential problem areas Performance of new people Effectiveness of new or revised procedures and/or controls Existing condition of equipment Nov 2006 Mod 12
Types of Observations Exercise Briefings Drills Training session Operations Maintenance Surveillances Tours System startup Briefings Training session Critique Meetings Turnovers Walk downs Alarm response Nov 2006 Mod 12
Important Inter-relationships Nov 2006 Mod 12
Fundamentals of Good Observations - Observation Interaction with observed personnel: Establish rapport; Explain purpose of observation; Outline your role as an observer: Questioning techniques; Will not interfere with work; Be Transparent. Nov 2006 Mod 12
Fundamentals of Good Observations - Observation Take thorough notes in real time: Times actions occurred; Procedure details- have procedure in hand; Nomenclature and physical details; Be quantitative, record pertinent data; Include questions and follow-up items; Follow procedure as conducted. Nov 2006 Mod 12
Fundamentals of Good Observations - Observation Clearly understand what actually happened: Don't assume . . . Ask clarifying questions without interfering; Seek additional details as needed; Full understanding is imperative to accuracy of the observations; Coordinate with other Team Members who observed. Nov 2006 Mod 12
Fundamentals of Good Observations - Observation Start at the true beginning: Observe Pre-job briefings; Observe system alignment checks; Observe daily checks; Note the peripheral activity: People; Environment; Preparation/set-up activities. Become creatively inquisitive . . . Keep attention to important items . . . Nov 2006 Mod 12
Heisenberg Uncertainty Principle Difficult to make an accurate measurement without affecting what is being measured . . Difficult to measure people performance without influencing their performance . . Do not ask prompting type questions that could change performance . . . Nov 2006 Mod 12
Rules for Observations Do not act as a Safety Observer! Do not violate rules! Do not interfere with operations! Do not impose conditions, simulations, or anomalies! Be transparent . . . Request Facility to communicate roles and rules! Nov 2006 Mod 12
Fundamentals of Good Observations – Follow-up Valid conclusions require good follow-up Resolve open questions . . . Primary purpose of follow-up: What you observed are indicators . . . Follow-up will define real problem: need for improvement; underlying causes. Clarify any open questions Pull the Strings Nov 2006 Mod 12
Fundamentals of Good Observations – Reporting Observations provide the basis for: Findings; Observations; Correlation with Interviews/Document Reviews. Combine several observations if appropriate to draw conclusions concerning program status: Rarely should a single observations result in a finding; Adequacy is the standard, perfection is the goal. Record immediately after observation, Form 1 and 2 Nov 2006 Mod 12
Questions Nov 2006 23
Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Finding Categorization Process Nov 2006
Learning Objectives Distinguish between the various types of findings; Understand the methodology of finding categorization; Clearly document a finding. Nov 2006
Finding Identification and Categorization A single issue, finding, or group of related issues, which has been documented on a Form 2, may constitute a finding. Team members in the evaluation of specific Functional Areas should group similar issues to a single finding that concisely characterize the systemic deficiency. The team leader, in consultation with the team member, has the responsibility for making the determination of whether a finding is a Pre-or Post-start Finding. Nov 2006 Mod 12
Finding Identification and Categorization Pre-start Finding – A finding that must be corrected before an activity can be started. Post-start Finding – A finding that must be corrected, but may be corrected after the start of the activity. Post-start findings are addressed by a corrective action plan, which includes any compensatory measures taken. Observation – An issue of concern identified by the RA team that should be evaluated for possible corrective action in terms of opportunities for improvement. An observation is an issue that does not reach the level of being considered by the team as a finding but warrants evaluation. Nov 2006 Mod 12
Pre-Start / Post-start Categorization Prior to being published, each finding should be identified as to whether or not it is a Pre-Start or Post-Start. Guidance is in the Order, Standard, Handbook and may also be included in the POA and IP. Lets SEE The Example… EXAMPLE Nov 2006 Mod 12
Pre-Start / Post-start Categorization A non-compliance issue does not necessarily mean a Pre-Start finding! Does the non-compliance represent: significant risk? significant mission impact? significant political impact? significant lack of control? Nov 2006 Mod 12
Pre-Start / Post-start Categorization Unacceptable Risk Higher Consequence Acceptable Risk Probability Post start Nov 2006 Mod 12
Pre-Start / Post-start Categorization Does this finding represent a deficiency of such magnitude as to prevent startup until resolved? Clear failure to implement commitment or assumption of TSR or DSA? OR: Is it related to sustainability? Not a significant safety Issue? Limited impact to Activity within review scope? Nov 2006 Mod 12
Some Obvious Prestart Criteria Causes a loss of operability that prevents safe shutdown or loss of monitoring ability of safety related systems (i.e. important to safety) . . . Indicates a lack of control that can have near term impact on operability or functionality of safety related systems . . . Requires operator action in short period of time to prevent or mitigate consequences . . . Violates assumption or controls by which DOE/NNSA accepts risk of operations . . . Nov 2006 Mod 12
Some Obvious Prestart Criteria Results in operation outside of limits of TSRs, OSRs, etc. . . . Results in an Unreviewed Safety Question: Increase in the probability or consequence of accident or failure mode described in SAR? Introduction of a new accident or failure mode not described in SAR? Reduction in the margin of safety as described in SAR? Results in failure to comply with Authorization Bases commitments (TSR, LCOs, Surveillances etc.) . . . Nov 2006 Mod 12
Some Obvious Prestart Criteria Operations/Activities that can lead to: Environmental release of hazardous material in excess of reporting guidelines, permit limits, etc. Exposure of off-site personnel in excess of radiological guidelines for off-site exposure or chemical exposure in excess of minimum reportable amounts. Worker / on-site exposures in excess of guidelines or requirements. Nov 2006 Mod 12
Finding - example Two LCOs (10.1.1 and 10.2.2) specified in the activity SAR are not covered by approved surveillance procedures. Nov 2006 Mod 12
Finding - example Emergency exits from this facility do not meet the requirements of NFPA 101 (Life Safety Code). The exits are required to have quick- open "crash bar" hardware. Nov 2006 Mod 12
Finding - example Posting of Radiological areas and entry control programs for these areas per the requirements of 10 CFR 835 were inadequate. No control device was used as required on each entrance to the hot cell 2 area (high radiation area). Nov 2006 Mod 12
Finding - example Although an adequate energy isolation procedure has been developed for this activity, maintenance personnel that service equipment in the area were not familiar with the procedure. No program has been established for root cause determination of maintenance failures, as required by DOE O 4330.4B Nov 2006 Mod 12
Finding - example No valve identification was provided for the line emergency shutdown system. Although operators questioned were well aware of the shutdown procedure, new personnel may not be aware of these valves. Nov 2006 Mod 12
Finding - example The pipe content and direction flow labeling for this process has been limited to those lines with hazardous materials. Non-hazardous material lines have not yet been labeled. Nov 2006 Mod 12
Don't just identify the finding. . . Understand the potential impact of the finding! The possible consequences of not resolving it . . . The potential that those consequences will be realized and impact: - personnel; - environment; - facility; - mission; - public . . . Nov 2006 Mod 12
Questions Nov 2006 23
Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Final Report Format and Administration Nov 2006
Learning Objectives Describe all ORR/RA onsite administration as it relates to a Team Member including: General Time Line Final Report Process Form 1 Form 2 Executive Summary Lessons Learned Recording Dissent Peer Review Nov 2006
Typical RR Timeline SUN MON TUE WED THUR FRI SAT ARRIVE Opening Records Operations Operations Interviews Report Meeting Review Writing Drills Drills Records Mangmt.. Review Interviews Mangmt.. Interviews SUN MON TUE WED THUR FRI SAT Off Finish Finish Closeout Forms Forms Continue Thursday and Friday, as needed; Report is signed and Submitted . . . Catch Up Peer Rev. Peer Rev. Report Report Sections Sections Brief Prep Brief Prep Nov 2006 Mod 12
Final Report Documentation Team Leader (TL) will prescribe specifics of format . . . TL should provide examples, as early in the review as possible . . . IP will define content of final report . . . TL will assign report responsibilities . . . Nov 2006 Mod 12
Final Report Preparation Process Team Member complete assessment forms Team Member validates assessment forms with contractor and/or DOE Team Leader reviews and approves the assessment forms Team Member provides an evaluation of the Contractor Readiness Review and finding closure packages for their functional area Team Member submits any input to the Lessons Learned Nov 2006 138 Mod 12
Final Report Preparation Process … Team Leader composes executive summary and balance of report content Assessment Forms are included as appendix Team Leader and Team Members review and sign the report cover sheet Team Leader transmits report to the Authorization Authority with a Recommendation Nov 2006 Mod 12
Final Report Preparation Process … Differing Opinions or Minority Opinions The Readiness Review Report SHALL provide an opportunity for Team Members to include: Differing professional opinions Non-judgmental general comments Observations Dissenting opinions of Team Members should be documented and attached to the report Nov 2006 Mod 12
Final Report Format EXAMPLE TITLE PAGE (Cover) TEAM SIGNATURE PAGE OFFICIAL PAGE CHANGES AND EDITORIALS EXECUTIVE SUMMARY CONCLUSION INTRODUCTION NNSA ORR TEAM EVALUATION NNSA ORR TEAM INTEGRATED SAFETY MANAGEMENT FINDING CATEGORIZATION AND DOCUMENTATION APPENDIX A LESSONS LEARNED APPENDIX B FUNCTIONAL AREA FORMS 1 APPENDIX C FINDING FORM 2 APPENDIX D ACRONYM TABLE EXAMPLE Nov 2006 Mod 12
The Final Report … Documents the logic of the review Conveys the results of the review Forms the basis for recommendations for The readiness of the activity to proceed with startup or restart The readiness of the management system (Contractor and DOE) to manage and oversee the activity Nov 2006 Mod 12
Documenting Review Methods Assessment - Form 1 Use Form 1 to document methods and actions taken by team members in evaluation process. Each Form 1: Covers specific objective; Documents team members review methodology; Complete enough to allow a reviewer of form to follow evaluation logic; Follows approach detailed by CRAD (document differences, if any, and reasons for change); States specifically whether criteria met or not; Lists issues (Form 2) associated with that objective. Nov 2006 Mod 12
Form 1 Objective: CRA / Rev Criteria Met Functional Area: Yes Date: No Objective: (Core Requirement) Criteria (method of appraisal) Review Approach: Records Review - Personnel Interviewed Title (no names) Evolutions / Operations / Shift Performance Discussion of Results: Criteria by criteria to show basis for criteria met or not met Conclusion: Criteria met or not met. Paragraph discussion of overall results and criteria. Issues: Form 1 Inspected by: Approved by: Team Leader Nov 2006 Mod 12
Documentation of Deficiencies Deficiency Form 2 Used to document findings identified during criteria evaluation process . . . Requirement based = clear and specific description of the requirement and the condition which is deficient . . . Primary objective = clear communication . . . Nov 2006 Mod 12
ISSUE: the identified finding or observation Form 2 Finding: Functional CRA no. / title Prestart: Issue no.: Area: Observation: Post-start: Date: ISSUE: the identified finding or observation REQUIREMENT: requirement statement(s) from reference REFERENCES: specific as possible, including section and paragraph DISCUSSION: Inspected by: Approved by: Team Leader Nov 2006 Mod 12
Finding Documentation When documenting a finding the finding must include the following: The Criteria being assessed, or measured; The Review Approach used to derive a conclusion on the assessment of the criteria (document review, observation, or performance evaluation); A discussion regarding the Review Approach used, and; Identification of the Finding, with proper citation of the requirement. Nov 2006
Finding Documentation Finding documentation begins with: A brief finding statement, and A citation of the requirement, Followed by the discussion paragraph (which discusses the finding by emphasizing the review approach used, as-found conditions, and relationship towards the requirement.) Nov 2006
Finding Documentation Example Issue Statement: The Lightning Protection System (LPS), which is defined as a Safety Class system within the Documented Safety Analysis, does not meet the design specifications referenced by DOE O 420.1, Facility Safety, para. 4.1.1.2 Design Requirements. Nov 2006
Finding Documentation Example Discussion: DOE O 420.1, 4.1.1.2, states that all Facility safety class electrical systems shall be designed to the basic approach outlined in Section 5.2.3 (Electrical) of DOE G 420.1-1, Nonreactor Nuclear Safety Design Criteria and Explosives Safety Criteria Guide for Use with DOE O 420.1, Facility Safety, dated 3-28-00. Referenced under Section 5.2.3 is Table 5.5, which list hardware design requirements for both Safety Class and Safety Significant SSCs. As referenced, NFPA 780, Standard for the Installation of Lightning Protection Systems; requires that systems be bonded. A walk down of the LPS for the roof area identified several deficiencies, such as unbounded metallic penetrations and electrical conduit. Nov 2006
Other Sections of the Final Report Summary: Brief overview of functional area results; Less than one page. Executive Summary: Summaries each of the findings, Provides Functional Area Conclusion Provides Team’s Overall Conclusion Lessons Learned, “Required by Order”: TL+TMs review lessons learned on EH web site; All asked to contribute, one TM coordinates for TL. Nov 2006 Mod 12
Review and Approval of Your Forms Completed by Team Member; Approved by Team Leader; Senior Advisor/s involved and assist; Often an iterative process; Team Leader and Senior Advisor sign entire report; Team Member signs for Functional Area only; Dissenting opinion option, if no resolution. Nov 2006 Mod 12
Factual Accuracy Address Facts - Not Judgments . . . Quick turn-around at the working level will speed the process . . . Limit distribution of all draft products and e-mail prior to final approval/submission. Nov 2006 Mod 12
If the Outcome is not Successful Two Options: Suspend the Review. Interim Report including what was done and areas not done. Final report for completed Functional Areas (FA); High level discussion and findings for incomplete FAs. Terminate the Review. Report results with basis for repeat of entire ORR/RA. Clear discussion of what was and was not reviewed. What was and was not adequate! Nov 2006 Mod 12
Questions Nov 2006 23
Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Timelines, More Roles/Responsibilities, and Meetings Nov 2006
Readiness Review Process Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Initial Team Mtg Pre-Site Visit, IP Drafted) 6 Months Prior to RR 1 Year Prior to RR Generally 2 weeks Nov 2006 Mod 12
Learning Objectives Understand Readiness Review Timelines Identify Team Leader and Member Responsibilities Plan and Organize Opening, Daily, and Closeout Meetings Nov 2006
Development of Implementation Plan, ORR Time Line POA Approval Initial Pre- review visit Site Introduction meeting Conduct Review Close-out meeting Follow-up Review Team meeting 1 2 3 4 5 6 7 8 9 10 11 12 13 ORR Report Development of Implementation Plan, Generation & CRADs, & Checklists Review Nov 2006 Mod 12
Typical RR Timeline SUN MON TUE WED THUR FRI SAT ARRIVE Opening Records Operations Operations Interviews Report Meeting Review Writing Drills Drills Records Mangmt.. Review Interviews Mangmt.. Interviews SUN MON TUE WED THUR FRI SAT Off Finish Finish Closeout Forms Forms Continue Thursday and Friday, as needed; Report is signed and Submitted . . . Catch Up Peer Rev. Peer Rev. Report Report Sections Sections Brief Prep Brief Prep Nov 2006 Mod 12
Team Leader Responsibilities During the Review Ensure all team members have completed all access requirements prior to the review; Ensure all team members have completed all required reading prior to the review; Conducts the following meetings: Opening (kickoff); Conducts the Daily (team and site personnel); Conducts Daily Site Manager Briefings (as needed); Conducts Final Team; Conducts Final Out Brief. Nov 2006 Mod 12
Team Members Responsibilities During the Review Developing and Completing Form 1s and 2s: Provides draft forms to the Team Leader as soon as possible; Updates forms on a daily basis; Organizing and coordinating factual accuracy reviews; Completed Form 1s and 2s means that they have been: Completed, signed, and submitted to the Team Leader; Factual accuracy of the individual forms has been confirmed with the counterparts; Accepted by the Team Leader before the closeout meeting and before the team member leaves the site; Summary evaluations completed, submitted to, and signed-off by Team Leader prior to the closeout meeting. Nov 2006 Mod 12
Team Members Responsibilities During the Review Maintains accurate and complete notes of activities performed; Updates schedule of activities on a daily basis; Provides support in developing a final conclusion for following subject areas: Intergraded Safety Management; Lessons Learned; Adequacy of contractors review; and As assigned by Team Leader. Supports and attends ALL meeting as requested by Team Leader; Nov 2006 Mod 12
Team Members Responsibilities During the Review Work with other ORR/RA members: to resolve and communicate issues; to prepare the review documentation and report; to eliminate redundant reviews. Submit completed review documents for review and approval. Concur with determination of operational readiness and conclusions of ORR/RA report (in their area of assessment). Nov 2006 Mod 12
An Effective ORR/RA Requires Effective Coordination Avoid duplication of: Interviews, procedure reviews; Drills, exercise observations . . . Know time line of the review Schedule driven; Defined sequences; Stay on point, avoid distractions; Preparation of report forms critical and critical path . . . Nov 2006 Mod 12
Meetings Opening (Kickoff) Meeting; Daily Team and Site Personnel Meeting; Daily Site Management Briefing, if needed; Final Team Meeting; Final Out Brief Meeting. Nov 2006 Mod 12
Opening (Kickoff) Meeting Team Leaders Meeting Make introductions. Allow for a series of briefs by the site explaining the site's management and organization, its functions, operations and interfaces. Brief the audience on the ORR process and the ORR schedule. Brief site audience and team on the rules for interaction between team members and site personnel. Nov 2006 Mod 12
Opening (Kickoff) Meeting (Cont)… Rules for Interaction Between Team Members and Site Personnel: The team will strive to be invisible so as to not influence the performance of the facility; Request that all operators be briefed on these rules and in particular that no action should be taken in response to questions or discussion by the team; The team will initiate action, preferable through a supervisor or escort, if an immediate unsafe condition is encountered. Nov 2006 Mod 12
Opening (Kickoff) Meeting (Cont)… Rules for Interaction Between Team Members and Site Personnel: The team will not act as safety monitors or direct operations; The team will refrain from distracting the operators; The team will not violate site rules, cross boundaries without authority, violate control room/area boundaries or in any other way become an initiating event during the ORR; The team will not act as evolution or drill observers; Nov 2006 Mod 12
Daily Team and Site Personnel Meeting Meeting Performed at the end of each day Team Meeting First – Site Personnel Meeting Second; All Team Members Present; Site Personnel Meeting includes Management and all POCs; Remind site personnel that they are observers not participators. Nov 2006 Mod 12
Daily Team and Site Personnel Meeting Identification of any past, new, or unclear issues; Re-emphasize who is responsible for what area; Discuss and restate scope of review; Discuss process for going beyond the scope of the review; Discuss process for making contact outside of their assigned areas (e.g., interviews with senior management); Discuss validation of facts; Keep a running record of all the issues; Nov 2006 Mod 12
Daily Site Management Briefing Held only if requested by management (if management cannot attend daily meetings) Presents current and potential findings; Presents status of the review and support; Discusses any changes if needed. Nov 2006 Mod 12
Final Team Meeting Last opportunity to work and clarify any questions with Team Members; Due prior to this meeting are all: Final Forms 1 and 2 Draft summary evaluations (ISM, Lessons Learned, Contractor Eval, Summary Conclusion) Last opportunity to discover/address dissenting opinions and conflicts Any such issues must be addressed at this time to avoid any confusion at the closeout meeting. Meeting for TEAM Members ONLY, since no new information will be discussed or presented. Nov 2006 Mod 12
Final Out Brief Meeting Team Leader informs the site management of the major issues identified during the ORR; The closeout meeting kept short; Invite Site Management and POCs; No surprises at this meeting – no new issues or findings; As a goal, provide a draft copy of the final report to the senior managers; At a minimum, prepare the executive summary of the draft report as a handout to everyone attending the briefing; Nov 2006 Mod 12
Final Out Brief Meeting Team Leader informs the site management of the major issues identified during the ORR; Discuss the final report schedule and deliverables; Teams consensus will be presented: Their final conclusion, Any major issues, and Overall themes, if any will be presented. Exit briefing based on the Forms 1, Forms 2, and Team’s conclusions; Nov 2006 Mod 12
Questions Nov 2006 23
Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Document Evaluation Process Nov 2006
Readiness Review Process Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Initial Team Mtg Pre-Site Visit, IP Drafted) 6 Months Prior to RR 1 Year Prior to RR Generally 2 weeks Nov 2006 Mod 12
Learning Objectives Describe the preparation for onsite document review; Discuss the ORR/RA principle of sampling versus auditing; Discuss the complete trail sampling technique; Discuss good Team Member record / note keeping practices related to records review; Nov 2006
Document Evaluation Process Preparation Execution Follow-up Reporting Nov 2006 Mod 12
Document Evaluation Process Preparation Request documents before the Review Request those which address or describe the following Regulatory Drivers (DOE Orders, Federal Law) Safety Basis Drivers (DSA, TSR, etc.) Facility Procedures Process Procedures Nov 2006 Mod 12
Example of Documents to Request- Hierarchy Requirements Objective Core Orders, Codes, Requirements Standards, Regulations Criteria Identification of Required Program & Processes Identification of important implementing Procedures / Policies Identification of important Systems, Structures, Components Identification of important Job Factors Nov 2006 Mod 12
Performance Review Flow Top Tier Requirement Performance Review Flow DOE Site Office Requirements Requirement Specific Contract (what) Requirements Company Policy / Procedure Manual Project Procedures Implementation Specific (how) Line Instructions Result Project Records Nov 2006 Mod 12
Document Evaluation Process Preparation Request pertinent documents; Sampling - Not a complete document audit; Communicate with Contractor counterpart to determine appropriate documents; Communicate with the Team; Avoid duplication of review, if possible Ensure that Team Members are aware of documents that may affect their review Prioritize the selected target documents for review. Nov 2006 Mod 12
Document Evaluation Process Execution Read and understand the documents Determine if document contains required information to address applicable Criteria, Review and Approach Document; Determine if regulatory requirements are met or described; Determine how document is controlled and approved; Determine if procedures have been written correctly. Nov 2006 Mod 12
Document Evaluation Process Execution Do procedures accurately document and implement the following: Regulatory Drivers . . . Programs and Processes . . . Commitments . . . DSA/SAR / TSR / Accident assumptions; BIO / JCO / FHA, and EIS; List A/B, WSS, S/RIDs; Nov 2006 Mod 12
Document Evaluation Process Execution Do procedures accurately define and communicate Roles and Responsibilities for individuals as they relate to specific requirements.. Are they understood by those identified individuals . . . Do they comply with “writing guides”.. Nov 2006 Mod 12
Document Evaluation Process Execution Document Reviews do not have to be line by line copies of Order or regulatory driver, but must: Detail the mechanisms to fulfill the intent and requirements of the regulatory/contract driver . . . Detail the mechanisms and requirements to adequately control and document the activity . . . Focus on implementation . . . Nov 2006 Mod 12
Document Evaluation Process Follow-up Follow-up is needed if you are “Pulling the String”; Request additional documents if needed; Check beyond the document evaluation Training Controls to ensure procedures are followed Document configuration control Nov 2006 Mod 12
Document Evaluation Process Reporting Record the information relating to the acceptance and review criteria Check reference information Integrate information gained into observation and interview processes Discuss conclusions with Team Leader and Team Members as applicable Take Accurate Notes Nov 2006 Mod 12
Notes and Records Your personal permanent record of the review . . . Thorough enough to reconstruct: Names; Titles; Dates, Times; Specific Documents; Equipment identification; Events; Quotations or paraphrasing; Transfer to Forms daily – “Write Every Night”! Nov 2006 Mod 12
Summary… Does the documentation accurately and completely implement the requirements: Contractual? Authorization Basis? Programmatic? Performance? Is the documentation being implemented and requirements met? Is “Flow-Down” consistent, complete, and accurate? Nov 2006 Mod 12
Questions Nov 2006 23
Conduct of the Review – Mod 12 Interviewing Techniques Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Interviewing Techniques Nov 2006
Learning Objectives Describe the two major types of interviews; Describe how to prepare for an interview; Compare the strengths and weaknesses of various questioning techniques; Discuss the timing / scheduling of interviews during an ORR/RA. Nov 2006
Two Types of Interviews Management Learn how they do business Address issues discovered during review of their processes and procedure documentation Level of Knowledge Systems Safety Basis Processes Precautions/Hazards Principles Nov 2006 Mod 12
Interviewing Fundamentals of Good Interviewing Planning Opening Questioning Closing Post Interview Nov 2006 Mod 12
Fundamentals of Good Interviewing - Planning Plan questions in advance Define the purpose of the interview Keep the interview on track Focus on one issue at a time Cover important subject matters Gather data that will provide knowledge Review previous findings related to the topic Know the interviewee’s functions and responsibilities Control the interview setting (if possible) Nov 2006 Mod 12
Fundamentals of Good Interviewing - Opening Set the tone State the purpose of the interview Start with general topic and move to more detail as the interview progresses Obtain the title of the interviewee Establish and keep control of the interview Put the interviewee at ease State you will be taking notes and why Nov 2006 Mod 12
Fundamentals of Good Interviewing - Questioning Ask clear and concise questions Avoid leading questions Allow interviewee time to think through the response Keep interview on track Restate questions Ask follow-up questions Ask for clarification or basis of opinion Get complete answer to one question before progressing to the next Ask for illustrations or examples of documents alluded to by the interviewee Limit participation to two team members per interviewee ABOVE ALL - LISTEN CAREFULLY Nov 2006 Mod 12
Example Interview Questions Leading “How often do you discuss policy with a worker?” Neutral “How do the workers gain knowledge of company policy?” Primary “What tools are required by the Preventative Maintenance Procedure?” Secondary “What tools are actually used?” “Why are all the other tools taken to the work site?” Nov 2006 Mod 12
Example Interview Questions Open “Can you describe your relationship with your subordinates?” “Could you describe how preventative maintenance is scheduled over a month?” Closed “How often do you see a worker in a typical week?” “How far in advance does a worker know what work he will do?” Nov 2006 Mod 12
Fundamentals of Good Interviewing - Closing Closing should be a natural part of the interview Conclude the interview when the requested information is obtained You are not obligated to fill the scheduled time Provide interviewee with a summary and what will happen next Leave the door open for follow-up time if necessary Nov 2006 Mod 12
Fundamentals of Good Interviewing – Post Interview Immediately summarize information or conclusions! Check information or documentation referred to during the interview . . . Discuss conclusions with Team as appropriate . . . Conduct self critique of the interview: Determine if your technique created problems . . . Modify your approach as necessary! Nov 2006 Mod 12
Interviewing Short pertinent notes Necessary, but can be distracting to the interview Pre-written questions are helpful Provide time for yourself immediately after the interview to clarify your notes Nov 2006 Mod 12
Interviewing Listening Process Active Listening: 70% of an interview should be listening Hearing the message Processing the message Providing feedback to ensure you understand Nov 2006 Mod 12
Interviewing Barriers to Effective Listening Preconceived ideas Competing demands Excess mental time Emotionally charged words Viewing listening as a passive process Detrimental actions Nov 2006 Mod 12
Interviewing Don’ts Interviewer Talk too much Communicate desired response Fail to follow-up leads Interrupt Manage time poorly Write too much Form premature conclusions Nov 2006 Mod 12
Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Observation Process Nov 2006
Learning Objectives Describe how to prepare for an observation; Explain the principle of observation vice participation (Heisenberg Principle); Describe requirements based observation techniques; List several types of ORR/RA evolutions; Explain the need for a facility to run a drill vice only respond to a drill. Nov 2006
Purpose of Observations To perform, to the maximum extent possible, a representative sample of anticipated operations and evolutions, and responses to credible casualties and “off-normal events” in order to demonstrate the facility operational readiness . . . Single most important step in the process: Serves as a foundation; Team must select representative operations; Surrogate material use . . . Nov 2006 Mod 12
Observation Process Preparation Perform Observation Follow-up “You can see a lot by watching” - Berra’s Law - Lawrence Peter (Yogi) Berra Preparation Perform Observation Follow-up Reporting Nov 2006 Mod 12
Fundamentals of Good Observations - Preparation Decide what is needed Is it important to the facility, personnel, procedures, management systems Is it a known or suspected problem area Are there corrective actions to rectify previous problems Nov 2006 Mod 12
Fundamentals of Good Observations - Preparation Select the Activity Targets of Opportunity Drills/Dry runs System Operability Checks Maintenance Work In-Progress Importance to safety and reliability Problems or potential problem areas Performance of new people Effectiveness of new or revised procedures and/or controls Existing condition of equipment Nov 2006 Mod 12
Types of Observations Exercise Briefings Drills Training session Operations Maintenance Surveillances Tours System startup Briefings Training session Critique Meetings Turnovers Walk downs Alarm response Nov 2006 Mod 12
Important Inter-relationships Nov 2006 Mod 12
Fundamentals of Good Observations - Observation Interaction with observed personnel: Establish rapport; Explain purpose of observation; Outline your role as an observer: Questioning techniques; Will not interfere with work; Be Transparent. Nov 2006 Mod 12
Fundamentals of Good Observations - Observation Take thorough notes in real time: Times actions occurred; Procedure details- have procedure in hand; Nomenclature and physical details; Be quantitative, record pertinent data; Include questions and follow-up items; Follow procedure as conducted. Nov 2006 Mod 12
Fundamentals of Good Observations - Observation Clearly understand what actually happened: Don't assume . . . Ask clarifying questions without interfering; Seek additional details as needed; Full understanding is imperative to accuracy of the observations; Coordinate with other Team Members who observed. Nov 2006 Mod 12
Fundamentals of Good Observations - Observation Start at the true beginning: Observe Pre-job briefings; Observe system alignment checks; Observe daily checks; Note the peripheral activity: People; Environment; Preparation/set-up activities. Become creatively inquisitive . . . Keep attention to important items . . . Nov 2006 Mod 12
Heisenberg Uncertainty Principle Difficult to make an accurate measurement without affecting what is being measured . . Difficult to measure people performance without influencing their performance . . Do not ask prompting type questions that could change performance . . . Nov 2006 Mod 12
Rules for Observations Do not act as a Safety Observer! Do not violate rules! Do not interfere with operations! Do not impose conditions, simulations, or anomalies! Be transparent . . . Request Facility to communicate roles and rules! Nov 2006 Mod 12
Fundamentals of Good Observations – Follow-up Valid conclusions require good follow-up Resolve open questions . . . Primary purpose of follow-up: What you observed are indicators . . . Follow-up will define real problem: need for improvement; underlying causes. Clarify any open questions Pull the Strings Nov 2006 Mod 12
Fundamentals of Good Observations – Reporting Observations provide the basis for: Findings; Observations; Correlation with Interviews/Document Reviews. Combine several observations if appropriate to draw conclusions concerning program status: Rarely should a single observations result in a finding; Adequacy is the standard, perfection is the goal. Record immediately after observation, Form 1 and 2 Nov 2006 Mod 12
Questions Nov 2006 23
Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Finding Categorization Process Nov 2006
Learning Objectives Distinguish between the various types of findings; Understand the methodology of finding categorization; Clearly document a finding. Nov 2006
Finding Identification and Categorization A single issue, finding, or group of related issues, which has been documented on a Form 2, may constitute a finding. Team members in the evaluation of specific Functional Areas should group similar issues to a single finding that concisely characterize the systemic deficiency. The team leader, in consultation with the team member, has the responsibility for making the determination of whether a finding is a Pre-or Post-start Finding. Nov 2006 Mod 12
Finding Identification and Categorization Pre-start Finding – A finding that must be corrected before an activity can be started. Post-start Finding – A finding that must be corrected, but may be corrected after the start of the activity. Post-start findings are addressed by a corrective action plan, which includes any compensatory measures taken. Observation – An issue of concern identified by the RA team that should be evaluated for possible corrective action in terms of opportunities for improvement. An observation is an issue that does not reach the level of being considered by the team as a finding but warrants evaluation. Nov 2006 Mod 12
Pre-Start / Post-start Categorization Prior to being published, each finding should be identified as to whether or not it is a Pre-Start or Post-Start. Guidance is in the Order, Standard, Handbook and may also be included in the POA and IP. Lets SEE The Example… EXAMPLE Nov 2006 Mod 12
Pre-Start / Post-start Categorization A non-compliance issue does not necessarily mean a Pre-Start finding! Does the non-compliance represent: significant risk? significant mission impact? significant political impact? significant lack of control? Nov 2006 Mod 12
Pre-Start / Post-start Categorization Unacceptable Risk Higher Consequence Acceptable Risk Probability Post start Nov 2006 Mod 12
Pre-Start / Post-start Categorization Does this finding represent a deficiency of such magnitude as to prevent startup until resolved? Clear failure to implement commitment or assumption of TSR or DSA? OR: Is it related to sustainability? Not a significant safety Issue? Limited impact to Activity within review scope? Nov 2006 Mod 12
Some Obvious Prestart Criteria Causes a loss of operability that prevents safe shutdown or loss of monitoring ability of safety related systems (i.e. important to safety) . . . Indicates a lack of control that can have near term impact on operability or functionality of safety related systems . . . Requires operator action in short period of time to prevent or mitigate consequences . . . Violates assumption or controls by which DOE/NNSA accepts risk of operations . . . Nov 2006 Mod 12
Some Obvious Prestart Criteria Results in operation outside of limits of TSRs, OSRs, etc. . . . Results in an Unreviewed Safety Question: Increase in the probability or consequence of accident or failure mode described in SAR? Introduction of a new accident or failure mode not described in SAR? Reduction in the margin of safety as described in SAR? Results in failure to comply with Authorization Bases commitments (TSR, LCOs, Surveillances etc.) . . . Nov 2006 Mod 12
Some Obvious Prestart Criteria Operations/Activities that can lead to: Environmental release of hazardous material in excess of reporting guidelines, permit limits, etc. Exposure of off-site personnel in excess of radiological guidelines for off-site exposure or chemical exposure in excess of minimum reportable amounts. Worker / on-site exposures in excess of guidelines or requirements. Nov 2006 Mod 12
Finding - example Two LCOs (10.1.1 and 10.2.2) specified in the activity SAR are not covered by approved surveillance procedures. Nov 2006 Mod 12
Finding - example Emergency exits from this facility do not meet the requirements of NFPA 101 (Life Safety Code). The exits are required to have quick- open "crash bar" hardware. Nov 2006 Mod 12
Finding - example Posting of Radiological areas and entry control programs for these areas per the requirements of 10 CFR 835 were inadequate. No control device was used as required on each entrance to the hot cell 2 area (high radiation area). Nov 2006 Mod 12
Finding - example Although an adequate energy isolation procedure has been developed for this activity, maintenance personnel that service equipment in the area were not familiar with the procedure. No program has been established for root cause determination of maintenance failures, as required by DOE O 4330.4B Nov 2006 Mod 12
Finding - example No valve identification was provided for the line emergency shutdown system. Although operators questioned were well aware of the shutdown procedure, new personnel may not be aware of these valves. Nov 2006 Mod 12
Finding - example The pipe content and direction flow labeling for this process has been limited to those lines with hazardous materials. Non-hazardous material lines have not yet been labeled. Nov 2006 Mod 12
Don't just identify the finding. . . Understand the potential impact of the finding! The possible consequences of not resolving it . . . The potential that those consequences will be realized and impact: - personnel; - environment; - facility; - mission; - public . . . Nov 2006 Mod 12
Questions Nov 2006 23
Readiness Review Course Screening or Scoping Meeting (ORR vs RA, Authorization Authority (AA) Defined, Startup Notification Report (SNR) Discussed, Schedule) Startup Notification Report Prepared and Submitted AA Approves SNR and Nominates DOE Team Leader (TL) Readiness Review Process Begins Contractor Plan of Action (CPOA) Prepared DOE Plan of Action (POA) Prepared by DOE Line Mgr AA Reviews and Approves CPOA and POA Contractor RR Implementation Plan (IP) prepared by Contractor RR Team and approved by CTL DOE RR IP approved by TL Prerequisites met (MSA) Contractors RR and Final Report Corrective Action Plan (CAP) Submitted and Approved Prestart Findings Closed Contractor Issues Readiness to Proceed (RTP) DOE Endorsement of RTP and MSA AA Authorizes Start of RR to RR TL DOE RR and Final Report CAP Submitted and Approved AA Authorizes Startup DOE TL Pre-Assessment Preparation and Planning (Team Selected Pre-Site Visit, IP Drafted) Initial Team Mtg Conduct of the Review – Mod 12 Final Report Format and Administration Nov 2006
Learning Objectives Describe all ORR/RA onsite administration as it relates to a Team Member including: General Time Line Final Report Process Form 1 Form 2 Executive Summary Lessons Learned Recording Dissent Peer Review Nov 2006
Typical RR Timeline SUN MON TUE WED THUR FRI SAT ARRIVE Opening Records Operations Operations Interviews Report Meeting Review Writing Drills Drills Records Mangmt.. Review Interviews Mangmt.. Interviews SUN MON TUE WED THUR FRI SAT Off Finish Finish Closeout Forms Forms Continue Thursday and Friday, as needed; Report is signed and Submitted . . . Catch Up Peer Rev. Peer Rev. Report Report Sections Sections Brief Prep Brief Prep Nov 2006 Mod 12
Final Report Documentation Team Leader (TL) will prescribe specifics of format . . . TL should provide examples, as early in the review as possible . . . IP will define content of final report . . . TL will assign report responsibilities . . . Nov 2006 Mod 12
Final Report Preparation Process Team Member complete assessment forms Team Member validates assessment forms with contractor and/or DOE Team Leader reviews and approves the assessment forms Team Member provides an evaluation of the Contractor Readiness Review and finding closure packages for their functional area Team Member submits any input to the Lessons Learned Nov 2006 138 Mod 12
Final Report Preparation Process … Team Leader composes executive summary and balance of report content Assessment Forms are included as appendix Team Leader and Team Members review and sign the report cover sheet Team Leader transmits report to the Authorization Authority with a Recommendation Nov 2006 Mod 12
Final Report Preparation Process … Differing Opinions or Minority Opinions The Readiness Review Report SHALL provide an opportunity for Team Members to include: Differing professional opinions Non-judgmental general comments Observations Dissenting opinions of Team Members should be documented and attached to the report Nov 2006 Mod 12
Final Report Format EXAMPLE TITLE PAGE (Cover) TEAM SIGNATURE PAGE OFFICIAL PAGE CHANGES AND EDITORIALS EXECUTIVE SUMMARY CONCLUSION INTRODUCTION NNSA ORR TEAM EVALUATION NNSA ORR TEAM INTEGRATED SAFETY MANAGEMENT FINDING CATEGORIZATION AND DOCUMENTATION APPENDIX A LESSONS LEARNED APPENDIX B FUNCTIONAL AREA FORMS 1 APPENDIX C FINDING FORM 2 APPENDIX D ACRONYM TABLE EXAMPLE Nov 2006 Mod 12
The Final Report … Documents the logic of the review Conveys the results of the review Forms the basis for recommendations for The readiness of the activity to proceed with startup or restart The readiness of the management system (Contractor and DOE) to manage and oversee the activity Nov 2006 Mod 12
Documenting Review Methods Assessment - Form 1 Use Form 1 to document methods and actions taken by team members in evaluation process. Each Form 1: Covers specific objective; Documents team members review methodology; Complete enough to allow a reviewer of form to follow evaluation logic; Follows approach detailed by CRAD (document differences, if any, and reasons for change); States specifically whether criteria met or not; Lists issues (Form 2) associated with that objective. Nov 2006 Mod 12
Form 1 Objective: CRA / Rev Criteria Met Functional Area: Yes Date: No Objective: (Core Requirement) Criteria (method of appraisal) Review Approach: Records Review - Personnel Interviewed Title (no names) Evolutions / Operations / Shift Performance Discussion of Results: Criteria by criteria to show basis for criteria met or not met Conclusion: Criteria met or not met. Paragraph discussion of overall results and criteria. Issues: Form 1 Inspected by: Approved by: Team Leader Nov 2006 Mod 12
Documentation of Deficiencies Deficiency Form 2 Used to document findings identified during criteria evaluation process . . . Requirement based = clear and specific description of the requirement and the condition which is deficient . . . Primary objective = clear communication . . . Nov 2006 Mod 12
ISSUE: the identified finding or observation Form 2 Finding: Functional CRA no. / title Prestart: Issue no.: Area: Observation: Post-start: Date: ISSUE: the identified finding or observation REQUIREMENT: requirement statement(s) from reference REFERENCES: specific as possible, including section and paragraph DISCUSSION: Inspected by: Approved by: Team Leader Nov 2006 Mod 12
Finding Documentation When documenting a finding the finding must include the following: The Criteria being assessed, or measured; The Review Approach used to derive a conclusion on the assessment of the criteria (document review, observation, or performance evaluation); A discussion regarding the Review Approach used, and; Identification of the Finding, with proper citation of the requirement. Nov 2006
Finding Documentation Finding documentation begins with: A brief finding statement, and A citation of the requirement, Followed by the discussion paragraph (which discusses the finding by emphasizing the review approach used, as-found conditions, and relationship towards the requirement.) Nov 2006
Finding Documentation Example Issue Statement: The Lightning Protection System (LPS), which is defined as a Safety Class system within the Documented Safety Analysis, does not meet the design specifications referenced by DOE O 420.1, Facility Safety, para. 4.1.1.2 Design Requirements. Nov 2006
Finding Documentation Example Discussion: DOE O 420.1, 4.1.1.2, states that all Facility safety class electrical systems shall be designed to the basic approach outlined in Section 5.2.3 (Electrical) of DOE G 420.1-1, Nonreactor Nuclear Safety Design Criteria and Explosives Safety Criteria Guide for Use with DOE O 420.1, Facility Safety, dated 3-28-00. Referenced under Section 5.2.3 is Table 5.5, which list hardware design requirements for both Safety Class and Safety Significant SSCs. As referenced, NFPA 780, Standard for the Installation of Lightning Protection Systems; requires that systems be bonded. A walk down of the LPS for the roof area identified several deficiencies, such as unbounded metallic penetrations and electrical conduit. Nov 2006
Other Sections of the Final Report Summary: Brief overview of functional area results; Less than one page. Executive Summary: Summaries each of the findings, Provides Functional Area Conclusion Provides Team’s Overall Conclusion Lessons Learned, “Required by Order”: TL+TMs review lessons learned on EH web site; All asked to contribute, one TM coordinates for TL. Nov 2006 Mod 12
Review and Approval of Your Forms Completed by Team Member; Approved by Team Leader; Senior Advisor/s involved and assist; Often an iterative process; Team Leader and Senior Advisor sign entire report; Team Member signs for Functional Area only; Dissenting opinion option, if no resolution. Nov 2006 Mod 12
Factual Accuracy Address Facts - Not Judgments . . . Quick turn-around at the working level will speed the process . . . Limit distribution of all draft products and e-mail prior to final approval/submission. Nov 2006 Mod 12
If the Outcome is not Successful Two Options: Suspend the Review. Interim Report including what was done and areas not done. Final report for completed Functional Areas (FA); High level discussion and findings for incomplete FAs. Terminate the Review. Report results with basis for repeat of entire ORR/RA. Clear discussion of what was and was not reviewed. What was and was not adequate! Nov 2006 Mod 12
Questions Nov 2006 23