1. Self-Identified Checklists and Phased Audit Training
Susan Aderholdt
ASP Workshop
September 19-22, 2016

2. Phased Audit using Self-Identified Checklists
Phased Audit Overview
Phase I Audits
Phase II Audits
Phase III Audits
Checklists
Lessons Learned
Please feel free to ask questions at any time during my presentation.
TALK SLOW

3. Phased Audits The Pilot Program
DOE Order 435.1, Radioactive Waste Management
Free Up Auditors
Money Saving Opportunity
Updated the checklists
In an effort to be proactive with the DOECAP Auditing Program, we have initiated what we originally referred to as the Pilot Program.
requires the field element manager to determine if a TSDF is acceptable based on an annual DOE review.
It is hoped that this would free up auditors so they could perform additional audits that were being requested.
We also hope it will be a money saving opportunity by reducing travel costs through desktop audits.
to allow the sites to perform their own audits incorporating the DOECAP lines of inquiry into their own reviews and initiating corrective actions.

4. Phased I Audits In Phase I
A full DOECAP audit team then descends upon the site
Development and completion of adequate corrective actions
This does not preclude the auditors writing findings for issues not previously identified by the site.
If the Phase I audit is successful, the audit team and the Operations Team provide a recommendation to the ASP Manger to proceed with a Phase II audit.
the sites complete the DOECAP Checklists noting any findings and initiating corrective actions.
and performs the audit as usual with the exception of using the site-completed checklists.
have been developed, or completed, DOECAP does not issue finding for the self-identified findings.

5. Phase II Audits Phase II Audits are limited-scope audits.
The audit disciplines that are not included in the on-site audit are the areas where the facility demonstrated during Phase I that it has implemented:
Mature Programs
Incorporated DOECAP lines of inquiry into its operational activities, assessments and audits
Adequately addressed DOECAP findings
Routinely corrects other identified deficiencies
Phase II audits begin with a request for pre-audit material as usual and the completed checklists from the previous year are sent to the facility.
with a reduced number of auditors with both onsite and desktop auditors

6. Phase II Audits
The facility completes the DOECAP checklists and provides:
Detailed documentation from its issues management system in a summary table that includes all internal and external findings issued since the last DOECAP audit
Associated causal analyses, corrective actions, and extent of condition
All open findings from previous years
Summary of any regulatory issues since the last DOECAP audit
Summary of major changes in facility operations or staffing since the last DOECAP audit
A modified audit team provides on-site auditing with the remaining disciplines being audited from the auditors desk.
All audit modules are provided to the lead auditor while on-site so a draft audit report may be left with the facility.

7. Phase III Audits
All pre-audit material requested remains the same as in Phase II
All audits are conducted at the auditors desk, no travel is involved
The material must be provided to the auditors 30 days prior to the scheduled audit
The auditors have 30 days to review the materials and write their section of the report, including the executive summary
Auditors send by their completed reports and checklist to the lead auditor who compiles the draft report that is sent to the facility
Facility has 5 working days for factual accuracy and must return the redline report to the lead for submittal to the DOECAP operations team

8. As you can see the checklists have been modified to supply a facility status and response and columns for DOECAP auditor status and response. One of the changes I am going to make to the checklists is state at the top of the far right column DOECAP Auditor. This caused some confusion at one site. The facility provided a condition report with their corrective actions performed to the findings they issued to themselves.

9. Not being present to view was a problem in all areas audited
Not being present to view was a problem in all areas audited. In some cases you must take their word. Every checklist were noted to have this problem.

10. You must ask the facility to supply certain documents
You must ask the facility to supply certain documents. One lesson learned is the facility should put the SOP they are referencing in their response. In this case the auditors had to look through hundreds of SOPs to find the one referenced to verify the program existed as required.

11. Be thorough. Supply as much information as possible
Be thorough. Supply as much information as possible. This goes for the facility and the desktop auditors.

12. Several conversations were held with facility representatives for clarification. That is why the facility needs to provide us a list of potential contacts with phone numbers and address so it we have questions we can contact the appropriate employ just as we would if we were on site.

13. Be specific on the documents you review
Be specific on the documents you review. Request specific docs you want to review. You will have an org chart request training documents for specific individuals just as you would on site.

14. The facility must be very thorough as to what they review using the LOIs. More so than when we are on site because we are not there to see.

15. Sometimes the facility performs their reviews months before we do and things like training have expired when we perform our review. And then again sometimes they miss things that we catch.

16. Finding
Continuing training was not completed at the required frequency and a documented extension for completion was not approved as required by ZZ-XX-PR-100, ABC Facility Training. Also records of completion of continuing training was not available for one individual.
Supporting Discussion:
Equipment Operator II, Orange Badge, 24 hour OSHA was completed on 11/12/14. The 8-hour HAZWOPER refresher and Asbestos training were completed on 1/26/16. Over 60 days later, there was no Memo To File documenting an approved extension for either class.
Manager, Shipping and Receiving, Orange Badge, Waste Worker, Spill Response training was taken 1/21/14, there was no evidence of completion for 2015 or 2016 training.
Safety and Health Specialist, Orange Badge, Waste Worker, took DOT Hazmat Employee Awareness, 24 Hr. HAZWOPER, and Asbestos training, on 10/30/14 and again on 2/25/16, almost 4 months late, and a Memo to File approving an extension was not found.
QA Manager, Orange Badge, Waste Worker, took Defensive Driver, required every 3 years, on 11/18/09. A 3 year retake should have occurred in 2012 and It did not.
Director, Safety and Health, Orange Badge, Waste Worker, is behind on several classes. The ABC Safety and Health Manager stated he “…is a corporate employee and did not attend the site refresher for 2016 and is not approved to go into the restricted area at this time.”
This is the finding written by the auditor for the training that was not completed. This is not the format that it appeared in the report. This was in the checklist notes. Excellent write-up and will be easy to copy and paste into the report. This auditor did an excellent job with the desktop audit.

17. Ask for specific calibration data.

18. Ask for shipping and manifest records for a time period, for example all data for a specific 6 month or 1 year period of time.

19. Here we had an observation after reviewing the data provided
Here we had an observation after reviewing the data provided. This is a requirement for labs, but not for TSDFs who are only using the data for their own knowledge.

20. Again, in every checklist there are items that cannot be verified from a desktop audit.

21. This is an example of what I found during a partial desktop audit, a phase II audit. The facility performed their audit 7 months prior to me performing my desktop audit. I had to request updated documents. The ones they provided were now out of date or expired.

22. Another one where I had to talk to several facility personnel and request new data. Also, this was a Phase II so we had auditors on site in this case I thought I had two findings but I was able to talk to the auditors on site and they went to check for me and I was able to eliminate the findings. Without an auditor on site, I could not have verified that these findings were not valid.

23. Lessons Learned
While performing desktop audits several issues were identified as lessons learned
timeliness of data
calibration had expired in November 2015 desktop in May 2016
incorporated our lines of inquiry into various walk-throughs during the year thus the time difference
contact information for the facility for updated calibration results
same problem for several other instruments
some instruments in no longer in use
procedures no longer in use
The DOECAP audits were performed throughout the year during the facility's walkthroughs data was collected and sent to us upon our request for pre-audit information Calibrations had expired, instruments taken out of service, procedures no longer used, but the information was current when they performed their walkthrough, but not current at the time of the desktop.

24. Lessons Learned We need to: We need more time
request all phone numbers and s addresses for the employees we may need to talk to.
be notified of procedures that were no longer in use or equipment taken out of service
be more specific in what we request
Ask for specific training documents, calibration records, manifests, etc…
We need more time
the auditors need at least a month to review the material
lead auditor needs the auditor’s report by Tuesday or as early as possible prior to completion of the draft report, there will be questions
One example, I needed to talk directly with the laboratory employees and did not have contact information for them. It took three days to get the correct employees information and to talk to them. We need that up front.
Procedures and equipment were taken out of service between the November collection date and the time of the audit in May.

25. Lessons Learned
The Sampling and Data Quality has no real references and it is difficult to write findings (this is true for onsite audits too).
Examples of difficulty of not being on site:
A site’s procedure for Gamma Spec addressed efficiency calibration, but the response provided by the facility addressed Alpha. (without being present to interview the lab personnel and view the handling of this process, we cannot know what they are actually doing)
Housekeeping, PPE use, fire extinguisher checks, and too many more to mention
The auditors and the audited facility believe a full desktop audit is not very effective and at most there should be a modified audit with at least three auditors on site.

26. Lessons Learned
All auditors who have participated in the desktop audits agree that it is very difficult to do, it cannot be done in three days.
Even with manager support while at the office there are too many interruptions, you cannot maintain the concentration you have at the audit.
The facility who participated would prefer we show up for the three day audit in person.

27. Audit Expectations for the Facility
Each modified audit checklist section provides fields for the facility to enter the name of the auditor/subject matter expert who filled out that section and the date completed.
The response for each audit checklist line of inquiry must list
the date completed,
the activity performed,
identification of the associated documentation (e.g., inspection, report, procedure, etc.) and the date it was issued, and
the name and title of the person who performed the activity so that the DOECAP auditor can more easily verify the information provided.

28. Audit Expectations for the Facility
Name and contact information for the facility’s staff member who has overall responsibility for each audit checklist must be identified on the list of contact information provided to DOECAP.
If the DOECAP auditor has questions about the contents of the audit checklist, the auditor’s starting point is to contact the identified staff member.
Facility must review all completed audit checklists and ensure their overall quality and continuity prior to submitting them to DOECAP.
Names, e‑mail addresses, and telephone numbers of any other POCs identified for the DOECAP audit disciplines must be provided to the DOECAP Operations Team.
DOECAP will provide the contact information to the audit team in the pre-audit information.

29. Audit Expectations for the Facility
The facility must provide detailed documentation from its issues management system. This documentation must:
Be organized by the DOECAP audit disciplines (i.e., the six laboratory disciplines or the seven TSDF disciplines, as appropriate).
Include all findings from all reviews performed at the site, such as the daily, weekly, monthly, or annual walk-downs and other internal/external assessments, audits, and surveillances.
Information must include:
the type and date of surveillance/assessment/audit status
name of the person who evaluated the area
the corrective actions and any updates
findings that the facility identified when it assessed itself against the lines of inquiry in
the DOECAP audit checklists

30. Audit Expectations for the Facility
Include any open findings from previous years.
open findings identified by other auditing entities (external findings)
open self-identified findings noted during assessments, audits, surveillances, walkthroughs, and inspections (weekly, monthly, and annual),
any open findings that were identified by the facility’s employees.
List the open findings for each audit discipline in a separate file that is clearly labeled (electronic file name and document title) to match it to the associated audit checklist.
Provide details:
the causal analysis
identified corrective actions for each of the findings identified since the last DOECAP audit
the current status of each issue (i.e., open or closed)

31. Audit Expectations for the Facility
The facility must also provide the following information:
A summary of any major changes in facility operations or staffing.
A summary of any regulatory issues since the last DOECAP audit (e.g., fines or other penalties, Notices of Violation).
In addition to the usual pre-audit document package (e.g., procedures, plans, permits), the facility must electronically provide the other requested documentation (listed above) to the DOECAP Operations Team four weeks before the start of the audit.

32. Facility Must Provide to DOECAP
The facility must respond to an e‑mail request from the Lead Auditor two weeks before the audit starts by providing a list of:
documents that have been revised since the facility performed its audit
equipment taken out of service
procedures no longer used
changes to the facility
changes to the staff

33. Procedure U.S. Department of Energy
Office of Environment, Health, Safety and Security
Procedure AD-1-2A
Conducting DOECAP Laboratory and TSDF Audits
Revision 1.0 (DRAFT D - 2/8/2016)