1. Company Confidential Registration Management Committee (RMC) 1 AQMS Accreditation Programs ANAB Findings Atlanta, GA July 22, 2010 Steve Holladay ANAB, Accreditation Assessor Auditor Workshop Atlanta, GA July 22-23, 2010

2. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 Objectives Provide an overview of the NCR’s identified from the witness audits. Present in process approach. Discuss steps that could have been taken to prevent the NCR’s. Rules of engagement. 2

3. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 Goal Provide knowledge/information to the AS auditor pool with the goal to improve auditor/CB performance that will reduce repeat NCR’s which will, in turn, add value to the assessment process and the industry as a whole. 3

4. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 4 120 NCR’s in 54 Witnessed Audits (WA) –2.22 Average per WA 50 NCR’s in approx 40 Office Assessments (OA) –1.25 Average per OA Overall

5. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 5 3 NCR DETAIL: –Assigning Auditors who either did not have the proper NACE code or AS91XX qualifications. –Client not made aware of the OP assessor. LEARNINGS: –Verify auditors are qualified for the full dynamics of the audit. –Communicate clearly with the client. Pre-Audit Assignment

6. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 6 11 NCR’s DETAIL: 19011, 6.4.1.c –(4) The audit plan did not ID the organizations functional units and processes to be audited. LEARNINGS: –The audit plan ensure the client will have the proper resources available to the audit. –Clearly demonstrates what processes are intended to be audited. Pre-Audit Planning

7. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 7 11 NCR’s DETAIL: 17021. 9.3.2.1 –(2) Audit Planning not effective in assuring the AQMS is assessed to the min requirements for surveillance audits. LEARNINGS: –Ensure the auditors understand where the full auditing requirements are for surv audits if not clearly ID in the audit plan. Pre-Audit Planning

8. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 8 11 NCR’s DETAIL: MD 5 and AS9104 –(5) Insufficient Auditor Days in the planning without clear justification. LEARNINGS: –If deviation…..JUSTIFY. –If the audit is being witnessed, provide to ANAB in the pre-audit information. Pre-Audit Planning

9. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 9 5 NCR’s, several OFI’s. DETAIL: –Information missing or not correct in EQM. –Client Profile form not complete or accurate. (scope, head count, regulatory, ITAR) LEARNINGS: –Ensure the person entering information is knowledgeable about the information. –Verify with Client and lead auditor before submitting. Pre-Audit Preparation

10. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 10 Total of 24 NCR’s DETAIL: –Sampling part of an element and making a conclusion on the whole requirement. LEARNINGS: –Audit plan and report accuracy is critical. On-site Audit Depth of auditing

11. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 11 DETAIL: –Auditing to the scope of certification is not adequate. LEARNINGS: –Ensuring the scope of certification, scope of audit and audit plan is consistent. –Validate the exclusions. Is scope consistent with the QM. –Ensure the address and scope on the Certificate is correct. On-site Audit Depth of auditing

12. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 12 DETAIL: –Not following audit trails when objective evidence suggest otherwise. LEARNINGS: –A plan is a plan. Follow the trail to its natural conclusion if potential findings are evident. –Keep head on a swivel. Don’t ignore clear findings. On-site Audit Depth of auditing

13. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 13 DETAIL: –Not fully verifying the effectiveness of the actions taken on nonconformities identified during the previous audit. LEARNINGS: –Ensure the evidence is more than a record review. On-site Audit Depth of auditing

14. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 14 4 NCR’s DETAIL: –Design Applicability LEARNINGS –More of an issue with AS9110. On-site Audit Interpretation

15. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 15 DETAIL: –Not fully auditing Outsourcing when applicable. LEARNINGS –Standard clearly requires “control of such outsourced processes shall be identified” and therefore subject to audit. Not just a document verification. On-site Audit Interpretation

16. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 16 DETAIL: –Changes to the capability listing of the client. LEARNINGS –Similar to the scoping discussion. Verify any changes to the capability listing as this could affect the scope of certification or introduce new processes and technology. On-site Audit Interpretation

17. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 17 11 NCR’s DETAIL: –Calling the Clients CA/PA system effective with ineffective/inadequate correction, root cause or corrective action. LEARNINGS: –Deep dive the CA/PA system. Validate the information. If it doesn’t make since….make the call. On-site Audit Decisions

18. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 18 DETAIL: –Soft Grading 8 NCR’s –Clearly stating findings during the audit and not raising the NCR or improper categorization. – Accepting informal correction during the audit. LEARNINGS –Learn the definitions and follow them. Keep track of the “verbal” findings identified during the audit. Accepting correction to verbal findings without reporting is consulting. –As an auditor it is not your job to justify an NCR down. On-site Audit Decisions

19. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 19 DETAIL: –Continuing the audit when the audit objectives are clearly unattainable. LEARNINGS: –Only the client can make the decision to continue an audit. –There should be a clear conclusion that the objectives are unattainable and reported to the client with options. »At the time this is realized not at the end!! On-site Audit Decisions

20. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 20 DETAIL: –Closing meeting does not address all of the requirements. LEARNINGS –Clearly address the CA follow up and impacts on the existing cert. –Ensure the CA are presented to the correct requirements. On-Site Audit Closing

21. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 21 22 NCR’s DETAIL: –Accuracy of the Audit Report and Checklist to the observed audit. LEARNINGS –Anecdotal evidence should be documented as such. Same for deductive evidence. On-site/Off-site Audit Reporting

22. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 22 DETAIL: –Report not in conformance with AS9014/AS91XX LEARNINGS –Justify/explain all NA. –Clear evidence to support the conclusion. –Clear evidence to support the SCOPE. –Include detailed notes On Site/Off-Site Audit Reporting

23. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 23 3 NCR’s DETAIL: –Report did not contain mandatory items. LEARNINGS –Ensure the Checklist are complete or references the location of the information. –Explain and differences between the information left on site to the published report. Post Audit Reporting

24. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 24 8 NCR’s DETAIL: Improper closing of NCR’s and Improper Certification Decision. LEARNINGS: –Clear evidence of Correction, RC and CA. –Ensure persons closing NCR are AEA. –Ensure the NCR’s are closed with appropriate evidence PRIOR to the cert decision. Post Audit NCR Closure

25. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 Office Audit NCR’s OASIS data base admin/accuracy Verification of Client OASIS data admin Justification of auditor days. Improper qualification of AS cert decision maker. Improper decision made with certification information. 25

26. Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 26 Questions