Determining Effectiveness of Internal Audits Presented by: Tony Gutierrez R. Darrell Taylor Minneapolis, MN July 19 & 20, 2012

To discuss how an auditor audits the Internal Audit Process Purpose To discuss how an auditor audits the Internal Audit Process and determine how effective is the process

Definition Effective - producing a decided, decisive, or desired effect. (Merriam- Webster) In addition to establishing conformance with a set of rules, quality audits may measure the adequacy of procedures and the effectiveness of implementation to which those rules assist in achieving basic goals. (JP Russell, The Quality Audit Handbook)

Process Based Auditing History Lesson Metrics Method Infrastructure People Input Output Process Process Based Auditing Risk Based Auditing Risk Issues Effectiveness Value Added Document Control Record Control Purchasing Production Calibration Internal Audit Corrective Action Compliance Auditing 11/22/201811/22/2018

The Requirement - AS9100C 8.2.2 Internal Audit The organization shall conduct internal audits at planned intervals to determine whether the quality management system a) conforms to the planned arrangements (see 7.1), to the requirements of this International Standard and to the quality management system requirements established by the organization, and NOTE: Planned arrangements include customer contractual requirements.

The Requirement AS9100C (cont.) b) is effectively implemented and maintained. An audit program shall be planned, taking into consideration the status and importance of the processes and areas to be audited, as well as the results of previous audits. The audit criteria, scope, frequency and methods shall be defined. The selection of auditors and conduct of audits shall ensure objectivity and impartiality of the audit process. Auditors shall not audit their own work.

The Requirement AS9100C (cont.) A documented procedure shall be established to define the responsibilities and requirements for planning and conducting audits, establishing records and reporting results. Records of the audits and their results shall be maintained (see 4.2.4).

The Requirement AS9100C (cont.) The management responsible for the area being audited shall ensure that any necessary corrections and corrective actions are taken without undue delay to eliminate detected nonconformities and their causes. Follow-up activities shall include the verification of the actions taken and the reporting of verification results (see 8.5.2).

The Internal Audit System Is a Process Metrics Method INPUT PROCESS OUTPUT Infrastructure People

Audit Process/Plan Diagram Purpose: Identify the elements of the standard applicable to and reviewed during this audit Metrics Process Methods Inputs Outputs Infrastructure People Hand Out

DISCUSSION

Discussion - Example Metrics 8.2.3 Monitoring and Measurement of Processes 8.4 Data Analysis Process 8.2.2 Internal Audit Methods 4.2.3 Control of Documents 4.2.4 Control of Records Inputs 4.2.2 Quality Manual Outputs 5.2 Customer Focus 8.5.2 Corrective Action Infrastructure 6.3 Infrastructure 6.4 Work Environment People 6.2 Human Resources 6.2.2 Competence, Training and Awareness Hand Out

Audit Process/Plan Diagram Expectations Metrics Process Methods Inputs Outputs Infrastructure People Hand Out

DISCUSSION

Previous Audit Results Example Metrics Better - Quality On Time- Schedule Within Budget - Cost Process Methods Documented Procedure Inputs Previous Audit Results Company Data Outputs Audit Report Corrective Actions Infrastructure Controlled Databases People Training Certifications Competency Preparation Performance Reporting Follow-up Hand Out

Case Study

Objective Evidence Record Metrics Process Methods Inputs Outputs Infrastructure People Hand Out

DISCUSSION

Effectiveness – Top Five How does the Internal Audit Team Define Effectiveness? AS9101 D Block 11 Organization’s method for determining process effectiveness:

Effectiveness – Top FIVE How does the Internal Audit Team Measure Effectiveness? Metrics Better - Quality On time - Schedule Within budget - Cost Process Methods Well-documented Procedures Inputs Previous Audit Results Company Data Outputs Audit Report Corrective Actions Process Improvement Infrastructure Controlled Databases Inter-related Processes People Training Certifications Competency

Effectiveness – Top FIVE How did the Internal Audit Team Perform? Metrics Better - Quality Faster - Schedule Cheaper - Cost Process Methods Documented Procedure Inputs Previous Audit Results Company Data Outputs Audit Report Corrective Actions C/A Effective? Infrastructure Controlled Databases Inter-related processes People Training Certifications Competent?

Effectiveness – Top FIVE How did the Inputs match up to the Outputs? Metrics Better - Quality Faster - Schedule Cheaper - Cost Process Methods Documented Procedure Inputs Previous Audit Results Company Data Outputs Audit Report Corrective Actions Process Improvement? Infrastructure Controlled Databases People Training Certifications

Effectiveness – Top FIVE Did the Internal Audit Plan For Effectiveness ? Metrics Better - Quality Faster - Schedule Cheaper - Cost Process Methods Documented Procedure Inputs Previous Audit Results Company Data Outputs Audit Report Corrective Actions Infrastructure Controlled Databases People Training Certifications PRE – Planning Preparation Performance Reporting Follow Up

Mirror WE are auditors – time to look in the mirror!! How do you - DEFINE EFFECTIVENESS MEASURE EFFECTIVENESS PERFORM DESCRIBE INPUTS – OUTPUTS PLAN

COMPARE Your Program Their Program

Preparation How do you prepare for audit performance verification Describe inputs and outputs in the planning stages (objectives, scope…) What do you expect to review and the depth to which auditing will be performed How do you plan to gather objective evidence?

Effective Approach From the auditor’s point of view – what would be an effective approach? What records will be evaluated in order to adequately verify the process or system ? Should you verify system documentation in an “inch wide and a mile deep” mode or the inverse “a mile wide and an inch deep”? What would be your approach to generate desirable outcomes ? (Good planning is important.)

Expectations What should be the auditor’s expectation? Did the auditor take the appropriate consideration related to but not limited to company size, system documentation and data gathering, time frame to complete the audit, and auditee controls and diverting tactics ? Did self verification result in process improvement upon effective implementation of C/A? What would be considered an adequate verification of an audit performance process?

Outcome Outcomes and results Do evaluated results meet the audit performance expectation? Is the audit performance an effective one ? Is the auditor satisfied with the results ? What would be considered an adequate verification of an audit performance process ? Was the audit report thorough, citing all pertinent information ?

Corrective Action Follow up How well was C/A written ? Was C/A implemented timely and effectively ? Did it yield adequate improvements to the process or system ? Will implemented changes ultimately make a more efficient QMS system, is it more robust ?

Discussion Points At what point do you audit the internal audit program? Beginning, Middle or End? If the audit program is compliant, but you are writing majors for system breakdowns, how do you handle this? What action do you take if the Internal audit team can not define effectiveness? How would you communicate and/or recommend process/system effectiveness of self verification?