1. Determining Effectiveness of Internal Audits Presented by: Tony Gutierrez R. Darrell Taylor
Minneapolis, MN
July 19 & 20, 2012

2. To discuss how an auditor audits the Internal Audit Process
Purpose
To discuss how an auditor audits the
Internal Audit Process
and determine how
effective
is the process

3. Definition
Effective - producing a decided, decisive, or desired effect.
(Merriam- Webster)
In addition to establishing conformance with a set of rules, quality audits may measure the adequacy of procedures and the effectiveness of implementation to which those rules assist in achieving basic goals.
(JP Russell, The Quality Audit Handbook)

4. Process Based Auditing
History Lesson
Metrics
Method
Infrastructure
People
Input
Output
Process
Process Based Auditing
Risk Based
Auditing
Risk
Issues
Effectiveness
Value Added
Document Control
Record Control
Purchasing
Production
Calibration
Internal Audit
Corrective Action
Compliance Auditing
11/22/201811/22/2018

5. The Requirement - AS9100C 8.2.2 Internal Audit
The organization shall conduct internal audits at planned intervals to determine whether the quality management system
a) conforms to the planned arrangements (see 7.1), to the requirements of this International Standard and to the quality management system requirements established by the organization, and
NOTE: Planned arrangements include customer contractual requirements.

6. The Requirement AS9100C (cont.)
b) is effectively implemented and maintained.
An audit program shall be planned, taking into consideration the status and importance of the processes and areas to be audited, as well as the results of previous audits. The audit criteria, scope, frequency and methods shall be defined.
The selection of auditors and conduct of audits shall ensure objectivity and impartiality of the audit process. Auditors shall not audit their own work.

7. The Requirement AS9100C (cont.)
A documented procedure shall be established to define the responsibilities and requirements for planning and conducting audits, establishing records and reporting results.
Records of the audits and their results shall be maintained (see 4.2.4).

8. The Requirement AS9100C (cont.)
The management responsible for the area being audited shall ensure that any necessary corrections and corrective actions are taken without undue delay to eliminate detected nonconformities and their causes.
Follow-up activities shall
include the verification of the actions taken and the reporting of verification results (see ).

9. The Internal Audit System Is a Process
Metrics
Method
INPUT
PROCESS
OUTPUT
Infrastructure
People

10. Audit Process/Plan Diagram Purpose: Identify the elements of the standard applicable to and reviewed during this audit
Metrics
Process
Methods
Inputs
Outputs
Infrastructure
People
Hand Out

11. DISCUSSION

12. Discussion - Example Metrics
8.2.3 Monitoring and Measurement of Processes
8.4 Data Analysis
Process
8.2.2 Internal Audit
Methods
4.2.3 Control of Documents
4.2.4 Control of Records
Inputs
4.2.2 Quality Manual
Outputs
5.2 Customer Focus
8.5.2 Corrective Action
Infrastructure
6.3 Infrastructure
6.4 Work Environment
People
6.2 Human Resources
6.2.2 Competence, Training and Awareness
Hand Out

13. Audit Process/Plan Diagram Expectations
Metrics
Process
Methods
Inputs
Outputs
Infrastructure
People
Hand Out

14. DISCUSSION

15. Previous Audit Results
Example
Metrics
Better - Quality
On Time- Schedule
Within Budget - Cost
Process
Methods
Documented Procedure
Inputs
Previous Audit Results
Company Data
Outputs
Audit Report
Corrective Actions
Infrastructure
Controlled Databases
People
Training
Certifications
Competency
Preparation
Performance
Reporting
Follow-up
Hand Out

16. Case Study

17. Objective Evidence Record
Metrics
Process
Methods
Inputs
Outputs
Infrastructure
People
Hand Out

18. DISCUSSION

19. Effectiveness – Top Five
How does the Internal Audit Team
Define
Effectiveness?
AS9101 D Block 11
Organization’s method for determining process effectiveness:

20. Effectiveness – Top FIVE
How does the Internal Audit Team Measure Effectiveness?
Metrics
Better - Quality
On time - Schedule
Within budget - Cost
Process
Methods
Well-documented Procedures
Inputs
Previous Audit Results
Company Data
Outputs
Audit Report
Corrective Actions
Process Improvement
Infrastructure
Controlled Databases
Inter-related Processes
People
Training
Certifications
Competency

21. Effectiveness – Top FIVE
How did the Internal Audit Team Perform?
Metrics
Better - Quality
Faster - Schedule
Cheaper - Cost
Process
Methods
Documented Procedure
Inputs
Previous Audit Results
Company Data
Outputs
Audit Report
Corrective Actions
C/A Effective?
Infrastructure
Controlled Databases
Inter-related processes
People
Training
Certifications
Competent?

22. Effectiveness – Top FIVE
How did the
Inputs match up to the Outputs?
Metrics
Better - Quality
Faster - Schedule
Cheaper - Cost
Process
Methods
Documented Procedure
Inputs
Previous Audit Results
Company Data
Outputs
Audit Report
Corrective Actions
Process Improvement?
Infrastructure
Controlled Databases
People
Training
Certifications

23. Effectiveness – Top FIVE
Did the Internal Audit
Plan
For Effectiveness ?
Metrics
Better - Quality
Faster - Schedule
Cheaper - Cost
Process
Methods
Documented Procedure
Inputs
Previous Audit Results
Company Data
Outputs
Audit Report
Corrective Actions
Infrastructure
Controlled Databases
People
Training
Certifications
PRE – Planning
Preparation
Performance
Reporting
Follow Up

24. Mirror WE are auditors – time to look in the mirror!! How do you -
DEFINE EFFECTIVENESS
MEASURE EFFECTIVENESS
PERFORM
DESCRIBE INPUTS – OUTPUTS
PLAN

25. COMPARE
Your Program
Their Program

26. Preparation How do you prepare for audit performance verification
Describe inputs and outputs in the planning stages (objectives, scope…)
What do you expect to review and the depth to which auditing will be performed
How do you plan to gather objective evidence?

27. Effective Approach
From the auditor’s point of view – what would be an effective approach?
What records will be evaluated in order to adequately verify the process or system ?
Should you verify system documentation in an “inch wide and a mile deep” mode or the inverse “a mile wide and an inch deep”?
What would be your approach to generate desirable outcomes ? (Good planning is important.)

28. Expectations What should be the auditor’s expectation?
Did the auditor take the appropriate consideration related to but not limited to company size, system documentation and data gathering, time frame to complete the audit, and auditee controls and diverting tactics ?
Did self verification result in process improvement upon effective implementation of C/A?
What would be considered an adequate verification of an audit performance process?

29. Outcome Outcomes and results
Do evaluated results meet the audit performance expectation?
Is the audit performance an effective one ?
Is the auditor satisfied with the results ?
What would be considered an adequate verification of an audit performance process ?
Was the audit report thorough, citing all pertinent information ?

30. Corrective Action Follow up How well was C/A written ?
Was C/A implemented timely and effectively ?
Did it yield adequate improvements to the process or system ?
Will implemented changes ultimately make a more efficient QMS system, is it more robust ?

31. Discussion Points
At what point do you audit the internal audit program?
Beginning, Middle or End?
If the audit program is compliant, but you are writing majors for system breakdowns, how do you handle this?
What action do you take if the Internal audit team can not define effectiveness?
How would you communicate and/or recommend process/system effectiveness of self verification?