1. Quality Assurance and Improvement Program
Quality Assurance and Improvement Program
October 2015

2. Learning Objectives
Understanding Quality Assurance Review (QAR) Practices Review of current standards and expectations for quality assurance and improvement program Leading practices and approaches for quality assurance and improvement

3. Understanding Quality Assurance Review Practices The Standards
The International Standards for the Professional Practice of Internal Auditing (ISPPA) represent principle focused standards intended to provide a framework for performing and promoting internal auditing. Standard 1312 – External Assessments must be conducted once every five years by a qualified, independent assessor or assessment team from outside the organization.
Most internal audit departments view IIA standards as mandatory.
Many IA functions use the ISPPA standards as their framework for executing their IA departments and communicate to their Audit Committee and other stakeholders that they follow these standards.

4. Understanding Quality Assurance Review Practices The Standards
IIA Standards Internal Audit departments are assessed against 11 Standards developed by the IIA. Four standards ( ) address the attributes of Internal Audit (i.e., who or what internal audit is); seven standards ( ) address the performance of Internal Audit (i.e., how internal audit conducts its work).
Standard Number
Summary of IIA Standards
1000
Purpose, authority, and responsibility
1100
Independence and objectivity
1200
Proficiency and due professional care
1300
Quality assurance & improvement program
2000
Managing the internal audit activity
2100
Nature of work
2200
Engagement planning
2300
Performing the engagement
2400
Communicating results
2500
Monitoring progress
2600
Communicating the acceptance of risk
The Standards are viewed by the IIA as fundamental attributes for an internal audit function (i.e., they represent the minimum acceptable level of performance).
1000
The purpose, authority, and responsibility of the internal audit activity must be formally defined in an internal audit charter, consistent with the Definition of Internal Auditing, the Code of Ethics, and the Standards. The chief audit executive must periodically review the internal audit charter and present it to senior management and the board for approval.
1100
The internal audit activity must be independent, and internal auditors must be objective in performing their work.
1200
Engagements must be performed with proficiency and due professional care.
1300
The chief audit executive must develop and maintain a quality assurance and improvement program that covers all aspects of the internal audit activity.
2000
The chief audit executive must effectively manage the internal audit activity to ensure it adds value to the organization.
2100
The internal audit activity must evaluate and contribute to the improvement of governance, risk management, and control processes using a systematic and disciplined approach.
2200
Internal auditors must develop and document a plan for each engagement, including the engagement’s objectives, scope, timing and resource allocations.
2300
Internal auditors must identify, analyze, evaluate, and document sufficient information to achieve the engagement’s objectives.
2400
Internal auditors must communicate the engagement results.
2500
The chief audit executive must establish and maintain a system to monitor the disposition of results communicated to management.
2600
When the chief audit executive believes that senior management has accepted a level of residual risk that may be unacceptable to the organization, the chief audit executive must discuss the matter with senior management. If the decision regarding residual risk is not resolved, the chief audit executive must report the matter to the board for resolution.
Each standard area should be reviewed to determine where current performance does or does not meet the Standards. Conformance with both the spirit and letter of the Standard should be considered. The assessment should conclude for each standard area with one of the following ratings:
Generally conforms – the internal audit activity has policies, processes and practices that are in accordance with the Standards. Opportunities for enhancements may exist.
Partially conforms – deviations from the Standards exist, but did not preclude the internal audit activity from performing its responsibilities in an acceptable manner.
Does not conform – deficiencies in practice are so significant as to seriously impair or preclude the internal audit activity from performing adequately in all or in significant areas of its responsibilities.

5. Understanding Quality Assurance Review Practices Types of External Strategic Assessments (ESA) services
Companies typically to perform an ESA for a variety of reasons, ranging from developing a strategic plan to benchmarking to complying with the IIA standards. We can break down ESAs into two types: • Type 1: Full ESA – This assessment provides the greatest value to companies as it assesses 1) stakeholder expectations and opinions on Internal Audit’s current performance and compares those opinions against Internal Audit’s current operating practices; 2) Internal Audit’s operating practices against peer results; and 3) Internal Audit’s operating practices against the IIA standards. • Type 2: IIA Standards Assessment – This is a subset of the full ESA, with more limited insight as it evaluates only whether Internal Audit operating practices conform with the IIA standards and how the departments operating practices compare against peers.
Note - Type 2 engagements do not typically result in a high level of value for a Company as they are less focused on determining whether the internal audit or other quality assurance program is aligned with the expectations of key stakeholders than a Type 1.

6. External Strategic Assessments Types of ESA services (continued)
The table below provides a summary of the objectives, deliverables and value for each type of service:
Type 1: ESA
Type 2: IIA Standards Assessment
Objective
Assess Internal Audit for the following:
Stakeholder expectations and perception of IA’s performance against the eight attributes of excellence 
Maturity of IA operating practices against the eight attributes of excellence
IA operating practices against peer company operating practices
Conformance to IIA Standards
Insights Obtained
The following information is summarized to gain insight into the IA department:
Results of stakeholder assessment of Internal Audit (i.e., the stakeholder’s expectations vs their perception of performance)
Comparison of IA’s operating practice results against 1) stakeholder expectations; and 2) stakeholder perception of IA’s performance
Results of operating practices for each of the 8 attributes and overall
Benchmarking of operating practice results against peers
Conformance to IIA Standards and actions warranted to achieve conformance, as needed
Value Delivered
Strategic assessments allow departments to assess the value they deliver:
Insight into where Internal Audit is not meeting the expectations of their stakeholders
Insight into whether that misalignment is a result of under-performing teams or a need to enhance existing operating practices
Understanding of IA’s operating capabilities compared against peers
Roadmap of actions warranted to achieve conformance with the IIA Standards
Achievement of requirements for IIA Standard 1312
External Assessments External assessments must be conducted at least once every five years by a qualified, independent assessor or assessment team from outside the organization. The chief audit executive must discuss with the board:  The form and frequency of external assessment; and  The qualifications and independence of the external assessor or assessment team, including any
Typical Calendar Timeframe & level of PwC effort
8-12 week duration
(from kick off to report)
4-6 week duration
hours (highly dependent on number of interviews and complexity of environment
hours(dependent on size of department and number of interviews)
potential conflict of interest.

7. External Strategic Assessments Overview
The primary internal audit performance improvement service offered by PwC is an External Strategic Assessment (ESA), performed using a proprietary approach and technology known as Profiler™. Companies typically may require such a service if they desire a perspective on how their internal audit group is performing relative to leading practices and/or professional standards, or at the onset of developing a Strategic Plan. Areas to be reviewed may encompass the entire spectrum of internal audit strategy and operations or be very specific to a certain area. A full external strategic assessment consists of:
Understanding internal audit stakeholders’ perspectives of internal audit’s performance and value. Stakeholders typically include: Audit Committee &/or Board members, Executives and Senior leadership, other risk and compliance leaders, internal audit staff and external auditors;
Evaluating internal audit working practices, including evaluation of select audits, to understand the maturity of the department’s current operating capabilities;
An assessment of conformance against each of the 11 Standards within the Institute of Internal Auditors' ("IIA") International Standards for the Professional Practice of Internal Auditing ("IIA Standards" or "the Standards"); and
Benchmarking of internal audit working practices against peer companies from Profiler™.
Stakeholder Value
Stakeholder Expectations & Alignment
Performance
Operational
Capability
Compliance with IIA standards

8. External Strategic Assessments (continued) The ESA framework
Our ESA framework is built off of the Internal Audit Maturity scale across the internal audit Eight Attributes of Excellence. This means that we assess Internal Audit’s operating practices as well as stakeholder expectations and opinion of Internal Audit’s performance against each of the Eight Attributes of Excellence. The Maturity scale and Eight Attributes of Excellence are detailed below.
PwC’s Maturity Model
Providing value-added services and proactive strategic advice to the business well beyond the effectiveness and efficient execution of the audit plan
Bringing analysis and perspective on root causes of issues identified in audit findings, to help business units take corrective action
Trusted Advisor
Insight Generator
Through research conducted for PwC’s 2014 State of the Internal Audit Profession Study, we found that Internal Audit functions provide value across a spectrum of delivery approaches – from assurance provider to trusted advisor. Those functions that add significant value are purposefully deciding where they want to fall on this spectrum and aligning capabilities accordingly. Those functions that define themselves as a trusted advisor typically have a much broader mandate, tend to be more aligned with stakeholder expectations, have higher performance and are seen as adding more value to the organization.
Problem Solver
Business Value
Taking a more proactive role in suggesting meaningful improvements and providing assurance around risk
Assurance Provider
Problem Finder
Delivering objective assurance on the effectiveness of an organization’s internal control
Minimum Contributor
Immature
Core
Maturity of Internal Audit Practice

9. External Strategic Assessments The ESA framework (continued)
PwC’s Eight Attributes of Excellence
Focuses on the development of quality standards, performance of formal reviews against quality standards and promotion of a culture that supports and rewards innovation and improvement
Focuses on Internal Audit’s strategic planning, communication of expections and the measurement of progress towards the stated mission and vision of the department
Internal audit
Business alignment
Risk focus
Service Culture
Technology
Talent Model
Stakeholder management
Cost effectiveness
Quality and innovation
Protect the business
Focuses on providing professional services to their stakeholders throughout the organization in a flexible, responsive, and professional manner
Focuses on the design of a dynamic audit plan which addresses both strategic and risk-based approach
Deliver measurable value
Focuses on Internal Audit’s use of technology to assist in identifying risks and business issues and to generate efficiencies within the business and audit process
Focuses on the approximate mix of core internal audit and subject matter specialists to meet required expectations. This model includes the incorporation of performance feedback for staff and department to facilitate growth and development
During the study, we noted that each of these eight key attributes was shared by high-performing internal audit functions, regardless of their scope of work. We believe these eight attributes are still as equally critical for high-performing Internal Audit departments today.
Focuses on the efficient delivery of internal audit services through use of staffing models, productivity analysis, audit process and audit infrastructure
Focuses on Internal Audit’s management of both internal and stakeholder relationships including stakeholder expectations, communication strategies, delivery of value and incorporation of feed back

10. Expectations for a Quality Assurance Program Review Engagement Overview
The ESA and IIA Standards Assessment are typically performed in three phases of work depicted in the picture below.
Project planning
Data collection
Analysis & reporting

11. Expectations for a Quality Assurance Program Review Engagement Overview
Review internal audit operating practices, documentation and tools The internal audit operating practices review will assess various components of the Internal Audit function, spanning across the Eight Attributes of Excellence, to determine what foundational components are in place to assist Internal Audit in effective operations. The assessment includes but is not limited to a review of Internal Audit’s charter, a selection of work papers and audit reports, communications with stakeholders, etc. This portion of a Strategic Assessment is similar to other audit procedures in that there is a client request list, workprogram and meetings with appropriate individuals to gain evidence on each topic and determine conformance with the IIA standards.
Possibly delete

12. Expectations for a Quality Assurance Program Review Engagement Overview
Review internal audit operating practices, documentation and tools (continued) When assessing conformance with the IIA Standards, it is important to note that the Standards also address 'implementation standards' which provide further clarification of the 11 IIA Standards at a more granular level. Companies should not only be assessed by the 11 IIA Standards but also the implementation standards included in the International Standards for the Professional Practice of Internal Auditing. This will result in an assessment of an Internal Audit department's operating capabilities as well as conformance with the IIA Standards.

13. Expectations for a Quality Assurance Program Review Project planning
Understand the environment
Various factors need to be considered when gaining an understanding of the Internal Audit department and the overall environment of the company.
Key stakeholders - consider Audit Committee members and executive leadership's possible perceptions and past experiences with internal audit as well as expectations of internal audit that have already been articulated.
Enterprise strategies and risks - review analyst reports and the CEO's letter in the latest Annual Report to understand the company's current position, three to five year strategy, and potential changes to major risks identified by Internal Audit or other Risk Management functions.
Industry and regulatory issues - consider industry and regulatory changes that may impact the company's risk environment.
Internal audit cost and size benchmarks - look for significant under or overspend based on relevant data from the IIA's GAIN benchmarking reports.
Internal audit trends - consider recent and planned developments and trends within the profession.

14. Expectations for a Quality Assurance Program Review Data Collection
Understand the environment (continued)
Strong IA departments have the following:
Internal Audit Charter: Internal Audit’s charter to better understand the mission of Internal Audit and further assess components of the charter that are required within the IIA Standards.
Risk Assessment: Teams should obtain evidence of Internal Audit’s risk assessment process as well as the steps taken to execute risk assessment(s) during the period under review.
Final deliverables provided for a sample selection of audits: Upon selecting a sample of Internal Audit projects during the testing period, teams should request final deliverables and issues reported to auditees to better understand the reporting and wrap up stages of Internal Audit engagements.
Audit Methodology: Teams should obtain Internal Audit’s methodology and other policies and procedures and should take steps to better understand how these are maintained and communicated to relevant Internal Audit practitioners.
Not only did you do it and does it exist, but evaluate how they did it.

15. Expectations for a Quality Assurance Program Review Data Collection (continued)
Conduct stakeholder interviews and complete electronic survey Interviews are the recommended technique for capturing and understanding the needs and expectations of key internal audit stakeholders. A typical engagement will likely require between 10 and 25 interviews of board members and executives, depending on the size and scope of internal audit activities and the stakeholder group.

16. Expectations for a Quality Assurance Program Review Data Collection (continued)
Conduct stakeholder interviews and complete electronic survey (continued)
Companies also have the option to send an electronic survey directly to stakeholders. Stakeholders can typically be grouped into two or three categories:
Top executives (C-suite, Audit Committee, CAE, etc.): These stakeholders can be interviewed only or they can answer an electronic stakeholder survey and then participate in an interview to discuss specific answers and comments from the survey.
Other stakeholders (Internal audit staff, compliance, other key mid-level finance or operations management): These stakeholders can generally follow a similar method for gaining knowledge as top executives, however more reliance on the electronic stakeholder survey to obtain input could allow organizations to reach a broader group of stakeholders.
To execute the electronic survey, stakeholders are granted access to the Profiler™ Stakeholder survey and are asked to assess Internal Audit’s current performance and to also provide their expectations of Internal Audit. To provide access to the Stakeholder Survey for a listing of Stakeholders, refer to the ‘Set Up Client in Profiler™’ task above.

17. Expectations for a Quality Assurance Program Review Data Collection (continued)
Conduct stakeholder interviews and complete electronic survey (continued) Stakeholder expectations By assessing Internal Audit operating practices against stakeholder expectations, engagement teams are able to identify where stakeholder expectations are not being met. Additionally, teams are able to distinguish whether misalignment is due to under- performance or under-developed operating practices. Typically, the greatest degree of misalignment is caused by under-developed operating practices. By enhancing operating practices, Internal Audit should also see an increase in the level of performance identified by stakeholders.

18. Expectations for a Quality Assurance Program Review Data Collection (continued)
Peer benchmarking A Peer Benchmark can be a valuable tool for assessing a Company’s quality assurance and improvement program by evaluating benchmark scores specific to operating capabilities of peers. While this information does not help to achieve alignment of stakeholder expectations with Internal Audit’s performance and operating capabilities, some company’s find comparative data against peer companies to be insightful. Additionally, recommendations based on quantitative benchmarking data from IIA GAIN reports may be provided.

19. Expectations for a Quality Assurance Program Review Analysis & Reporting
Develop final deliverable
Depending on the initial scoping and assessment level chosen, the content of the report may differ.
The final report may include the following:
Executive summary
Stakeholder expectations/voice of the stakeholder
Areas or processes that do not align with stakeholder expectations and areas or processes where incorporation of enhanced practices addressed in the Eight Attributes of Excellence will result in improve performance on the Internal Audit Maturity Scale
Results of the IIA Standards assessment
Profiler™ best practice analysis with recommended actionable solutions

20. Expectations for a Quality Assurance Program Review Analysis & Reporting
Each standard area should be reviewed to determine where current performance does or does not meet the Standards. Conformance with both the spirit and letter of the Standard should be considered. The assessment should conclude for each standard area with one of the following ratings:
Generally conforms – the internal audit activity has policies, processes and practices that are in accordance with the Standards. Opportunities for enhancements may exist.
Partially conforms – deviations from the Standards exist, but did not preclude the internal audit activity from performing its responsibilities in an acceptable manner.
Does not conform – deficiencies in practice are so significant as to seriously impair or preclude the internal audit activity from performing adequately in all or in significant areas of its responsibilities.

21. Leading Practices and Approaches of High Performing Quality Assurance & Improvement Programs
Common Pitfalls
In delivering numerous engagements, some common themes have emerged:
Lack of documented and supported strategic direction and supporting initiatives
Department charter/activities are misaligned with stakeholder expectations
Inadequate sponsorship of the internal audit department
Department structural issues, e.g. reporting lines
Department structure not aligned with the business both in terms of skill set and geographic coverage
Inadequate risk assessment process and alignment with company Risk Management activities
Poor linkage between risk assessment and audit plan
Too little input to the risk assessment from departments outside of internal audit
Issues identified not aligned with the high risk areas of the company
Lack of use of technology for workpapers, data analysis, and knowledge management
Ineffective communication/reporting
Consider adding a slide talking about value provided to the board and audit committee, value to the internal audit function, value to management/other organizational stakeholders.

22. © 2015 PwC. All rights reserved
© 2015 PwC. All rights reserved. PwC refers to the US member firm or one of its subsidiaries or affiliates, and may sometimes refer to the PwC network. Each member firm is a separate legal entity. Please see for further details.