Quality Assurance & Improvement Program: Audit Process Versus Program: The Difference…and Why It Matters Presenter: Brian E. Kruk, CIA, CCSA, CGAP, CCA, CISA Director Contract and Construction Audit Union Pacific Railroad Topeka Chapter April 5, 2016

Today’s Agenda A brief history of QA Discuss the available QA&IP guidance Examine common misconceptions in QA&IP development Explore the differences between basic internal audit processes and effective components of a QA&IP Utilization of the Old IIA PA to create an appropriate, right-sized QA&IP Understand how a CMM can be used to facilitate the path to quality

Today’s Focus -Has anyone recently completed a QA? -Has anyone performed as a Validator? -Is anyone working on their Internal Assessment or Self Assessment? -What do you want out of today’s session? -Are there any questions before we begin?

“Quality is not an act – it is a habit.” ~Aristotle “Quality means doing it right when no one is looking.” ~ Henry Ford

Quality Assessment Defined The process of evaluating the efficiency and effectiveness of an internal auditing organization through a comprehensive, qualitative review of audit procedures, leading to recommendations for improving controls, reducing risk and the introductions of successful innovative best practices. It should also provide assurance conformity with the International Standards for the Professional Practice of Internal Auditing and other relevant organizational and departmental policies and procedures.

QAR 1984

Synopsis of QA History - Other professions have required peer reviews -IIA first publication on QA in IIA recommended peer reviews in previous Standards -IIA began conducting QAs in Some QAs also conducted by other providers -GTF Brings Focus to Quality Initiative -QA Manual, 4 th Edition, released in QA Manual, 5 th Edition, released in QA Manual, 6 th Edition, released in QA Manual, 7 th Edition, released in 2013

Report of GTF to IIA Board of Directors –Adopt New Framework –Revise Definition of IA –Update Code of Ethics and Standards –Establish Oversight Committee –Develop Guidance to Support the Standards A Vision for the Future: Professional Practices for Internal Auditing

Professional Practices Framework OH 2-3 The “Path to Quality” gets its formal start with the creation of: 7 New Quality Standards & 5 Practice Advisories

Continuous Improvement Highlights Onward and Upward

Continuous Improvement Highlights Examples of Shortfalls Addressing the applicability of the Standards for specialty groups Further clarification of Assurance & Consulting services Need for some level of basic fraud (Red Flags) Knowledge of key IT risk, controls and technology-based audit techniques Periodic Internal and External QA and ongoing monitoring as part of QA&IP Inclusion of overall opinion and/or conclusion where appropriate, in final communications

Continuous Improvement Highlights By January changes to the PPF 11 New Standards 13 Additions to Glossary 11 New Practice Advisories 5 Revisions to PA’s

Continuous Improvement Highlights July Arrival of the New International Professional Practice Framework

Continuous Improvement Highlights By the end of changes to the IPPF 6 New Standards 19 New Interpretations 13 Additions to Glossary Practice Advisories reduction to 58 3 New Practice Guides, New 13 GTAG’s New 3 GAIT’s

Continuous Improvement Highlights 2010 to changes to the IPPF 3 New 1 Deleted 15 Revised Standards 9 New and Revised Interpretations 5 Revisions to Glossary 13 New Practice Advisories 8 New Practice Guides, 3 New GTAG’s

Continuous Improvement Highlights

The New IPPF Mandatory Guidance –Core Principles –Standards –DIA –COE Recommended Guidance –Implementation Guidance –Supplemental Guidance (PGs, GTAGs, & GAITs)

IIA - Core Principles Demonstrates integrity. Demonstrates competence and due professional care. Is objective and free from undue influence (independent). Aligns with the strategies, objectives, and risks of the organization. Is appropriately positioned and adequately resourced. Demonstrates quality and continuous improvement. Communicates effectively. Provides risk-based assurance. Is insightful, proactive, and future-focused. Promotes organizational improvement.

Attribute Standards Attribute Standards address the attributes of organizations and individuals performing internal auditing : Purpose, Authority and Responsibility -1100: Independence and Objectivity -1200: Proficiency and Due Professional Care -1300: Quality Assurance and Improvement Program

Performance Standards Performance Standards describe the nature of internal auditing and provide quality criteria against which the performance of these services can be measured : Managing the Internal Audit Activity -2100: Nature of Work -2200: Engagement Planning -2300: Performing the Engagement -2400: Communicating Results -2500: Monitoring Progress -2600: Management’s Acceptance of Risks

QA Related Standards 1300: Quality Assurance and Improvement Programs The chief audit executive should develop and maintain a quality assurance and improvement program that covers all aspects of the IAA and continuously monitors its effectiveness. The program should be designed to help the internal auditing activity add value and improve the organization’s operations and to provide assurance that the internal audit activity is in conformity with the Standards and the Code of Ethics. Note: 2 nd half drop in new Standard: See Interpretation next slide

QA Related Standards Standard 1300 – Interpretation A quality assurance and improvement program is designed to enable an evaluation of the IAA’s conformance with the Standards and an evaluation of whether internal auditors apply the COE. The program also assesses the efficiency and effectiveness of the IAA and identifies opportunities of improvement.

QA Related Standards Original 1310: Quality Program Assessments The internal audit activity should adopt a process to monitor and assess the overall effectiveness of the quality program. The process should include both internal and external assessments. Revised 1310 – Requirements of the QA&IP The QA&IP must include both internal and external assessments.

QA Related Stand ards Original Internal Assessments Should include: Ongoing reviews of the performance of the IAA; and Periodic reviews performed through self-assessment or by other persons within the organization, with knowledge of internal auditing practices and the Standards. Revised Internal Assessments Internal Assessments must include: Ongoing monitoring of the performance of the IAA; and Periodic self-assessment or assessments by other persons within the organization with sufficient knowledge of internal audit practices.

QA Related Standards Internal Assessments Interpretation: Ongoing monitoring is an integral part of the day-to-day supervision, review and measurement of the IAA. Ongoing monitoring incorporated into the routine policies and practices used to manage the IAA and uses processes, tools and information considered necessary to evaluate conformance with the DIA, COE and Standards. Periodic reviews are assessments conducted to evaluate conformance with the DIA, COE and Standards. Sufficient knowledge of IA practices requires at least an understanding of all elements of the IPPF.

QA Related Standards Original 1312: - External Assessments External assessments such as quality assurance reviews, should be conducted at least once every five years by a qualified, independent reviewer or review team from outside the organization.

QA Related Standards Revised 1312: External Assessments External assessments should be conducted at least once every five years by a qualified, independent reviewer or review team from outside the organization. The potential need for more frequent external assessments as well as the qualifications and independence of the external reviewer or review team, including any potential conflict of interest, should be discussed by the CAE with the Board. Such discussions should also consider the size, complexity and industry of the organization in relation to the experience of the reviewer or review team.

QA Related Standards Current 1312 : External Assessments External assessments must be conducted at least once every five years by a qualified independent assessor or assessment team from outside the organization. The CAE must discuss with the board: The form and frequency of external assessment; and The qualifications and independence of the external assessor or assessment team, including any potential conflict of interest.

External Assessments Original Interpretation: A qualified reviewer or review team consists of individuals who are competent in the professional practice of internal auditing and the external assessment process. The evaluation of the competency of the reviewer and review team is a judgment that considers the professional internal audit experience and professional credentials of the individuals selected to perform the review. The evaluation of qualifications also considers the size and complexity of the organizations that the reviewers have been associated with in relation to the organization for which the IAA is being assessed, as well as the need for particular sector, industry, or technical knowledge. An independent reviewer or review team means not having either a real or an apparent conflict of interest and not being a part of, or under the control of, the organization to which the IAA belongs. QA Related Standards

External Assessments Revised Interpretation: A qualified reviewer or review team demonstrates competence in two areas: the professional practice of internal auditing and the external assessment process. Competence can be demonstrated through a mixture of experience and theoretical learning. Experience gained in organizations of similar size, complexity, sector or industry, and technical issues is more valuable than less relevant experience. In the case of a review team, not all members of the team need to have all the competencies; it is the team as a whole that is qualified. The CAE uses professional judgment when assessing whether a reviewer or review team demonstrates sufficient competence to be qualified. An independent reviewer or review team means not having either a real or an apparent conflict of interest and not being a part of, or under the control of, the organization to which the IAA belongs. QA Related Standards

External Assessments Proposed Interpretation: External assessments enhance a complete QA&IP and may be accomplished through a full external assessment, or a self- assessment with independent validation. The external assessor must conclude as to conformance with the Standards; the external assessment may also include operational and strategic comments. 3 rd paragraph adjustments “ real or an apparent ” changed to read “ actual or a perceived” conflict of interest ____* Added 3 rd sentence: The CAE should encourage board participation in the QA&IP to reduce perceived or potential conflicts of interest.

QA Related Standards Original 1320 – Reporting on Quality Program The chief audit executive should communicate the results of external assessments to the board. Revised 1320 – Reporting on Quality Program The CAE must communicate the results of the QA&IP to senior management and the board. Review interpretation narrative

QA Related Standards Reporting on the QA&IP Interpretation: The form, content and frequency of communicating the results of the QA&IP is established through discussions with the senior management and the board and considers the responsibilities of the IAA and CAE as contained in the IA Charter. To demonstrate conformance with the DIA, the COE, and the Standards, the results of external and periodic internal assessments are communicated upon completion of such assessments and the results of ongoing monitoring are communicated at least annually. The results include the assessor’s or assessment team’s evaluation with respect to the degree of conformance.

Original -1330: Use of “Conducted in Accordance with the Standards” Internal auditors are encouraged to report that their activities are “conducted in accordance with the International Standards for the Professional Practice of Internal Auditing.” However, internal auditors may use the statement only if assessments of the quality improvement program demonstrate that the internal audit activity is in compliance with the Standards. Current -1321: Use of “Conforms with the International Standards for the Professional Practice of Internal Auditing” Indicating that the IAA conforms with the ISPPIA is appropriate only if the results of the QA&IP supports such a statement. QA Related Standards

Original 1340: Disclosure of Noncompliance Although the IAA should achieve full compliance with the Standards and internal auditors with the Code of Ethics, there may be instances in which full compliance is not achieved. When noncompliance impacts the overall scope or operation of the IAA, disclosure should be made to senior management and the board. Current – 1322: Disclosure of Nonconformance When nonconformance with DIA, the COE, or the Standards impacts the overall scope or operation of the IAA, the CAE must disclose the nonconformance and the impact to senior management and the board. QA Related Standards

QA Related Practice Advisories Quality Assurance & Improvement Program 1310 – 1 Requirement of the QA&IP (Deleted from IPPF) Internal Assessments Internal Assessment: Establishing Measures to Support Reviews of IAA (Deleted from IPPF) External Assessments External Assessment- SAWIV Independence of External Assessment Team – Private Independence of External Assessment Team – Public Reporting Results of QA&IP Use of “Conforms with the International Standards for the Professional Practice of Internal Auditing” Disclosure of Nonconformance w/ the ISPPIA Managing the Risk of the IAA

QA Related Practice Advisories PA Managing the Risk of the Internal Audit Activity 1.Managing the risk of not achieving IA Objectives 2.IAA must manage its own risk 3.3 categories: audit failure, false assurance, and reputation risks 4.Where were the internal auditors? 5.IAA can implement the practices to mitigate its risk: –QA&IP –Periodic reviews of audit plan –Effective planning –Effective audit design –Effective management review and escalation –Proper Resource Allocation 6.6 through 14 - additional topics of further guidance

External Assessments Areas of focus: -Review IA Activity’s charter, audit plans, policies and procedures -Review a sample of audit reports, special projects and supporting work papers -Review staff composition, supervision, professional development and response to client needs

External Assessments Areas of focus: -Assess staff and client satisfaction through interviews and surveys -Specifically interview audit committee chairperson, a representative sample of officers, senior executives and management clients and the external auditing partner -Risk assessment methodology -Approach and adequacy of IT audit coverage

External Assessment Activities Tools Review Self Study/Benchmarking Customer/Staff Survey On-site Activities –Interviews (Board, Management, External Auditor, Staff) –QA Program –Work Paper Reviews Issue Report

QA - Assessment Objectives -Assess the efficiency and effectiveness of the internal audit activity in light of: - Its charter and mission - Expectations of the board, senior management, audit clients, and the CAE - Identify opportunities and offer ideas and counsel to the CAE and staff for: - Improving their performance - Increasing the value they add to the enterprise - Provide an opinion on the internal audit activity’s conformance to the spirit and intent of the Standards

QA - Assessment Approach - Self Study & Audit Management Questionnaire - Survey of Clients and Staff - Interviews with Senior Managers & Staff - Review Tools (Programs) Organization of the Internal Audit Activity Risk Assessment and Engagement Planning Staff Professional Proficiency Information Technology Production and Value Added Sample of Workpapers and Reports - Rating of Conformity with IIA Standards

QA – Conforming Evaluation Definitions GC – “Generally Conforms” means the assessor has concluded that the Activity’s charter, structure, policies, and procedures, as well as the processes by which they are applied, are judged to be in conformity with a majority of the Standards with some opportunities for improvement being possible. PC – “Partially Conforms” means the assessor has concluded that a good faith effort exist but deviations from conformity for a majority of the Standards exists and corrective action is needed. These deviations are not, however, significant enough to preclude the Activity from carrying out its responsibilities in an acceptable manner. DNC – “Does Not Conform” means the evaluator has concluded that the Activity is not aware of, is not making good-faith efforts to comply with, or is failing to achieve conformity with the majority of the Standards, thus impacting its ability to carry out its mission.

QA Overall Evaluation OVERALL EVALUATION-Generally Conforms (GC) Attribute StandardsGC 1000Purpose, Authority & ResponsibilityGC 1100Independence & ObjectivityGC 1200Proficiency and Due Professional CareGC 1300Quality Assurance and Improvement PC Performance StandardsGC 2000Managing the IA ActivityGC 2100Nature of WorkGC 2200Engagement PlanningGC 2300Performing the EngagementGC 2400Communicating ResultsGC 2500Monitoring ProgressGC 2600Communicating the Acceptance of RiskGC IIA Code of EthicsGC

QA - Potential Issues Reporting Categories Opportunities to Improve Conformity with Standards Opportunities for IA Consideration Suggestions for Senior Management Verbal Comments

QA – Validation Reporting Process Two Options: –Validator signs internally prepared report –Validator prepares separate report referencing internally prepared report

Quality Assessment Process Map (IIA Manual 7 th Edition)

IA Governance (1000,1100,1300, COE, & DIA) IA Staff (1200) IA Management (2000, 2100, & 2600) IA Process (2200, 2300, 2400, & 2500)

5 Minute Break

QA Related Standards - Revisit Original Internal Assessments Should include: Ongoing reviews of the performance of the IAA; and Periodic reviews performed through self-assessment or by other persons within the organization, with knowledge of internal auditing practices and the Standards. Revised Internal Assessments Internal Assessments must include: Ongoing monitoring of the performance of the IAA; and Periodic self-assessment or assessments by other persons within the organization with sufficient knowledge of internal audit practices.

Internal Assessment Exercise #2 (5 minutes) What type of items would you see as part of your on-going program? What type of items would you see as part of your periodic program?

Internal Assessment Internal Assessment Program –Ongoing Performance Reviews of the IA Activity Work Paper Reviews Performance Evaluations Actual vs. Budgeted Analysis Various Monitoring Metrics Customer Surveys –Periodic Reviews Self-Assessment –Annually – Covering all Standards over 5 years –Quarterly/Semi-Annual – Portions of Standards each year –Assess compliance with IA Activity Charter

Internal Assess ment Ongoing Assessments –…routine policies and practices used to manage the IA activity… Engagement supervision Checklists and other means Feedback from IA clients/stakeholders Project budgets, timekeeping systems, audit plan completion, cost recoveries and other performance metrics (e.g. cycle times and recommendations accepted) Conclusions, follow-up, and implementation

Internal Assessment Periodic Assessments (Snapshot In Time) –Non-routine special purpose reviews and testing More in-depth interviews & surveys of stakeholder groups May be performed via self-assessment or by other competent audit professionals within organization May include self-assessments, preparation of materials and benchmarking subsequently reviewed by others Can facilitate & reduce external assessment costs Conclusions, follow-up, and implementation Communicating Results –Share with various appropriate stakeholders

QA&IP Design Individual exercise: Please list 3 components or task performed by your IAA that you feel illustrate your working QA&IP.

QA&IP Design What would an effective QA&IP (performance measurement and reporting process) include ?

Why is QA&IP Important? Reasons for setting up QA&IP – Know where your group stands at all times – Potential external QA cost savings – Reduce risk of external QA “surprises” – Improve the IA environment/process – Reasonable assurance to audit committee – Quality does matter (i.e. Org. initiatives & SOX) – Required by the Standards What reasons do you see out there?

QA&IP Design Program vs. Process Differing Perspectives

QA&IP Design IIA Sample QA&IP

QA&IP Design Deleted - Practice Advisory Establishing the performance measure process The CAE Should: Identifying critical performance categories Identifying performance category strategies & measurement Establish process for measurements to be monitored, analyzed & reported Ensure measures used are appropriate to size & type of IAA

QA&IP Design Identifying critical performance categories Suggested categories: Key stakeholder satisfaction Internal audit processes Innovation Capability

QA&IP Design Key stakeholder satisfaction: Who are the stakeholders? Internal – Audit committee – Executive management – Operating management – Internal audit clients – Audit staff – External – External government bodies and/or regulators External auditors

QA&IP Design Key stakeholder satisfaction: How do you identify stakeholders? Consider the following: Products & services being provided Extent to which organization is regulated Relationship with internal & external parties Nature of the organization (public vs. private)

QA&IP Design Key stakeholder satisfaction: Satisfaction levels must be assessed and gaps identified! – Interviews – Facilitated sessions – Questionnaire Develop appropriate plan for corrective action Execute, monitor & re-evaluate periodically

QA&IP Design Internal audit processes: Risk assessment Annual & long range planning Engagement planning & performance – Proper scope, objectives, timing & resources – Conducted using established methodologies & practices On-going communications Reporting Follow-up Consulting Fraud investigations

QA&IP Design Innovation & capability : Training & competence – Documented training plan by position – Minimum annual training hours – Certification requirements & levels attained Utilization of technology – Staff training goals – Audit staff satisfaction – Data extraction & analysis, automated work papers Industry knowledge – Periodic staff interaction – Employee loan programs – Formalized rotation programs

Supplemental Guidance - Practice Guide Measuring Internal Audit Effectiveness and Efficiency Defining Internal Audit Effectiveness & Efficiency Internal & External Stakeholders Internal Audit Performance Metrics/Measures of Effectiveness & Efficiency Monitoring and Reporting Results

Selected Narrative – Executive Summary: “To maintain and enhance IA credibility, its effectiveness and efficiency must be monitored.” “Identify key performance measures for IA activities that stakeholders believe add value and improve the organization’s operations.” “Effectiveness and efficiency measurements can be quantitative and qualitative.” “Adequacy of engagement planning and supervision.” Supplemental Guidance - Practice Guide Measuring Internal Audit Effectiveness and Efficiency

Selected Narrative – Defining IA Effectiveness & Efficiency: “A general description of E &E is the degree (including quality) to which established objectives are achieved.” IA E&E should be monitored and assessed periodically as part of the IA process.” Selected Narrative – Internal & External Stakeholders: “Specific feedback will provide insight into; understanding of purpose, adequacy, deliverables, expectations, priorities, & shortcomings.” Supplemental Guidance - Practice Guid e Measuring Internal Audit Effectiveness and Efficiency

Selected Narrative – IA Performance Metrics/Measures of E&E: “Identifying critical performance categories such as stakeholder satisfaction, IA processes, and innovation and capabilities.” “Routinely monitoring, analyzing, and reporting performance measures.” Selected Narrative – Monitoring and Reporting: “E&E should be reported to stakeholders periodically.” “ Consistent processes are needed for gathering, summarizing, & analyzing measurement data. “ Supplemental Guidance - Practice Guide Measuring Internal Audit Effectiveness and Efficiency

QA&IP Implementation Implementation should include: Measuring alignment with IIA Standards, key strategic objectives, & applicable laws & regulations Timely gathering, summarizing & analyzing data Ensure measurements kept current & consideration for changing expectations, conditions, priorities & objectives Effective, efficient on-going reporting to stakeholders Annual reporting on IA effectiveness to AC Appropriate internal resourcing Documented methodology Staff involvement & buy-in

QA&IP Should Reveal IAA is: Efficient & effective Structured & staffed appropriately Has an approach that is adequate & meet stakeholder expectations Fully complying with the Standards Utilizes sound testing techniques, methods & technology Considers innovative practices & adopted them, when appropriate

Guiding Concepts Design a program that fits your IAA Utilize available internal resources Treat as a project, start with a detailed plan Promote total team involvement Hold regularly scheduled update meetings Educate all constituencies (IA staff, executive management, & the audit committee) on objectives & progress Make the process as transparent, objective & participatory, as possible Conceptualize on synergies with external QA

Supplemental Guidance – 2 nd PPG Quality Assurance and Improvement Program

Capabilities Maturity Model Example

Performing the Validation -Key Points for Consideration -General considerations -Planning and preparation -Interviews -Self-assessment fieldwork -Self-assessment results, recommendations and implementation plans

Performing the Validation Key Points for Consideration –Perception of lower cost – More time invested by IA Activity –Project timeline controlled by IA Activity –No or limited best practice enhancements –Less independent as much of the work is done by the IA Activity –Key Point-Validator should be qualified –Interview and survey limitations

Performing the Validation Overview and details: -General considerations -Planning and preparation -Interviews -Self-assessment fieldwork -Self-assessment results, recommendations and implementation plans

Performing the Validation General considerations: -Alternative means for complying with Standard 1312 external assessments -Benefits -Economics/Practicality -Expand external assessments to more IA activities

Performing the Validation General considerations: -Scope Limitations -Scope more targeted/limited than full external assessment -Focused on basic IA expectations -Fulfillment of IA mission -Conformance to the Standards -Areas where in-depth analyses may be curtailed or excluded

Performing the Validation Planning and preparation: -Designate project leader and team -Select external independent validator -Agree on scope and responsibilities -Prepare self study -Consider/ Conduct client surveys -Select audit/consulting engagements for review -Select interview candidates for team and validator

Performing the Validation Interviews: -Audit Committee Chair -Executive to whom the CAE Reports -Senior and Operating Manager -CAE -IA Staff -External Auditor

Performing the Validation Fieldwork: -Departmental structure and organization -Risk assessment and engagement planning -Staffing skills and experience -IT review -Assessing productions and value added -Individual W/P file review Utilize Same Concepts (Tools) as External Summarized QAS Tool 12.doc Summarized QAS Tool 13.doc Summarized QAS Tool 14.doc Summarized QAS Tool 15.doc Summarized QAS Tool 16.doc Summarized QAS Tool 17.doc

Performing the Validation Results, recommendations & implementation plans -Major results/findings with emphasis on: -Opportunities for process improvement -Enhancing customer relations -Evaluation summary -Conclusion on conformity to the Standards

Performing the Validation - Recap Validation Process: -Independent validation of the self-assessment -Advance Prep review -AMQ review -Report review -On-site review - Documentation of self-assessment - Limited testing - Evaluation summary - Draft report/communication -Interviews -Memorandum/Closing conference/report

REMEMBER!!!!!! “You manage what you measure.” ~Brian E, Kruk

Questions on QA&IP?

Thanks for your participation! Brian E. Kruk, CIA, CCSA, CGAP, CCA, CISA Director Contracts and Construction Union Pacific Railroad