SAI Performance Measurement Framework Presented by: Date: INTOSAI Working Group on the Value and Benefits of SAIs

Presentation Outline Introduction Background and rationale for developing a SAI Performance Measurement Framework (PMF) Responsibilities and reporting lines Indicative content and design Looking ahead Further information

1. Introduction

What is a Performance Measurement Framework? Tool for measuring performance …of an individual, system or organization …against an agreed performance scale …accommodating local circumstances To enable Better management of performance Monitoring over time Evaluation and lesson learning Contributing to Understanding performance Improving performance Not everything that matters can be measured

What is the SAI Performance Measurement Framework (PMF)? A Performance Measurement Framework tailored to SAIs Intended to give a high level assessment of SAI performance against established INTOSAI good practice: The International Standards of Supreme Audit Institutions (ISSAI) framework The Framework on the Value and Benefits of SAIs Other INTOSAI guidance material Combination of objective measurement and qualitative assessment Analyzes 7 domains of SAI performance in the country context and aims at assessing the value and benefits of the SAI to society (impact)

SAI PMF – How will it be used in Practice? Using the SAI PMF will be voluntary for the SAIs – the tool will be a public good which SAIs can make use of if it helps them achieve their objectives The SAIs themselves will make the key decisions regarding the assessment The SAI will also decide whether or not to publish results Because it is voluntary, the SAI PMF is not intended to replace any internal performance measurement procedures Who decides to initiate a SAI PMF? Head of SAI Who decides when to do a SAI PMF? Head of SAI Consulting with INTOSAI regions & donors as appropriate Every 3-5 years recommended Head of SAI Approach selected to meet the SAI’s objectives 3 broad approaches Quality assurance to promote objectivity and credibility Who decides how to do a SAI PMF? Head of SAI Who decides if, when and how to publish a SAI PMF?

3 Approaches to Conducting a SAI PMF Assessment There are three main approaches to conducting an assessment using the SAI PMF. The PMF will be equally well suited for all three: 1. SAI Self Assessment Trained Team from within the SAI 2. INTOSAI Peer Review Peer SAI INTOSAI Regional Body 3. External Assessment Professional PFM Body Donor Agency Consultant External auditor . SAI Involvement The approach will not necessarily affect how the assessment is carried out… … but the SAI may use the results for different purposes depending on the approach

Background and rationale for developing a SAI Performance Measurement Framework (PMF)

INTOSAI Framework on the Value and Benefits of SAIs adopted 2010 Background Full set of International Standards of Supreme Audit Institutions (ISSAIs) adopted by INTOSAI in 2010 INTOSAI Framework on the Value and Benefits of SAIs adopted 2010 Increased focus on use of country public financial management systems within the international donor community

Mandate: Johannesburg Accords (INCOSAI XX, 2010) Responsibility Working Group on the Value and Benefits of SAI Task Develop further credible measurement tools to support evaluations in terms of the framework for Communicating and Promoting the Value and Benefits of SAIs Collaborating with INTOSAI Capacity Building Committee INTOSAI-Donor Steering Committee Cognizance of Tools that are already available at the level of… international organizations Timetable For consideration at XXI INCOSAI in 2013

SAI PMF Stakeholders and Purpose Rationale for INTOSAI to Develop a SAI Performance Measurement Framework SAI PMF Stakeholders and Purpose Benefit to the SAI Improve SAI approaches to capacity development through use of performance management Identify opportunities to strengthen SAI performance Supreme Audit Institutions Promote transparency and accountability through credible public reporting of SAI performance (ISSAI 20) Domestic Parliaments and Citizens Enable better peer learning INTOSAI Regions and INTOSAI Global Community Demonstrate SAI performance changes to key stakeholders International Development Community (Donors and SAI Providers of Support) Stronger SAI-led capacity development programs SAIs become model learning organizations SAI Enhanced SAI integrity and credibility with domestic stakeholders Evidence-based policies and better targeted global and regional programs Maintain external support for SAI capacity development

Why should SAIs use the PMF (1) As a step towards ISSAI implementation Increased understanding in the INTOSAI community of what good practice for SAIs entails Individual SAIs can benchmark their performance against good practice – and get an indication of where there is room for improvement They can use the assessment as a basis for more detailed ISSAI compliance assessments To demonstrate progress and value and benefits to society The framework will be well suited for measuring progress over time and demonstrating this to external stakeholders Show to stakeholders how the SAI contributes to strengthening public financial management

Why should SAIs use the PMF (2) Internal performance measurement / annual reporting Help SAIs better manage their own performance ISSAI requirement to assess and report on own performance – lead by example and become a model organization Inform and secure support for capacity development initiatives Identify strengths and weaknesses, increase understanding of reasons for good and weak performance Help build a culture for performance improvement SAIs can use assessment results to get support from donors and other external stakeholders

The Role of SAI PMF Assessments in SAI Capacity Development Implement SAI Development Action Plan High Level Performance Overview Progress Monitoring Identify Main SAI Strengths and Weaknesses Investigate Underlying Causes Recommend SAI Capacity Building Measures Formulate or Update SAI Development Action Plan (Medium Term) Formulate or Update SAI Strategic Plan (Long Term) SAI PMF Assessment SAI Capacity Building Needs Assessment

Responsibilities and reporting lines

Responsibilities and Reporting Lines INTOSAI Congress (2013) Approval INTOSAI Working Group on the Value and Benefits of SAIs Oversight and Direction INTOSAI Capacity Building Committee INTOSAI-Donor Steering Committee Other Relevant INTOSAI Committees In Consultation with: WGVBS SAI PMF Task Team Development of SAI Performance Measurement Framework Coordinator: INTOSAI-Donor Secretariat @ IDI SAI PMF Reference Group Technical Advice, Review and Quality Assurance INTOSAI Capacity Building Committee Members INTOSAI-Donor Steering Committee Members Other Groups: PEFA Secretariat, International Budget Partnership INTOSAI Capacity Building Committee Members INTOSAI-Donor Steering Committee Members

Task Team Reference Group INTOSAI Regional Organizations: African Organization of English Speaking SAIs(AFROSAI-E) African Organization of Francophone SAIs (CREFIAF) SAIs: Bahrain Brazil Chile Germany India Mexico Russia United Kingdom Development Agencies: Inter American Development Bank Coordinator: INTOSAI-Donor Secretariat @ IDI Reference Group INTOSAI Regional Organizations: Pacific Association of SAIs (PASAI) SAIs: Canada Denmark Estonia European Court of Auditors Japan Netherlands Norway Peru Sweden Development Agencies: African Development Bank OECD European Commission World Bank Civil Society Organizations: International Budget Partnership Global Reporting Initiative (GRI) PEFA Secretariat CREFIAF = Le Conseil Régional de Formation des Institutions Supérieures de Contrôle des Finances Publiques d'Afrique Francophone Subsaharienne

Indicative content and design

INTOSAI Working Group on the Value and Benefits of SAIs: Criteria For an Effective SAI PMF Comprehensiveness: Broad coverage of all key aspects of an SAIs work and its contribution to good governance, accountability and transparency Objectivity: Indicators to measure performance and progress are objective Subjectivity: Subjective factors can also be captured, e.g. through a narrative performance report International agreement: performance indicators and measurement scales reflect agreed international standards and good practices where available Relevance: Relevant to all SAIs, regardless of their administrative heritage or level of development Performance: Enables analysis of SAI performance and identification of development challenges and opportunities Progress: Facilitates the consistent measurement of SAI performance over time Consistency: Consistent with pre-existing tools (both SAI coverage within broad public financial management measurement frameworks, and SAI specific performance measurement tools) Compliance: Measures actual audit practices, as well as the quality of the SAI’s legislative / regulatory framework and internal guidelines / manuals Brevity: Comprises the minimum number of performance indicators possible to cover key aspects of an SAI’s performance, so as make the tool of practical benefit to SAIs User-Friendly: facilitates easy use by SAIs as self assessment or peer assessment tool Endorsed at the 4th meeting of the Working Group on the Value and Benefits of SAIs in Jamaica, August 2011.

SAI PMF Task Team: Initial Work and Decisions No existing framework fully meets INTOSAI needs Develop a new tool – drawing heavily on existing ones (AFROSAI-E, PEFA, INTOSAI Peer Review Guide, IDI Capacity Building Needs Assessment Handbook) Mapping of 20 Existing SAI Assessment Tools ISSAI Framework where possible – recognizing it will continue to evolve over time The SAI’s mandate INTOSAI guides & other generally accepted international standards or good practices What to Measure Performance Against? External performance – delivery of core audit activities (audit coverage, submission of audit reports, etc.) Internal enablers of performance – SAI strategy, audit processes, SAI support services SAI’s operating environment (i.e. independence and legal framework) – recognizing this is largely beyond the control of the SAI Reported in the context of the specific country governance and public financial management environment and its impact on the SAI’s performance HOLISTIC MEASUREMENT OF THE SAI AND ITS ENVIRONMENT What is ’performance’? What are we going to measure?

SAI PMF: Evidence Based Assessment of Performance Qualitative assessment (Performance Report) Measurable indicators Holistic assessment of the SAI and its environment – including factors which are not covered by the indicators Objective measurement to inform qualitative assessment and track progress over time

SAI Performance Measurement Framework – Structure SAI Environment SAI Capability Impact SAI STRATEGY SAI Performance 3 indicators C. Strategy for Organizational Development 1 indicator B. Independence and Legal Framework 3 indicators SAI CORE BUSINESS Contribution to Public Financial Management, Governance and Tackling Corruption D. Audit Standards and Methodology 3-5 indicators E. Management & Support Structures 2 indicators F. Human Resources 3 indicators G. Communication and Stakeholder Management 3 indicators SAI SUPPORT SERVICES Public Financial Management Environment

SAI Performance Measurement Framework – Structure Note: all indicators should try to capture both what is written in legislation and procedures (de jure) and what happens in practice (de facto) SAI Performance SAI Capability SAI Environment Impact Under the control of the SAI Under the control of the SAI Foundational factors set by the legislative and/or executive, but which can be influenced by the SAI in the long term SAI STRATEGY Measured using other assessments. Achievement of impacts can not be wholly or directly attributed to SAI results. However, the way in which the SAI results are likely to contribute to high level impacts should be summarized in the narrative SAI performance report. C. Strategy for Organizational Development Strategy for organizational development B. Independence and Legal Framework Independence of the SAI Independence of the Head of the SAI and its Officials Mandate of the SAI A. SAI Performance Financial Audit Results Compliance Audit Results Performance Audit Results SAI Value Added Services SAI Annual Report SAI CORE BUSINESS D. Audit Standards and Methodology Audit Planning for the SAI Financial Audit Compliance Audit Performance Audit Quality Control & Quality Assurance of Audit Processes SAI Contribution to Aggregate Fiscal Discipline Efficient Resource Allocation Efficient Service Delivery Good Governance, Accountability and Tackling Corruption E. Management & Support Structures Management and Internal Control Support Structures & Infrastructure F. Human Resources Management of Personnel Professional development Leadership G. Communication and Stakeholder Management Communications Strategy, and Internal Communication Communication with the Government Branches Communication with the Media, the Citizens and the Civil Society SAI SUPPORT SERVICES Public Financial Management Environment Influences the effectiveness of the SAI but not under direct control of the SAI. Measured through other assessments but referred to explicitly in the narrative SAI performance report. Quality and Timeliness of Annual Financial Statements (PEFA PI-25) (Including completeness, timeliness and accounting standards used) Legislative Scrutiny of External Audit Reports (PEFA PI-28) (Including timeliness & extent of hearings, recommendations of the legislative & implementation by the executive)

SAI Performance Report - end product from an assessment (30-40 pages?) Enable SAI heads and senior management to Have a high level overview of the SAI’s performance Identify barriers to improved performance Track performance changes over time Explores how SAI external performance is constrained by Country context and PFM environment SAI environment SAI internal capability Provides a coherent and holistic assessment of SAI performance Informed by objective and evidence based performance indicators Analyzed within context Considered as a whole – not the sum of its indicator scores

SAI Performance Report i. Quality Assurance Statement ii. Observations on the SAI’s Performance and Impact (4-5 pages?) Integrated assessment of SAI performance: Linkages between the country context, SAI environment, SAI internal capability and SAI external performance Impact of SAI performance on public financial management, governance, accountability and tackling corruption Analysis of the SAIs capacity development efforts and prospects for further capacity development iii. SAI Management reaction Introduction Country and SAI Background Information Assessment of the SAI’s Environment, Capability and Performance Assessment against seven domains of SAI performance (evidence based indicator scores) Impact of the country context and PFM system on SAI effectiveness SAI Capacity Development Process Annex 1: Performance Indicator Summary & Tracking Performance Over Time Annex 2: Sources of Information & Evidence to Support Indicator Scoring SAI Performance Report

Indicator Design Each indicator provides an objective measure of current, not planned, performance …through 1-4 dimensions Measured on a scale of 0-4 Translated into a single indicator score SAI enables continual improvement Managed Level LEVEL 4 Measurement Scale Accountability for Government performance Established Level (Follows ISSAIs) LEVEL 3 Accountability for use of public funds Development Level LEVEL 2 Executive agents held to account Founding Level LEVEL 1 Each level provides a basis for accountability Activity not Established or does not Function LEVEL 0

Quality Assurance Statement Quality assurance crucial for objectivity and credibility of assessments Enables a single assessment to meet different stakeholder needs Most appropriate approach depends on purpose of the assessment Good practice: SAIs select QA process to meet the assessment purpose QA process is transparently disclosed in the performance report Separation between those undertaking the assessment, and those performing QA QA should cover: Is the report factually correct? Has the SAI PMF tool been applied correctly? Is there sufficient evidence to justify the indicator scores? Quality Assurance Statement Who has prepared the assessment? Who has provided quality assurance of the assessment? What were their quality assurance responsibilities? How have matters raised in the quality assurance process been addressed in the final report?

4. Looking ahead: Responsibilities and timelines

Potential risks related to having a SAI PMF Revelations of performance weaknesses It will be up to the SAI to decide if results are to be shared Openness and commitment to reform likely to encourage external partners to support capacity development May help draw attention to constraints outside the SAI’s control Possible negative effect of scoring: Use of performance league tables Intention is not to aggregate scores to the level of the entire SAI – as all indicators may not be of equal importance Scores partly linked to achievements of SAI’s own mandate Narrative report provides context for scoring Scoring is already applied in INTOSAI (AFROSAI-E) Scoring very useful for measuring progress over time within an SAI Profusion of Assessment Tools Risk that the SAI PMF becomes an additional tool Donors and SAIs continue to develop & use other tools

Identified Challenges in the Development of the SAI PMF (1) Different SAI models, mandates and country context make it challenging to develop a global tool ISSAIs and other established good practice is common benchmark Not measure at a too detailed level Assess against SAI’s own mandate Different SAI models and regions represented in Task Team and Reference Group Pilots to identify challenges and address them Possible to answer N/A and to add local indicators

Identified Challenges in the Development of the SAI PMF (2) Scoring / objective measurement not possible Performance report will provide qualitative assessment and put scoring into context Scoring useful for measuring progress over time Challenging timetable Johannesburg accords look for tool to be presented at INCOSAI XXI, China, 2013 WGVBS looking to approve Exposure Draft for INCOSAI Global piloting and consultations will extend beyond 2013

SAI PMF Timetable    WGVBS: Set Direction 2011, Q3 2011, Q4 2012, Q1 2012, Q2 2012, Q3 2012, Q4 2013, Q1 2013, Q2 2013, Q3 2013, Q4 Mapping Existing SAI Measurement Tools Concept Note for SAI PMF Develop SAI PMF Version 2.0 Phase 1 Pilots in 3 SAIs & Comments from Stakeholders Develop SAI PMF Exposure Draft Phase 2 Piloting and Pre-Consultation (ongoing into 2014 & 2015)  WGVBS: Approve way forward   WGVBS: Approve Exposure Draft Exposure Draft at INCOSAI

Implementation and Maintenance (proposed) Present SAI PMF at INTOSAI and other stakeholder events Promotion and Awareness Raising of SAI PMF Development and refinement of training material Training of trainers within INTOSAI regional bodies Training of SAI staff Training of other stakeholders (donors, consultants, CSOs) Training on Use of SAI PMF Assessments Host and maintain guidance materials on internet Provide Guidance and Support to Conduct of SAI PMF Assessments Maintain and update global summary of the status of SAI PMF assessments, to enhance coordination Manage an e-library of published SAI PMF assessments Maintain database on (published) SAI PMF indicator scores for benchmarking, evaluation and research Monitor Progress on Use of SAI PMF and Maintain Consolidated Indicator Data Decision to modify or update the SAI PMF (e.g. as ISSAI framework develops) Develop and refine the SAI PMF Technical Development and Refinement of SAI PMF INTOSAI Community Donor Community INTOSAI-Donor Secretariat INTOSAI-Donor Secretariat INTOSAI-Donor Secretariat Approved Trainers INTOSAI Regional Bodies INTOSAI-Donor Secretariat Approved Trainers INTOSAI-Donor Secretariat INTOSAI-Donor Secretariat INTOSAI-Donor Secretariat Database of SAIs Task Force Database of SAIs Task Force INTOSAI WGVBS SAI PMF Task Team

6. Further Information www.idi.no E-mail: INTOSAI.Donor.Secretariat@idi.no Mapping of Tools for Assessing Performance of Supreme Audit Institutions IDI, May 2012 Supreme Audit Institutions Performance Measurement Framework Concept Note IDI, May 2012