1. SAI Performance Measurement Framework
Presented by:
Date:
INTOSAI Working Group on the Value and Benefits of SAIs

2. Presentation Outline Introduction
Background and rationale for developing a SAI Performance Measurement Framework (PMF)
Responsibilities and reporting lines
Indicative content and design
Looking ahead
Further information

3. 1. Introduction

4. What is a Performance Measurement Framework?
Tool for measuring performance
…of an individual, system or organization
…against an agreed performance scale
…accommodating local circumstances
To enable
Better management of performance
Monitoring over time
Evaluation and lesson learning
Contributing to
Understanding performance
Improving performance
Not everything that matters can be measured

5. What is the SAI Performance Measurement Framework (PMF)?
A Performance Measurement Framework tailored to SAIs
Intended to give a high level assessment of SAI performance against established INTOSAI good practice:
The International Standards of Supreme Audit Institutions (ISSAI) framework
The Framework on the Value and Benefits of SAIs
Other INTOSAI guidance material
Combination of objective measurement and qualitative assessment
Analyzes 7 domains of SAI performance in the country context and aims at assessing the value and benefits of the SAI to society (impact)

6. SAI PMF – How will it be used in Practice?
Using the SAI PMF will be voluntary for the SAIs – the tool will be a public good which SAIs can make use of if it helps them achieve their objectives
The SAIs themselves will make the key decisions regarding the assessment
The SAI will also decide whether or not to publish results
Because it is voluntary, the SAI PMF is not intended to replace any internal performance measurement procedures
Who decides to initiate a SAI PMF?
Head of SAI
Who decides when to do a SAI PMF?
Head of SAI
Consulting with INTOSAI regions & donors as appropriate
Every 3-5 years recommended
Head of SAI
Approach selected to meet the SAI’s objectives
3 broad approaches
Quality assurance to promote objectivity and credibility
Who decides how to do a SAI PMF?
Head of SAI
Who decides if, when and how to publish a SAI PMF?

7. 3 Approaches to Conducting a SAI PMF Assessment
There are three main approaches to conducting an assessment using the SAI PMF. The PMF will be equally well suited for all three:
1. SAI Self Assessment
Trained Team from within the SAI
2. INTOSAI Peer Review
Peer SAI
INTOSAI Regional Body
3. External Assessment
Professional PFM Body
Donor Agency
Consultant
External auditor .
SAI Involvement
The approach will not necessarily affect how the assessment is carried out…
… but the SAI may use the results for different purposes depending on the approach

8. Background and rationale for developing a SAI Performance Measurement Framework (PMF)

9. INTOSAI Framework on the Value and Benefits of SAIs adopted 2010
Background
Full set of International Standards of Supreme Audit Institutions (ISSAIs) adopted by INTOSAI in 2010
INTOSAI Framework on the Value and Benefits of SAIs adopted 2010
Increased focus on use of country public financial management systems within the international donor community

10. Mandate: Johannesburg Accords (INCOSAI XX, 2010)
Responsibility
Working Group on the Value and Benefits of SAI
Task
Develop further credible measurement tools to support evaluations in terms of the framework for Communicating and Promoting the Value and Benefits of SAIs
Collaborating with
INTOSAI Capacity Building Committee
INTOSAI-Donor Steering Committee
Cognizance of
Tools that are already available at the level of… international organizations
Timetable
For consideration at XXI INCOSAI in 2013

11. SAI PMF Stakeholders and Purpose
Rationale for INTOSAI to Develop a SAI Performance Measurement Framework
SAI PMF Stakeholders and Purpose
Benefit to the SAI
Improve SAI approaches to capacity development through use of performance management
Identify opportunities to strengthen SAI performance
Supreme Audit Institutions
Promote transparency and accountability through credible public reporting of SAI performance (ISSAI 20)
Domestic Parliaments and Citizens
Enable better peer learning
INTOSAI Regions and INTOSAI Global Community
Demonstrate SAI performance changes to key stakeholders
International Development Community (Donors and SAI Providers of Support)
Stronger SAI-led capacity development programs
SAIs become model learning organizations
SAI
Enhanced SAI integrity and credibility with domestic stakeholders
Evidence-based policies and better targeted global and regional programs
Maintain external support for SAI capacity development

12. Why should SAIs use the PMF (1)
As a step towards ISSAI implementation
Increased understanding in the INTOSAI community of what good practice for SAIs entails
Individual SAIs can benchmark their performance against good practice – and get an indication of where there is room for improvement
They can use the assessment as a basis for more detailed ISSAI compliance assessments
To demonstrate progress and value and benefits to society
The framework will be well suited for measuring progress over time and demonstrating this to external stakeholders
Show to stakeholders how the SAI contributes to strengthening public financial management

13. Why should SAIs use the PMF (2)
Internal performance measurement / annual reporting
Help SAIs better manage their own performance
ISSAI requirement to assess and report on own performance – lead by example and become a model organization
Inform and secure support for capacity development initiatives
Identify strengths and weaknesses, increase understanding of reasons for good and weak performance
Help build a culture for performance improvement
SAIs can use assessment results to get support from donors and other external stakeholders

14. The Role of SAI PMF Assessments in SAI Capacity Development
Implement SAI Development Action Plan
High Level Performance Overview
Progress Monitoring
Identify Main SAI Strengths and Weaknesses
Investigate Underlying Causes
Recommend SAI Capacity Building Measures
Formulate or Update SAI Development Action Plan
(Medium Term)
Formulate or Update SAI Strategic Plan
(Long Term)
SAI PMF Assessment
SAI Capacity Building Needs Assessment

15. Responsibilities and reporting lines

16. Responsibilities and Reporting Lines
INTOSAI Congress (2013)
Approval
INTOSAI Working Group on the Value and Benefits of SAIs
Oversight and Direction
INTOSAI Capacity Building Committee
INTOSAI-Donor Steering Committee
Other Relevant INTOSAI Committees
In Consultation with:
WGVBS SAI PMF Task Team
Development of SAI Performance Measurement Framework
Coordinator: INTOSAI-Donor IDI
SAI PMF Reference Group
Technical Advice, Review and Quality Assurance
INTOSAI Capacity Building Committee Members
INTOSAI-Donor Steering Committee Members
Other Groups:
PEFA Secretariat, International Budget Partnership
INTOSAI Capacity Building Committee Members
INTOSAI-Donor Steering Committee Members

17. Task Team Reference Group INTOSAI Regional Organizations:
African Organization of English Speaking SAIs(AFROSAI-E)
African Organization of Francophone SAIs (CREFIAF)
SAIs:
Bahrain
Brazil
Chile
Germany
India
Mexico
Russia
United Kingdom
Development Agencies:
Inter American Development Bank
Coordinator:
INTOSAI-Donor IDI
Reference Group
INTOSAI Regional Organizations:
Pacific Association of SAIs (PASAI)
SAIs:
Canada
Denmark
Estonia
European Court of Auditors
Japan
Netherlands
Norway
Peru
Sweden
Development Agencies:
African Development Bank
OECD
European Commission
World Bank
Civil Society Organizations:
International Budget Partnership
Global Reporting Initiative (GRI)
PEFA Secretariat
CREFIAF = Le Conseil Régional de Formation des Institutions Supérieures de Contrôle des Finances Publiques d'Afrique Francophone Subsaharienne

18. Indicative content and design

19. INTOSAI Working Group on the Value and Benefits of SAIs: Criteria For an Effective SAI PMF
Comprehensiveness: Broad coverage of all key aspects of an SAIs work and its contribution to good governance, accountability and transparency
Objectivity: Indicators to measure performance and progress are objective
Subjectivity: Subjective factors can also be captured, e.g. through a narrative performance report
International agreement: performance indicators and measurement scales reflect agreed international standards and good practices where available
Relevance: Relevant to all SAIs, regardless of their administrative heritage or level of development
Performance: Enables analysis of SAI performance and identification of development challenges and opportunities
Progress: Facilitates the consistent measurement of SAI performance over time
Consistency: Consistent with pre-existing tools (both SAI coverage within broad public financial management measurement frameworks, and SAI specific performance measurement tools)
Compliance: Measures actual audit practices, as well as the quality of the SAI’s legislative / regulatory framework and internal guidelines / manuals
Brevity: Comprises the minimum number of performance indicators possible to cover key aspects of an SAI’s performance, so as make the tool of practical benefit to SAIs
User-Friendly: facilitates easy use by SAIs as self assessment or peer assessment tool
Endorsed at the 4th meeting of the Working Group on the Value and Benefits of SAIs in Jamaica, August 2011.

20. SAI PMF Task Team: Initial Work and Decisions
No existing framework fully meets INTOSAI needs
Develop a new tool – drawing heavily on existing ones (AFROSAI-E, PEFA, INTOSAI Peer Review Guide, IDI Capacity Building Needs Assessment Handbook)
Mapping of 20 Existing SAI Assessment Tools
ISSAI Framework where possible – recognizing it will continue to evolve over time
The SAI’s mandate
INTOSAI guides & other generally accepted international standards or good practices
What to Measure Performance Against?
External performance – delivery of core audit activities (audit coverage, submission of audit reports, etc.)
Internal enablers of performance – SAI strategy, audit processes, SAI support services
SAI’s operating environment (i.e. independence and legal framework) – recognizing this is largely beyond the control of the SAI
Reported in the context of the specific country governance and public financial management environment and its impact on the SAI’s performance
HOLISTIC MEASUREMENT OF THE SAI AND ITS ENVIRONMENT
What is ’performance’? What are we going to measure?

21. SAI PMF: Evidence Based Assessment of Performance
Qualitative assessment (Performance Report)
Measurable indicators
Holistic assessment of the SAI and its environment – including factors which are not covered by the indicators
Objective measurement to inform qualitative assessment and track progress over time

22. SAI Performance Measurement Framework – Structure
SAI Environment
SAI Capability
Impact
SAI STRATEGY
SAI Performance
3 indicators
C. Strategy for Organizational Development
1 indicator
B. Independence and Legal Framework
3 indicators
SAI CORE BUSINESS
Contribution to Public Financial Management, Governance and Tackling Corruption
D. Audit Standards and Methodology
3-5 indicators
E. Management & Support Structures
2 indicators
F. Human Resources
3 indicators
G. Communication and Stakeholder Management
3 indicators
SAI SUPPORT SERVICES
Public Financial Management Environment

23. SAI Performance Measurement Framework – Structure
Note: all indicators should try to capture both what is written in legislation and procedures (de jure) and what happens in practice (de facto)
SAI Performance
SAI Capability
SAI Environment
Impact
Under the control of the SAI
Under the control of the SAI
Foundational factors set by the legislative and/or executive, but which can be influenced by the SAI in the long term
SAI STRATEGY
Measured using other assessments. Achievement of impacts can not be wholly or directly attributed to SAI results. However, the way in which the SAI results are likely to contribute to high level impacts should be summarized in the narrative SAI performance report.
C. Strategy for Organizational Development
Strategy for organizational development
B. Independence and Legal Framework
Independence of the SAI
Independence of the Head of the SAI and its Officials
Mandate of the SAI
A. SAI Performance
Financial Audit Results
Compliance Audit Results
Performance Audit Results
SAI Value Added Services
SAI Annual Report
SAI CORE BUSINESS
D. Audit Standards and Methodology
Audit Planning for the SAI
Financial Audit
Compliance Audit
Performance Audit
Quality Control & Quality Assurance of Audit Processes
SAI Contribution to
Aggregate Fiscal Discipline
Efficient Resource Allocation
Efficient Service Delivery
Good Governance, Accountability and Tackling Corruption
E. Management & Support Structures
Management and Internal Control
Support Structures & Infrastructure
F. Human Resources
Management of Personnel
Professional development
Leadership
G. Communication and Stakeholder Management
Communications Strategy, and Internal Communication
Communication with the Government Branches
Communication with the Media, the Citizens and the Civil Society
SAI SUPPORT SERVICES
Public Financial Management Environment
Influences the effectiveness of the SAI but not under direct control of the SAI.
Measured through other assessments but referred to explicitly in the narrative SAI performance report.
Quality and Timeliness of Annual Financial Statements (PEFA PI-25)
(Including completeness, timeliness and accounting standards used)
Legislative Scrutiny of External Audit Reports (PEFA PI-28)
(Including timeliness & extent of hearings, recommendations of the legislative & implementation by the executive)

24. SAI Performance Report - end product from an assessment (30-40 pages?)
Enable SAI heads and senior management to
Have a high level overview of the SAI’s performance
Identify barriers to improved performance
Track performance changes over time
Explores how SAI external performance is constrained by
Country context and PFM environment
SAI environment
SAI internal capability
Provides a coherent and holistic assessment of SAI performance
Informed by objective and evidence based performance indicators
Analyzed within context
Considered as a whole – not the sum of its indicator scores

25. SAI Performance Report
i. Quality Assurance Statement
ii. Observations on the SAI’s Performance and Impact (4-5 pages?)
Integrated assessment of SAI performance: Linkages between the country context, SAI environment, SAI internal capability and SAI external performance
Impact of SAI performance on public financial management, governance, accountability and tackling corruption
Analysis of the SAIs capacity development efforts and prospects for further capacity development
iii. SAI Management reaction
Introduction
Country and SAI Background Information
Assessment of the SAI’s Environment, Capability and Performance
Assessment against seven domains of SAI performance (evidence based indicator scores)
Impact of the country context and PFM system on SAI effectiveness
SAI Capacity Development Process
Annex 1: Performance Indicator Summary & Tracking Performance Over Time
Annex 2: Sources of Information & Evidence to Support Indicator Scoring
SAI Performance Report

26. Indicator Design
Each indicator provides an objective measure of current, not planned, performance
…through 1-4 dimensions
Measured on a scale of 0-4
Translated into a single indicator score
SAI enables continual improvement
Managed Level
LEVEL 4
Measurement Scale
Accountability for Government performance
Established Level
(Follows ISSAIs)
LEVEL 3
Accountability for use of public funds
Development Level
LEVEL 2
Executive agents held to account
Founding Level
LEVEL 1
Each level provides a basis for accountability
Activity not Established or does not Function
LEVEL 0

27. Quality Assurance Statement
Quality assurance crucial for objectivity and credibility of assessments
Enables a single assessment to meet different stakeholder needs
Most appropriate approach depends on purpose of the assessment
Good practice:
SAIs select QA process to meet the assessment purpose
QA process is transparently disclosed in the performance report
Separation between those undertaking the assessment, and those performing QA
QA should cover:
Is the report factually correct?
Has the SAI PMF tool been applied correctly?
Is there sufficient evidence to justify the indicator scores?
Quality Assurance Statement
Who has prepared the assessment?
Who has provided quality assurance of the assessment?
What were their quality assurance responsibilities?
How have matters raised in the quality assurance process been addressed in the final report?

28. 4. Looking ahead: Responsibilities and timelines

29. Potential risks related to having a SAI PMF
Revelations of performance weaknesses
It will be up to the SAI to decide if results are to be shared
Openness and commitment to reform likely to encourage external partners to support capacity development
May help draw attention to constraints outside the SAI’s control
Possible negative effect of scoring: Use of performance league tables
Intention is not to aggregate scores to the level of the entire SAI – as all indicators may not be of equal importance
Scores partly linked to achievements of SAI’s own mandate
Narrative report provides context for scoring
Scoring is already applied in INTOSAI (AFROSAI-E)
Scoring very useful for measuring progress over time within an SAI
Profusion of Assessment Tools
Risk that the SAI PMF becomes an additional tool
Donors and SAIs continue to develop & use other tools

30. Identified Challenges in the Development of the SAI PMF (1)
Different SAI models, mandates and country context make it challenging to develop a global tool
ISSAIs and other established good practice is common benchmark
Not measure at a too detailed level
Assess against SAI’s own mandate
Different SAI models and regions represented in Task Team and Reference Group
Pilots to identify challenges and address them
Possible to answer N/A and to add local indicators

31. Identified Challenges in the Development of the SAI PMF (2)
Scoring / objective measurement not possible
Performance report will provide qualitative assessment and put scoring into context
Scoring useful for measuring progress over time
Challenging timetable
Johannesburg accords look for tool to be presented at INCOSAI XXI, China, 2013
WGVBS looking to approve Exposure Draft for INCOSAI
Global piloting and consultations will extend beyond 2013

32. SAI PMF Timetable    WGVBS: Set Direction
2011, Q3
2011, Q4
2012, Q1
2012, Q2
2012, Q3
2012, Q4
2013, Q1
2013, Q2
2013, Q3
2013, Q4
Mapping Existing SAI Measurement Tools
Concept Note for SAI PMF
Develop SAI PMF Version 2.0
Phase 1 Pilots in 3 SAIs & Comments from Stakeholders
Develop SAI PMF Exposure Draft
Phase 2 Piloting and Pre-Consultation (ongoing into 2014 & 2015) 
WGVBS: Approve way forward  
WGVBS: Approve Exposure Draft
Exposure Draft at INCOSAI

33. Implementation and Maintenance (proposed)
Present SAI PMF at INTOSAI and other stakeholder events
Promotion and Awareness Raising of SAI PMF
Development and refinement of training material
Training of trainers within INTOSAI regional bodies
Training of SAI staff
Training of other stakeholders (donors, consultants, CSOs)
Training on Use of SAI PMF Assessments
Host and maintain guidance materials on internet
Provide Guidance and Support to Conduct of SAI PMF Assessments
Maintain and update global summary of the status of SAI PMF assessments, to enhance coordination
Manage an e-library of published SAI PMF assessments
Maintain database on (published) SAI PMF indicator scores for benchmarking, evaluation and research
Monitor Progress on Use of SAI PMF and Maintain Consolidated Indicator Data
Decision to modify or update the SAI PMF (e.g. as ISSAI framework develops)
Develop and refine the SAI PMF
Technical Development and Refinement of SAI PMF
INTOSAI Community
Donor Community
INTOSAI-Donor Secretariat
INTOSAI-Donor Secretariat
INTOSAI-Donor Secretariat
Approved Trainers
INTOSAI Regional Bodies
INTOSAI-Donor Secretariat
Approved Trainers
INTOSAI-Donor Secretariat
INTOSAI-Donor Secretariat
INTOSAI-Donor Secretariat
Database of SAIs Task Force
Database of SAIs Task Force
INTOSAI WGVBS
SAI PMF Task Team

34. 6. Further Information www.idi.no
Mapping of Tools for Assessing Performance of Supreme Audit Institutions
IDI, May 2012
Supreme Audit Institutions Performance Measurement Framework
Concept Note
IDI, May 2012