1. Supreme Audit Institution Performance Measurement Framework (SAI PMF)
Presented by: Yngvild H. Arnesen, World Bank
The Exchange, Abu Dhabi, May 2013

2. Presentation Outline What is the SAI PMF and why is it developed?
Why do a SAI PMF, and how?
Practical example: Audit Service Sierra Leone
What is next on the SAI PMF?

3. 1. What is the SAI PMF and why is it developed?

4. What is the SAI PMF?
A Performance Measurement Framework tailored to SAIs – developed by INTOSAI
Intended to give a holistic, high level assessment of SAI performance against established INTOSAI points of reference:
The International Standards of Supreme Audit Institutions (ISSAI) framework;
The Framework on the Value and Benefits of SAIs;
Other INTOSAI guidance material and international good practices.
Combination of objective measurement and qualitative assessment
Assesses SAI performance in the country context and contributes to identifying the value and benefits of the SAI to citizens

5. Why is the SAI PMF developed?
INTOSAI Framework on the Value and Benefits of SAIs adopted 2010
Increased focus on use of country public financial management systems within the international donor community
Full set of International Standards of Supreme Audit Institutions (ISSAIs) adopted by INTOSAI in 2010

6. How is the SAI PMF being developed?
Mandated by Johannesburg Accords, INTOSAI Congress, 2010
Responsibility of Working Group on the Value and Benefits of SAIs
SAI PMF Task Team
INTOSAI Regional Organizations:
African Organization of English Speaking SAIs(AFROSAI-E) and African Organization of Francophone SAIs (CREFIAF)
SAIs: Bahrain, Brazil, Chile, Germany, India, Mexico, Russia, United Kingdom
Development Agencies: Inter American Development Bank
Coordinator:INTOSAI-Donor IDI
SAI PMF Reference Group
INTOSAI Regional Organizations:
Pacific Association of SAIs (PASAI)
SAIs: Canada, Denmark, Estonia, European Court of Auditors, Japan, Netherlands, Norway, Peru, Sweden
Development Agencies:
African Development Bank, OECD, European Commission, World Bank
Civil Society Organizations:
International Budget Partnership, Global Reporting Initiative (GRI), PEFA Secretariat
Mention piloting and comments from 30 different organizations….

7. SAI PMF: Evidence Based Assessment of Performance
Qualitative Assessment (Performance Report)
Measurable indicators
Holistic, judgmental assessment of the SAI and its environment – including factors not covered by the indicators
Objective measurement to inform qualitative assessment and track progress over time
-in order to be objective: it consists of measureable indicators with a score from 0 – 4.
-the measureable indicators are input to a more qualitative assessment of the SAI. It also includes things which is not measureable, but essential for the SAIs performance. It also includes factors in the environment that will affect the performance of the SAI.
The Country context and PFM environment : does the Parliament receive its audits? Is it a huge backlog of the review of the Public Accounts Committee? Freedom of the Media?

8. SAI Performance Measurement Framework
SAI Reporting
SAI Environment
SAI Capability
SAI Impact
SAI STRATEGY
SAI Reporting
C. Strategy for Organizational Development
B. Independence and Legal Framework
SAI CORE BUSINESS
Value and Benefits of Supreme Audit Institutions
D. Audit Standards and Methodology
E. Management & Support Structures
F. Human Resources and Leadership
G. Communication and Stakeholder Management
In each domain there are indicators, and each indicator consists of several dimensions.
Each indicator provides an objective measure of current, not planned, performance
…through 1-4 dimensions on each indicator
Measured on a scale of 0-4
SAI SUPPORT SERVICES
Country Context, Governance and Public Financial Management

9. 2. Why do a SAI PMF, and how?

10. Why do a SAI PMF assessment?
Manage own performance:
Where are we?
Monitor progress over time
Implementation of International Standards for Supreme Audit Institutions (ISSAIs)
Demonstrate accountability
Provide evidence to donors on SAI performance

11. Voluntary use Who decides to initiate a SAI PMF? Head of SAI
Who decides when to do a SAI PMF?
Head of SAI
Head of SAI
Who decides how to do a SAI PMF?
Head of SAI
Who decides if, when and how to publish a SAI PMF?

12. How will the SAI PMF Assessment be conducted?
SAI Involvement
1. SAI Self Assessment
2. Peer Review
3. External Assessment
The approach will not necessarily affect how the assessment is carried out…
… but the SAI may use the results for different purposes depending on the approach

13. Issues in measuring SAI performance
SAI PMF is not designed to judge and rank SAI against each other, but aims to identify strengths and weaknesses of the individual SAI
SAI’s should not be held accountable for matters over which they have no control Solution: Clear distinction between factors within and outside the SAI’s control
Some aspects of SAI performance are difficult to measure, especially ‘quality’ Solution: Qualitative analysis in performance report, focus on processes and systems that should result in high quality audits
The indicator scores should not be captured into one aggregate score, because:
How would we weigh the relative importance of a 4 in compliance auditing against a 2 in SAI mandate?
Could the weightings be the same for Anglophone SAIs and Court SAIs?
Weightings would change as SAI matures and prioritises new areas
A SAI with low scores on the SAI PMF can perform better than a higher scoring SAI given its resources, mandate and history.
Solution issue 2: SAI PMF makes a distinction between:
A group of standardized indicators focused on SAI reporting, attributable to the SAI but bounded by its environment
A group of standardized indicators focused on SAI capability, largely under the SAI’s direct control
A group of standardized indicators focused on the SAI environment (such as independence and legal framework) which is specific to the SAI but only under the indirect influence.
A common problem in developing performance measurement systems is that an organization should not be held accountable for matters over which it has no control. However, it is known that the effectiveness of an SAI is intrinsically bound to the effectiveness of other parts of the public financial management system, such as legislative scrutiny of external audit reports. In practice there is a wide range of grey areas: foundational factors such as the legislative framework and important results which are not directly under the control of the SAI but which the SAI may seek to influence, or external impediments which may alter the way in which the SAI acts in order to achieve its objectives. Those things that matter to an SAI’s performance should be measured. But one should be clear about the extent to which these factors are under the control or influence of the SAI.
Solution issue 2:
Scope in a narrative SAI performance report to capture factors which matter to an SAI’s performance, but which can not be measured;
SAI PMF focuses on the presence of processes and systems that, if they are applied, should result in high quality audit reporting
ssues 3: Some aspects of SAI Performance are difficult to measure, especially ‘quality’
Solution issue 3:
Solution issue 4:
Assess the SAI against its own legal mandate
Examples:
The norm for the timeliness of the submission of reports should be based on the SAI’s mandate or national laws
Where SAI has no mandate, reflected in domain B and considered N/A in other domains (use of N/A to be decided at planning stage)
To ensure the SAI PMF is as relevant as possible for each individual SAI, participants agreed that while the ISSAIs largely determine what good practice looks like, the SAI PMF should seek to measure the extent to which the SAI delivers on its mandate. For example, the ISSAIs point out that reports should be submitted on a timely basis, but do not define timely. Where the SAI’s mandate or other national laws define the timeliness of submission of audit reports, the SAI PMF should measure actual report submission against the SAI’s mandate. Only in cases where there are no deadlines in the legal framework shall the SAI be assessed against a uniform set of global benchmarks.
Issue 5: The financial resources allocated to the SAI may not be sufficient to carry out its mandate. In such cases, it may not be appropriate to punish SAI for low performance on ‘coverage’, professional training etc.
Solution Issue 5:
Assess the SAI against its own legal mandate without regard of its financial resources.
In addition, SAI 5 (i) assesses whether the SAI has reported to Parliament to what extent it could not deliver on its mandate due to insufficient resources
SAI-PR includes the SAI’s budget request, approval and actual allocation; include pertinent issues in the narrative report

14. Issues in measuring SAI performance (2)
The mandate of a SAIs differs across countries and may not be aligned with the ISSAIs. Solution: Assess the SAI against its own mandate (but assess its mandate against ISSAIs)
The financial resources allocated to the SAI may not be sufficient to carry out its mandate; not appropriate to punish SAI for low performance on ‘coverage’, professional training etc. Solution: Assess SAI against its legal mandate, assess its reporting to parliament on coverage, include pertinent issues in performance report
The indicator scores should not be captured into one aggregate score, because:
How would we weigh the relative importance of a 4 in compliance auditing against a 2 in SAI mandate?
Could the weightings be the same for Anglophone SAIs and Court SAIs?
Weightings would change as SAI matures and prioritises new areas
A SAI with low scores on the SAI PMF can perform better than a higher scoring SAI given its resources, mandate and history.
Solution issue 2: SAI PMF makes a distinction between:
A group of standardized indicators focused on SAI reporting, attributable to the SAI but bounded by its environment
A group of standardized indicators focused on SAI capability, largely under the SAI’s direct control
A group of standardized indicators focused on the SAI environment (such as independence and legal framework) which is specific to the SAI but only under the indirect influence.
A common problem in developing performance measurement systems is that an organization should not be held accountable for matters over which it has no control. However, it is known that the effectiveness of an SAI is intrinsically bound to the effectiveness of other parts of the public financial management system, such as legislative scrutiny of external audit reports. In practice there is a wide range of grey areas: foundational factors such as the legislative framework and important results which are not directly under the control of the SAI but which the SAI may seek to influence, or external impediments which may alter the way in which the SAI acts in order to achieve its objectives. Those things that matter to an SAI’s performance should be measured. But one should be clear about the extent to which these factors are under the control or influence of the SAI.
Solution issue 2:
Scope in a narrative SAI performance report to capture factors which matter to an SAI’s performance, but which can not be measured;
SAI PMF focuses on the presence of processes and systems that, if they are applied, should result in high quality audit reporting
ssues 3: Some aspects of SAI Performance are difficult to measure, especially ‘quality’
Solution issue 3:
Solution issue 4:
Assess the SAI against its own legal mandate
Examples:
The norm for the timeliness of the submission of reports should be based on the SAI’s mandate or national laws
Where SAI has no mandate, reflected in domain B and considered N/A in other domains (use of N/A to be decided at planning stage)
To ensure the SAI PMF is as relevant as possible for each individual SAI, participants agreed that while the ISSAIs largely determine what good practice looks like, the SAI PMF should seek to measure the extent to which the SAI delivers on its mandate. For example, the ISSAIs point out that reports should be submitted on a timely basis, but do not define timely. Where the SAI’s mandate or other national laws define the timeliness of submission of audit reports, the SAI PMF should measure actual report submission against the SAI’s mandate. Only in cases where there are no deadlines in the legal framework shall the SAI be assessed against a uniform set of global benchmarks.
Issue 5: The financial resources allocated to the SAI may not be sufficient to carry out its mandate. In such cases, it may not be appropriate to punish SAI for low performance on ‘coverage’, professional training etc.
Solution Issue 5:
Assess the SAI against its own legal mandate without regard of its financial resources.
In addition, SAI 5 (i) assesses whether the SAI has reported to Parliament to what extent it could not deliver on its mandate due to insufficient resources
SAI-PR includes the SAI’s budget request, approval and actual allocation; include pertinent issues in the narrative report

15. 3. Practical Example: Audit Service Sierra Leone

16. Piloting the SAI PMF First round of pilots of SAI PMF:
Second half of 2012
Three SAIs: Djibouti, Norway, Sierra Leone
Carried out by SAI PMF Task Team
Pilot of Audit Service Sierra Leone:
Field work in October 2012 in cooperation with AFROSAI-E
Document analysis, review of audit files, interviews
Entire process: about six months

17. Results – Sierra Leone Independence & Legal Framework
Strategy for Organizational Development
SAI Reporting
Strategy guides organizational development & delivery of mandate
Monitoring weaknesses limits continual improvement of audit methods & public reporting of SAI performance
Financial audit coverage
Timely submission & publication of audit reports
Selection of performance audit topics
Performance audit coverage
Follow-up of audits
Reporting performance
Constitutional independence of SAI & AG
Mandate for public sector
Financial, compliance & performance audit
Freedom to publish reports
Financial independence:
Budget process & receipt of funds
Audit Standards & Methodology
Risk-based financial & compliance audit manual
Documentation & quality control
Supervision & review
Training based on manual
Performance audit capacity
Quality assurance system
Planning & monitoring delivery & cost of audits
Performance audit manual
Management & Support Structures
Human Resources & Leadership
Communication & Stakeholder Management
[CLICK]
The first domain on the SAI PMF diagram is ‘Independence and legal framework’. This measures the independence of the SAI against requirements of the Lima and Mexico declarations, both in terms of what is written in the constitution and relevant laws, as well as how it operates in practice. It also measures whether the SAI has a sufficiently broad mandate to audit the use of all public funds and carry out financial, compliance and performance audit activities. The findings for the Audit Service Sierra Leone were that most of the requirements were in place, but there were issues relating to financial independence, due to the way in which the Ministry of Finance controls the submission of the SAI’s budget to Parliament, and prevents the timely release of budgeted funds to the SAI. This is a very common independence issue for SAIs in developing countries.
Overall, the SAI obtained scores of 3s and 4s on this domain, on a scale of 0 to 4, with a 4 being the top level of performance.
The second domain examines the SAI’s strategy for organizational development. SAI owned and led strategic plans are essential for long term, sustainable capacity development and have been a priority initiative across INTOSAI in recent years. Overall, the strategic and operational plan of the SAI met many of the requirements taken from INTOSAI good practices, such as the IDI Strategic Planning guide. The main limitations related to how implementation of the strategic plan is measured and monitored, enabling the SAI to learn from its experiences.
This domain has one indicator, which scored a 3, meaning it basically meets good practices, but does not fully enable continual improvement.
The third domain is the heart of the SAI PMF. It measures the quality of audit, in terms of audit standards and methodology, both in design and in practice. It draws heavily on the results of reviews of samples of audit files. On the one hand, audit manuals and training is designed to give a sound foundation for financial, compliance and performance audit. However in practice, there are major weaknesses in the observed audit practices and documentation on audit files. Of particular concern is the lack of sufficient documentation to fully support all audit conclusions, and major gaps in implementation of the designed audit quality control system. Supervision and review by principal auditors and audit managers was limited or missing in many of the audits.
As a result, the scores in this domain were the lowest in the SAI PMF assessment, mainly 1’s (meaning the function exists but is very basic) and 2’s (meaning the function has been designed but only partly implemented).
The fourth domain is management and support structures. This examines the SAI’s assets and use of assets, financial management, and the general effectiveness of its management of staff. While there were some strengths such as its Code of Ethics and staff rotation policy designed to maintain independence from audited entities in a highly corrupt environment, there were major weaknesses in terms of monitoring the utilization of staff, as well as the inadequate physical infrastructure  and IT networks to support the effective functioning of the SAI. These factors can be see to help ‘explain’ the problems of audit quality in the domain above.
The SAI scored 2’s in this domain.
The fifth domain measures the human resource function and leadership of human resources. Here there were major weaknesses in the ability of the SAI to fill vacancies, especially at the middle management level, and challenges for the leadership of the SAI to ensure that delegated responsibilities where actually implemented: activities were frequently not performed by the staff to whom they were formally delegated, but by their supervisors and line managers. For example, audit work was undertaken by supervisors and managers rather than the auditors, and review of audit files and reports end up being passed up to the Deputy Auditor Generals and Auditor General. Again, the weaknesses here have a clear impact on the audit quality domain.
The indicators in this domain received a range of scores from 3 (for professional development and training) to 1 (for effective leadership of human resources).
The sixth domain is communications and stakeholder management. This domain showed the SAI had established a good and supportive relationship with the Public Accounts Committee, and was using good, appropriate tools to disseminate its audit findings and key messages, such as radio debates. However, it could do more in relation to proactive communication with audited entities.
In this domain, the SAI again received a range of scores from 1 to 3.
The final domain in the diagram measures the SAI reporting. Here, the SAI showed good coverage of financial and compliance audits (I.e. most of its mandate is subject to audit), good selection of performance audit topics, though performance audit was a relatively new discipline for the SAI and the actual coverage of performance audit is still quite limited. Reports were submitted to Parliament and others within statutory deadlines and published promptly. The SAI does not yet produce a report on its own performance.
Overall, the SAI scored 2’s and 3’s in this area.
External training support
Link staff plan to mandate
Staff accountability & reward for performance
Attract & retain staff
Managerial vacancies
Training resources
Code of ethics, job rotation, structures
Monitoring staff utilization
Infrastructure & IT
Reporting to PAC
Working with PAC
Radio debates
Internal comms.
Comms. with audited entities

18. Internal Factors Impacting on SAI Performance
Managerial vacancies
Planning & monitoring use of (staff) resources
Staff motivation & reward
Quality reports.
Coverage.
Timeliness.
Poor documentation
Quality control not implemented
Weak supervision & review
Quality assurance system
Audit manuals
Staff accountability for performance
Internal communications
Non-audit domains impacts on the audit quality.
No systems to reward performance.
Quality control & assurance still weak, although systems in place on paper.
Flinke til aa ha fysiske moeter selv om de er paa mange forskjellige steder.

19. External Factors Impacting on SAI Peformance – Sierra Leone
Strong donor support
Participation in INTOSAI & AFROSAI-E programs
Weak accountability undermines cooperation with SAI
Context
Weak record keeping, internal control & internal audit
Incomplete or missing financial statements
Unclear performance objectives & no measurement
PFM
Weak financial independence
Otherwise strong legal framework
SAI Legal Framework
Inadequate budget to hire & train sufficient, appropriate staff
Inadequate premises, vehicles, IT networks to function efficiently
SAI Resources

20. Value & Benefits of the Audit Service to Citizens
Fiscal Discipline
Identification of:
Major revenue discrepancies
Failings in collection of fisheries revenues
Audit of public debt management
Efficient Resource Allocation
Ensures funds spent for purposes intended
Effective Service Delivery
Legality of spending
Identification of failings in:
Anti-malarial drug delivery systems
Access to clean water program
Distribution of agricultural inputs
Note that the input for this section comes not only from scoring the indicators and doing the background analysis, but also reading the annual audit report, and selected audit reports including performance audit reports, as well as discussions with key external strakeholders (and possibly review of stakeholder surveys, if any).
SUMMARY
Audit service delivers significant, potential value and benefits to citizens.
Nevertheless, environmental constraints limit its value and benefits, as well as its audit activities
It submits and publishes sufficient audit reports in a timely manner
but still faces internal challenges – especially managerial – to improve the quality of audit work
Support from the INTOSAI & donor communities since 2004 has led to significant performance improvements
but there are questions on the sustainability of some reforms
Promoting fiscal transparency and accountability
Deterring, preventing & detecting corruption
Communicating with & engaging citizens: demonstrating relevance to stakeholders

21. Added Value for Audit Service Sierra Leone
Identification of strengths and weaknesses – knows what to focus on in future
Knowledge about weaknesses caused by internal factors which are under the SAI’s control…
… and weaknesses which are caused by external factors that the SAI may try to influence
Foundation for communicating value and benefits of the Audit Service
Used as baseline for tracking performance over time – linked to external funding
Used to inform design of capacity building interventions

22. 4. What is next in the development of SAI PMF?

23. SAI PMF Roll-out INTOSAI Congress 2013: Official exposure draft
Training: For assessors and future trainers – in all INTOSAI regions during 2013/2014
Piloting: Around 20 volunteer SAIs with different characteristics
Guidance material: Prepared by IDI
Support IDI

24. E-mail: INTOSAI.Donor.Secretariat@idi.no
For more information:

25. Thank you!