1. The PEFA Performance Measurement Framework
World Bank
PFM course for PFM Staff
PEFA Secretariat - June 2011

2. PEFA Secretariat - June 2011
Content
What is PEFA?
Mechanics of the Framework
The Assessment process
Comparisons (time; countries)
Use and roll-out
PEFA Secretariat - June 2011

3. PEFA Secretariat - June 2011
Content
What is PEFA?
Mechanics of the Framework
The Assessment process
Comparisons (time; countries)
Use and roll-out
PEFA Secretariat - June 2011

4. PEFA Secretariat - June 2011
What is PEFA?
‘Public Expenditure & Financial Accountability’
Established in 2001 by 7 agencies; now working in tandem with OECD-DAC Task Force on PFM
Context: ‘Strengthened Approach’ to supporting PFM Reform
Country-led PFM reform program, reflecting country priorities & implemented via government structures
Coordinated program of international support
Common information pool, for measuring performance & monitoring results over time: i.e. the PEFA PFM Performance Measurement Framework
PEFA Secretariat - June 2011

5. Purpose of the PEFA Framework
The Framework provides:
a high level overview of all aspects of a country’s PFM systems performance (including revenue, expenditure, procurement, financial assets/ liabilities): are the tools in place to help deliver the 3 main budgetary outcomes? (aggregate fiscal discipline; strategic resource allocation; efficient service delivery)
It does not provide:
An assessment of underlying causes for good or poor performance i.e. the capacity factors
An assessment of government fiscal & financial policies
THE PURPOSE OF THIS PRESENTATION is to give you an insight into the content of the PFM Performance Measurement Framework, what it can measure, what it covers, how it measures PFM performance and how the results are presented in a report. On this basis you should be able to go on to the practical case study exercises in the next session, where you will apply the Framework
***************
The Framework measures performance by looking at the outputs from the PFM systems, but not the reasons behind performance such as institutional, organizational and human resource aspects of capacity. [It can be considered as an annual medical with your GP. It will point to weaknesses, some of which are easy to identify the cause of and remedies to, whereas other will need to be investigated by specialist who can then propose detailed measures]
The Framework does not assess the government’s fiscal and expenditure policies, but only the performance of the systems which the government needs in order to implement such policies.
It is intended to be widely applicable across countries with different levels of economic development. No reference to poverty reduction and poverty reducing expenditures because such concepts specifically related to HIPC countries and may not be used in many middle (and high) income countries (ref. testing of indicators in the UK and Norway).
PEFA Secretariat - June 2011

6. PEFA Secretariat - June 2011
Content
What is PEFA?
Mechanics of the Framework
The Assessment process
Comparisons (time; countries)
Use and roll-out
PEFA Secretariat - June 2011

7. Components of the PEFA Framework
Standard high-level performance indicator set
31 indicators, cover all major aspects of PFM systems (PI 2, 3 & 19 recently revised – check website)
Compromise between simplicity & comprehensiveness
Widely applicable to countries at different levels of development
A standard ‘PFM performance report’ (PFM-PR)
developed to provide country background
evidence on the indicators
an integrated summary assessment
PEFA Secretariat - June 2011

8. Structure of the indicator set
C. Budget cycle
Policy based budgeting PI 11 & 12
D. Donor practices D1 – D3
A. PFM out-turns
B. Cross-cutting features
This is a another way to look at the six dimensions of the indicator set. It illustrates how a budget cycle might work:
Starting with policy based budget – plans should reflect priorities/ policies of government.
Having got this the budget then needs to be executed in a predictable fashion – the best budget in the world is of no use if it cannot be executed
Similarly accounting and reporting is essential so as to inform progress in execution of the budget
External audit and scrutiny not just for ensuring integrity but also to provide feedback to policy makers on the success or otherwise of policies – this requires more than just looking at the numbers
Across all of these are comprehensiveness and transparecny – has everything been captured ?
Budget credibility indicators reflect outputs of the system as a result of the institutional and more qualitative assessment covered by the remainder of the indicator set. Understanding links between the institutional strengths/weaknesses and budget credibility is important for summarizing the assessment.
Similar for donor indicator ratings’ relations to government performance and budget credibility.
See the list of indicators on page 9 of the English/blue booklet.
Budget credibility
PI 1 - 4
Predictability & control in budget execution PI
External scrutiny & audit PI
Comprehensiveness & transparency
PI 5 -10
Accounting, recording, reporting PI
PEFA Secretariat - June 2011

9. PEFA Secretariat - June 2011
Indicator calibration and scoring
Calibrated on 4-point Cardinal Scale (A, B, C, D)
Reflecting internationally accepted ‘good practice’ (not best)
Do not score if evidence is insufficient
Most indicators have 2, 3 or 4 sub-indicators
Each sub-indicator (dimension) must be rated separately
In total 76 dimensions to be rated
Each indicator represents a desirable result, output, function or institutional feature
The scoring system is base don a four point ordinal scale (A, B, C, D).
Performance is measured against a calibration based on ‘internationally accepted good practice’ representing a score ‘A’. This is not necessarily the same as international ‘best practice’.
Sophisticated reforms such as performance based budgeting are not incorporated in the Framework, since the salient features of such reforms often lack international consensus. So ‘A’ ratings do not mean New Zealand standard of system.
‘D’ represents poor/unsatisfactory performance. ‘C’ a rudimentary but crudely functioning system.
Performance is not measured against the local legislation, because this legislation may not always represent internationally accepted good practice. E.g. PI-11(iii) on approval of the budget before the start of the fiscal year.
A numerical scale has deliberately been avoided in order to discourage simplistic numerical aggregation, which may serve only the purpose of cross-country comparison. Such cross-country comparison was not the purpose of developing the PEFA framework and therefore not the basis for its design.
Look at an indicator example in the blue booklet e.g. PI-19 pages Note the 4 parts for each indicator: general description; definition of dimensions; calibration table with specific requirements for each score; scoring aggregation method.
It is useful to start scoring by considering the description of the D rating and whether the PFM system meets the requirements of a C rating. If so, would it also meet the criteria for a B rating etc. In this way, none of the requirements will be overlooked.
Do not score if evidence is insufficient. A ‘no score’ indicates a serious gap in information which the government should normally possess. This information gap should be a focus for improvement before the next assessment takes place. Do not score ‘D’ if there is no information available
Use of Arrow: (i) a performance change from last assessment , not reflected in the score (ii) a recent change in systems which is certain to have brought an improvement for which hard evidence is not yet available due to the time lag of obtaining such data – the use of arrows will increase as more repeat assessments come forward.
PEFA Secretariat - June 2011

10. PFM-PR – Assessment of PFM system
Indicator-led analysis
Description of actual performance based on factual evidence, listing sources of information& gaps
Scoring of indicator (by dimensions) with justification
Reporting on progress (for each indicator)
Description of recent or on-going reform measures, & indication if/when they are likely to change performance (even if not yet reflected in evidence, & hence scoring)
Country specific issues
Description of country characteristics relevant to understand functioning of PFM systems, e.g. public enterprises & revenue from extractive industries
This is the core of the report. Presenting evidence is key to a quality assessment.
Readers must understand what the assessors know and where they have knowledge gaps. Also what samples have been used to create an opinion about the total system where is includes many entities or fragmented systems e.g. PI-21 internal audit across MDAs or PI-18 across different payrolls for teachers, the army and other staff categories.
PEFA Secretariat - June 2011

11. PFM-PR – Summary assessment
The summary assessment brings together:
The assessment under each of the 6 critical dimensions of PFM system performance
The impact of PFM system performance on 3 budgetary outcomes
The Story Line
What is the story line, the number one message? - it may be all that readers remember!
Likely to be starting point for discussion of reform priorities
PEFA Secretariat - June 2011

12. PEFA Secretariat - June 2011
Content
What is PEFA?
Mechanics of the Framework
The Assessment process
Comparisons (time; countries)
Use and roll-out
PEFA Secretariat - June 2011

13. PEFA Secretariat - June 2011
Stages in a typical process
0. Agree the intention to undertake a PEFA based assessment
1. Agree purpose, scope and stakeholder roles
2. Prepare TOR
3. Mobilize assessment team
4. Introduction workshop for stakeholders
5. Review of existing information
6. Inception Report
7. Main field work
8. 1st Draft Report
9. Quality Review
10. Supplementary field work
11. Draft Final Report
12. Presentation seminar
13. Final report
14. Use of the report for reform dialogue
There is no “template” - each assessment should be driven by its own needs and requiremenets – this is just guidance and should not be seen as a blueprint.
Note:
The process presented start when an agreement in principle has been reached between a lead agency and the government on launching a PEFA based assessment. That process can itself be long and involve many steps.
The sheer number of steps in a typical process (in this simplified presentation)
This process does not include a government self-assessment and report, which may replace or run parallel to steps 5 and 6. Variation
The assessment team (consultants) may be needed in–country on 4 occasions (steps 4, 7, 10, 12). This is unfortunately often cut down in order to save on the budget. Variation
It may be useful to arrange a validation workshop with government as introduction to the supplementary field mission, step 10. Variation
PEFA Secretariat - June 2011

14. Requirements for a quality report
Adherence to PEFA methodology
Data / Information must be adequate & correct
Quality of Summary Assessment: strengths & weaknesses, implications for achieving budgetary outcomes
Structure, logic & language of report, to convey key messages
Timeliness & availability
Data / Information must be adequate and correct
The indicator ratings must be transparent and evidence based.
The indicator must be scored on the basis of information physically gathered.
Where there is a grey area or subjectivity applied in the evidence or in the calibration, the report should explicitly mention this, and offer the best estimate of the rating.
Summarizing strengths & weaknesses & implications on budgetary outcomes
A quality PFM report will capture and summarize the assessed strengths and weaknesses of the PFM system as identified by the performance indicators and appraise the extent to which the PFM system impacts on the achievement of outcomes of aggregate fiscal discipline, Strategic allocation of resources and efficient service delivery.
Structure, logic & language of the report
The report should be structured as closely as possible to the outline as set out in annex 2 of the Framework. The report should also be suitably referenced to the sources of information used. Where the same data source is used in a number of different places then cross referencing should be done to facilitate logical reading of the report.
Clearly conveying the key messages
The report needs to ensure that adequate clarifications are made on the outcomes that arise and that the key messages are captured into the priorities of the reform agenda.
PEFA Secretariat - June 2011

15. PEFA Secretariat quality review
Appraises application of PFM PIs & adequacy of background info (per chapter 1 & 2 of Framework)
Considers whether summary assessment brings out clear message consistent with PI analysis
Review of each PI
Correctly interpreted
Sufficient evidence
Scoring method correctly applied
Responses to review comments
PEFA involvement
When acting as Reviewers the PEFA Secretariat does not undertake an audit to establish the authenticity or correctness of the data given in the report. Reliance is placed on the integrity and professional competence of the consultants or team performing the assessment. However, where conflicting data or evidence becomes apparent then this will be pointed out and clarification sought.
The Secretariat’s review is undertaken by (i) appraising the application of the PFM performance indicators and (ii) considering whether the back-ground information as set out in chapters 1 & 2 of the framework has been given ; and whether the summary assessment brings out a clear message consistent with the indicator analysis.
PEFA Secretariat - June 2011

16. PEFA Secretariat - June 2011
Content
What is PEFA?
Mechanics of the Framework
The Assessment process
Comparisons (time; countries)
Use and roll-out
PEFA Secretariat - June 2011

17. Over time – what are we looking for?
Specific changes in system performance
What has changed?
How much?
But: changes may occur for other reasons than performance
Changes in definitions
Improved availability of or access to information
Different sampling
Different interpretation in borderline cases
Methodological errors in previous assessment
In order be able to properly track and understand PFM performance changes and to have confidence in the comparison of ratings of an indicator over time, it must be possible to understand from the report narrative what has changed and by how much.
PEFA Secretariat - June 2011

18. PEFA Secretariat - June 2011
Country comparisons
The PEFA Framework was developed to measure progress over time in one country – not for country comparisons
‘Summary assessment’ provides overview of strengths & weaknesses as basis for reform prioritization
No method given for arriving at single measure of ‘overall performance’ – no attempt to create ‘league table’
Comparing scores alone can be misleading
Must be approached very cautiously to ensure comparison of like with like
PEFA Secretariat - June 2011

19. PEFA Secretariat - June 2011
Content
What is PEFA?
Mechanics of the Framework
The Assessment process
Comparisons (time; countries)
Use and roll-out
PEFA Secretariat - June 2011

20. What can countries use the PEFA Framework for?
Inform PFM reform formulation, priorities
Monitor results of reform efforts
Harmonize information needs for external agencies around a common assessment tool
Compare to and learn from peers
PEFA Secretariat - June 2011

21. PEFA contribution to results orientation
Measuring changes
Results from 45 repeat assessments indicate pace of progress in different areas of PFM – global patterns emerging
Database available to researchers – almost 100 reports made public
Peer learning
Used as basis for discussions in regional groups of government PFM experts (e.g. Eastern Europe, West Africa, Caribbean)
From PFM assessment to reform program
PEFA assessment frequently used as basis for reform dialogue, but:
PEFA report is one of several inputs
Ownership means government decisions on priorities
Complementary analysis to PEFA required to identify root causes
Indicators used to set targets for reform project results
PEFA Secretariat - June 2011

22. Coverage of PFM-PR in Reform Cycle
Implement PFM reforms
High level performance overview
Formulate PFM reform program
A PFM Reform program should be formulated from recommendations arising from an investigation having been undertaken to establish the weaknesses in the performance of PFM systems, processes and institutions. This high level performance overview of the PFM system can be obtained from the application of the PEFA framework. The diagram illustrate the coverage of the PFM report in the PFM reform cycle and the elements which it covers. The PFM report will embrace the high level performance overview and identify the main PFM weaknesses.
PFM-PR
Identify main PFM weaknesses
Recommend PFM reform measures
Recommend PFM reform measures
Identify main PFM weaknesses
Investigate underlying causes
PEFA Secretariat - June 2011

23. Inform PFM reform formulation (1)
PEFA report is one of several inputs
Identification of main strengths and weaknesses – and potential impact on budgetary outcomes
Other factors: political economy, culture, constitution/legal, resources, capacity at entry
Ownership means government decisions on priorities
Government to consider all factors in deciding priorities
Ample space for government’s prioritization in the reform dialogue with International Agencies
Do not use Indicator scores simplistically
A low score is not sufficient justification for reform
Other factors: relative importance of subject, complexity /timeframe for improvement, interdependence with other elements
PEFA Secretariat - June 2011

24. Inform PFM reform formulation (2)
Complementary analysis to PEFA may be required
Detailed analysis of underlying causes needed for formulation of detailed action plan
Limit such analysis to priority areas
Drill-down tools – some exist, others under development
Guidance on using reports for reform formulation
How to establish if a reform program is ‘credible’
Challenge to develop general approach and toolkit to help government/donor teams identify priorities/sequencing
Such an approach could strengthen country ownership of reform and coordination of donor support
Work in progress
PEFA Secretariat - June 2011

25. Monitor results of reform efforts
Schedule full repeat assessments – every 3-5 years
Select a few indicators to monitor more frequently
Incorporate into the M&E component of the PFM reform program (Burkina, Kenya, Zambia)
Incorporate as monitoring tool in CAS (Bangladesh)
PEFA indicators being used for PFM reform program evaluations (IEG, Multi-donor evaluations)
PEFA Secretariat - June 2011

26. PEFA Secretariat - June 2011
Adoption of the PEFA Framework
Very good progress
Over 200 assessments substantially completed, covering almost 120 countries
High country coverage in many regions
Africa and Caribbean 90% of countries
Latin America, Eastern Europe, Asia, Pacific 45-65%
Wide stakeholder involvement
About 25 development banks & donor agencies involved (leading, financing, reference group)
World Bank & EC taken lead in 85% of all assessments
Government self-assessment increasing: not yet ‘norm’
PEFA Secretariat - June 2011

27. PEFA Secretariat - June 2011

28. PEFA Secretariat - June 2011
Stay in touch with PEFA
PEFA will be pleased to share its experience on development, implementation & use of the performance measurement framework
Visit our website:
Send us questions:
Get on our news distribution list: Send us your name and address
PEFA Secretariat - June 2011

29. Thank you for your attention
PEFA Secretariat - June 2011