1. PUBLIC EXPENDITURE AND FINANCIAL ACCOUNTABILITY (PEFA)-PERFORMANCE MEASUREMENT FRAMEWORK Module 5: Understanding PEFA Scores and Analyzing a PEFA Report

2. OUTLINE OF MODULE 5 Part 1: Scores and PEFA report; Part 2: The framework and some ‘‘weaknesses’’.

3. Part 1: Scores and PEFA report.

4. 4 The scorings of a PEFA Assessment Can be compared from period to period (given its reliance on high level indicators, this only in the medium term 3 to 4 years) Should not be averaged to a single score in the same way that the strength of a foundation cannot be expressed as a single average figure The scorings incorporate some judgement and are sometimes specific to a specific PFM system (e.g PI-5) therefore, though tempting, it is not instructive to compare results between countries Don’t: Attempt to average the scorings into a single score (remember the Tower of Pisa). Note that this was done in Mozambique and done for a dimension in Ghana

5. 5 The summary assessment Should tells us about How far the country has gone towards achieving aggregate fiscal discipline, strategic allocation of resources, efficient delivery of service (PEFA identifies this as area of weakness) How well each of six dimensions of PFM and donor practices are operating – credibility of the budget, comprehensiveness and transparency, policy based budgeting, predictability and control in budget execution, accounting and financial reporting, external scrutiny and oversight (more straight forward to derive from indicator scorings Ongoing PFM reforms

6. 6 The summary assessment (2) Aggregate fiscal discipline (spending only what you have) Credibility of the Budget scores well (PI-1 to PI-4); Oversight of fiscal risk scores well (PI-9) Cash and Debt management score well (PI-16, PI-17); No substantial losses in payroll and procurement (PI-18, PI-19) Pre-requisites include: effective expenditure controls (PI-20), effective internal audit (PI-21), timely, comprehensive, regular and accurate financial reporting (PI-22, PI-24,PI-25), effective external audit Efficient Delivery of Services (requires a well organised and functioning market) Achieve aggregate fiscal discipline; Have an orderly budget process (PI -11); Achieve predictability in budget releases (PI-13 to PI-15, PI-16); Achieve value for money (PI-19) Strategic allocation of resources (requires policy based budgeting) Achieve the efficient delivery of services; Budget classification to support policy based budgeting (A or B in PI-5); Policy bodies involved in budget cycle (A or B in PI-11) Multi-year perspective in fiscal planning and expenditure policy (PI-12)

7. 7 Interpreting scores for PI-1 to PI-4 (Budget credibility) If a budget is credible then it should score A’s on PI-1 to PI-4; however, that a country scores A’s does not necessarily mean that that the budget is credible. It depends upon the accuracy (and comparability) of the expenditure data. Consider: Are the expenditure out-turns consistent with the revenue outturns (PI-3), the change in accrued arrears (PI-4), the debt service (fiscal table Section 2), and the performance on direct budget support D1 (if applicable) The general performance for predictability and control in budget execution especially with regards to cash management and commitment control, and revenue collection in the case of PI-3. Whether or not there are extra-budgetary transactions PI-7 and how that should impact indicators PI-1 to PI-4 Is there regular account reconciliation and so can expenditure data be counted upon. Are internal audit and external audit effective.

8. 8 Interpreting scores for PI-5 (Budget classification) The scoring for PI-5 reflects the choice of a country to adopt performance or programme based budgeting rather than necessarily the quality of the budget classification: Therefore: A fair score (C) may be appropriate for a country whose PFM reform is still focussed upon aggregate fiscal discipline A very good score (A) may be partially responsible for weaknesses in in- year budget execution reports, delays in final accounts, weaknesses in expenditure controls, and delays in audited financial statements

9. 9 Interpreting scores for PI-7 (Extra-budgetary operations) - There is a logical conundrum presented by PI-7 – possibly concluding that there are no extra-budgetary transactions because none were found. Except where substantial extra-budgetary transactions are identified it is difficult to be certain that a high score proves little or no unreported government operations. - With increasing donor activities from Brazil, Russia, India, China & South Korea and others including Venezuela who are not usually engaged with the PFM Donor working groups there could be substantial donor activity that is not captured by this indicator unless the consultants actively pursue and are successful in gaining reliable information on donor funded projects

10. 10 Interpreting scores for PI-13 through PI-15 (Overall revenue administration) - The scorings for PI-13 to PI-15 must be derived from an aggregation of all the major revenue agencies some of which may not strictly be tax authorities and so a strict translation of the scoring criteria may not always be applicable. - The way that scores are aggregated for the different tax institutions (e.g. customs, VAT and Tax Authority) may not always be consistent and may vary from assessment to assessment. - Further, where the customs is a major source of revenue and the country participates in a customs union it becomes difficult to apply the scorings in a consistent way.

11. Interpreting scores for PI-18 (Payroll control) - The model for achieving effective payroll control is based upon a direct link between a personnel database and the payroll. There are arguments to be made that this may be incomplete and that it would also require incorporation of control of the personnel database by a database of fully funded posts ensure effective control. It may be argued then that an A PI-18(i) does not necessarily imply effective payroll control. - Without actual testing of the data it is difficult to ascertain the frequency and timeliness with which changes are made to personnel records and payroll in a decentralised payment system. Thus the interview method of obtaining evidence may be insufficient. 11

12. 12 Interpreting scores for PI-20 (Commitment control) - It is difficult to comment upon the degree of compliance with rules fro processing and recording transactions using a high-level approach and without resorting to carrying out a direct compliance test. Except where these are carried out routinely across a representative sample of spending units and consolidated into reports it is difficult (but not impossible) to measure this indicator with any meaningful accuracy or reliability. Except where there is a centralised computerised system which integrates the majority of the spending units, it becomes difficult to ascertain the scope of application of effective commitment controls except where this is reported upon in audit reports or other specific reviews.

13. 13 Interpreting scores for D-2 (Reporting on project aid) - The financial reporting of expenditure by donors, while timely and accurate, may not always be consistent with the Governments accounting standards and practices. - Where the country adopts a cash basis and the donor an accrual basis for example, the submission and consolidation of figures can lead to distortions. - Very few donors report on aid using the country’s budget classification

14. Part 2: The framework and some ‘‘weaknesses’’ -

15. Challenges in Scoring Assignments There are challenges in scoring the indicators/dimensions. Two such examples are developed in detail in the following slides. Note: This is may be relevant for policy dialogue and for managing expectations with respect to PFM performance progress over time. 15

16. Scoring Options (e.g. PI-23) How should the combinations be scored or Non Scored (NS)? 16

17. Scoring Options (e.g. PI-17(i) How should the combinations be scored or NS? 17

18. PEFA Phase IV (2012-2017) - Comprehensive review and revision of the Framework while recognizing need for comparability over time 18

19. 19 Developing a Healthy Skepticism Reader of a PEFA report: - Needs not to interpret the scorings too literally; - Should rely on the narrative; - Should also rely on an interpretation based upon a consideration of the consistency between the different indicators; - Should not conclude that a country with poor scorings has a poor PFM system; - Should not think that high scores automatically mean that a country has a good PFM system (although this possibility is not excluded either).

20. 20 CONCLUSION The PEFA Report is powerful, convenient and useful; However: 1.1) It cannot and should not be read and interpreted like a school report card. It can be and should be used to monitor and track progress (movement?) in PFM performance; 2) It requires that both the scorings along with the narrative be carefully considered; 3) Individual scorings are more useful when considering them for consistency with other scores; 4) Scoring is not always straight forward but the framework is being revised..

21. THANK YOU FOR YOUR ATTENTION