1. PEFA FRAMEWORK FOR ASSESSING PUBLIC FINANCIAL MANAGEMENT Module 9: Comparisons over time & between countries

2. Content Comparisons over time (repeat assessments) Comparisons between countries 2

3. Content Comparisons over time (repeat assessments) Comparisons between countries 3

4. Repeat Assessments At December 2015, more than 150 repeat assessments undertaken More planned, i.e. 3-4 years after the first series of baseline assessments But – Framework has been upgraded!

5. What do we want to determine? Specific changes in system performance What has changed? How much? Indicator scores will provide a crude overview of changes over time, but: Dimensions ratings may change differently Performance may not always change enough to change the score So more detailed explanation required

6. Non-performance reasons why scores may change Changes in definitions Improved availability of or access to information Different sampling Different interpretation in borderline cases Scoring methodology mistakes in previous assessment

7. If you find issues... Avoid temptation to re-rate previous assessment! Explain that: present & previous ratings are not comparable, & why different view in previous assessment may have influenced conclusions about direction Now using 2016 upgrade

8. Comparability: Dimension level 2016Comparability2011 14Scores directly comparable – same subject, data & calibration 14 24Score indirectly comparable – same subject & data24 25Subject only – indicator subject has been retained, but scope, requirements or calibration are different & score is not comparable 28 -Not comparable – subject of dimension discontinued 10 31Not comparable – new subject of a dimension (or substantially different) - Total 94 Total 76 8

9. Performance Report 4 – Conclusions from analysis of PFM system 4.1:Integrated assessment across indicators (previous ‘Summary Assessment’) 4.2:Effectiveness of internal control framework 4.3: PFM strength/weakness, related to 3 budgetary outcomes 4.4: Performance changes since previous assessment (linked to Annex) 9

10. Reporting on progress made Explain all factors that impact a change in rating indicator-by-indicator Identify the performance change Ensure that any reader can track the change from the previous assessment – what was performance change that changed the rating?

11. Transitional arrangements - 4.4: Changes since previous assessment For comparisons with previous assessments using PEFA 2005 or 2011, supplementary ‘Annex 4’ required to show what scores WOULD HAVE BEEN using earlier PEFA on current data (recalibrating previous assessment using PEFA 2016 NOT recommended) Main performance changes between assessments, based on Annex 4, should be outlined in executive summary & discussed in more detail in section 4.4 11

12. Good practice for managers /sponsors: CN/TOR stage Performance tracking clearly reflected in TOR Facilitate access to documentation from previous assessment Agency leading the assessment, if different, should be part of reference group for repeat Use of same assessment team desirable, but rarely possible

13. Good practice for assessors Preparation: obtain previous report, comments from peer reviewers; don’t assume errors or perfection! Field work Verify basis on which earlier score was assigned, but do not attempt to re-rate Drafting the report If mistakes found, explain that present & previous ratings are not comparable

14. Content Comparisons over time (repeat assessments) Comparisons between countries 14

15. Country Comparisons PEFA Framework was developed to measure progress over time in one country – not for Country Comparisons ‘Summary assessment’ to provide nuanced overview of strengths & weaknesses as basis for reform prioritization No method given for arriving at one measure for ‘overall performance level’ No attempts to create global performance list 15

16. Country Comparisons Country comparisons are an emerging issue due to demand from: Governments – to know how they perform compared to their peers Researchers – to identify global or regional PFM strengths and weaknesses; & country examples of strong performance in select areas Donors – for considering technical assistance to PFM & for aid allocations 16

17. Country data and how to use it Comparison of two countries must be done very cautiously: Resembles comparison of assessments over time in one country but more complex Technical definitions may be different Need to carefully read each report to understand performance differences behind the scores Consider country context, ensure comparison of like with like Comparing scores alone can be misleading 17

18. Issue of Guidance Note Guidance issued as Exposure Draft in March 2008 Publicized on www.pefa.org Comments received from practitioners and field work Final guidelines issued in 2013 – see website

19. Thank you for your attention: Questions?