Pay for performance initiatives a growing trend, but are they effective? Josephine Borghi London School of Hygiene & Tropical Medicine Ifakara Health Institute

Objective Provide an overview of the challenges in measuring the effectiveness of PBF – PBF refers to a diverse set of interventions – Implementation may not match the ideal ‘vision’ – Need a better understanding of the health system impact of PBF – What kinds of evaluation should we be doing? – What is feasible and politically acceptable?

Performance based financing – A range of applications Based on who is the recipient of the bonus payment – Donor to government (aid modalities) – Within the public sector (facilities or districts / regions) – Government/donor to non-state provider – Health workers Process or outcome indicator/target Payment method and size When we say PBF is effective what do we mean?

Implementation – potential constraints in low income settings Design constraints – Complexity versus simplicity – designing a simple system which is effective? Roll out constraints – Logistics Timely availability of funds Appropriate management of information Training of managers and health workers Monitoring constraints – Need for external validation - costly – Lack of reliable HMIS data

Tanzanian experience – vision versus reality Ideal Vision for P4PActual policy design HMIS strengthened alongside P4P HMIS not strengthened – no baseline Flexibility in way incentives are structured by district No flexibility – single system applied everywhere Financial autonomy at facility level Funds managed at district level Variable targets based on baseline performance Absolute targets irrespective of baseline performance Phased implementation to allow to learn from experience and fine tune policy National roll out in order to ensure rapid impact

PBF: Impact on the Health System? Potential Risks Understanding the processes of change as well as the impact on health outcomes

Potential risks Impact on health worker behaviour – Supplier induced demand – unnecessary service delivery – Quantity over quality – Opportunity cost – focus on targeted conditions Equity effects – Patients: Focus on easy to reach groups – Geographic: areas where health system stronger may have advantage – Health worker migration

Need for improved evaluation design

Key Components of an Evaluation Process Evaluation What is the status of implementation? How does it vary by area? What facilitates and impedes implementation? Impact Evaluation What changes take place that result in the observed impacts? Economic Evaluation What are the ‘positive’ effects of PBF on coverage of health services? Are there any unintended consequences of PBF? Is PBF a good use of resources? Need for qualitative assessment to determine causal pathway What are the health benefits of PBF?

An evaluation framework based on Tanzania

Introduction of PBF Training of key stakeholders and provision of guidelines Bonus payment to Providers Bonus payment to District and Regional stakeholders Positive EffectsNegative Effects Re-allocate resources Improve quality of care / increased patient satisfaction Increased utilisation of targeted health services Reduced maternal and newborn mortality Increased motivation and trust Coercive strategies to increase utilisation Mis-reporting performance Crowding out of non-targeted health services / reduced quality of care Damage intrinsic motivators Recommendations for improved performance

Impact Evaluation Design Options Probability design Plausibility design Adequacy design Randomised Control Strongest evidence of impact Political acceptability Non-randomised Control Interrupted time series Need to control for inherent differences Need good historical data Historical or no control Weakest evidence of impact

Practical challenges and need for innovative thinking Political resistance to controls Randomisation may not be possible – Randomisation of performance contracts (Rwanda) In absence of control or historical data importance of documenting the process

Summary Need for better understanding of which aspects of PBF are effective and pre-conditions for their effectiveness What is implemented may not match what was intended, need for simple and saleable models Some unanswered questions on health system impact Need for process evaluations and more impact evaluations within realm of political feasibility