1 1 IMPACT EVALUATION OF PERFORMANCE BASE FINANCING A collaboration between the Rwanda Ministry of Health, CNLS, and SPH, the INSP in Mexico, UC Berkeley and the World Bank FOR GENERAL HEALTH AND HIV/AIDS SERVICES In RWANDA 1

2 2 2 Presentation plan 1. Country profile 2. Program description 3. Impact evaluation design 4. Impact evaluation implementation 5. Lessons learned

3 2/7/2014 RWANDA general & health sector Total population : 9,038,001 (2005) 30 Administrative districts GDP per capita of $ District Hosp. and 369 Health centers HDI:ranked 158th (2004) MMR: 750 per 100,000 (DHS 2005) IMR:86 per 1,000 (DHS 2005) HIV: 3.1% (DHS 2005)

4 2/7/2014 Relevance and Severity of the Health Issue Addressed Diagnostic Shortage of human resources for health services Low levels of productivity and motivation High levels of absenteeism Low user satisfaction & poor quality of service lead to low use. Increase morbidity and mortality Goal Increase number of trained medical personnel Increase motivation Improve quality of services Increase personnel income Policy Response Performance Based Contracting & Financing

5 5 RWANDA Performance Based Financing (PBF) Raises the quantity and quality of health services provided Increases health worker motivation through a system of incentives payments based on performance Operates through contracts between the government and other partners (providing the financing) and health facilities (providing services)

6 6 History of PBF in Rwanda Three pilot schemes: Cyangugu (since 2001) Butare (since 2002) BTC (since 2005) National model implemented in 2006

7 7 National PBF Model: Creation of CAAC Cellule dAppui a lApproche contractuelle (MOH PBF department) : Dec 05 – April 06 All stakeholders involved through Technical Working Group: CAAC, Cordaid, HNI, MSH, BTCCTB, WB, SPH Complex processes as stakeholders tried to advocate for the adoption of their model (indicators, quality measure, payment and supervision) Finally, a common national model was chosen

8 National PBC model for Health Centers 16 Primary Health Care indicators, e.g.: New Curative Consultation = $0.27 Delivery at the HC = $3.63 Completely vaccinated child = $ HIV/AIDS indicators, e.g.: One Pregnant woman tested (PMTCT) = $1.10 HIV+ women treated with NVP = $1.10 Separation of Functions between stakeholders

9 2/7/2014 Implementing organizations

10 2/7/2014 Hypotheses For both general health services and HIV/AIDS, we will test whether PBC: Increases the quantity of contracted health services delivered Improves the quality of contracted health services provided Does not decrease the quantity or quality of non- contracted services provided, Decreases average household out-of-pocket expenditures per service delivered Improves the health status of the population

11 2/7/2014 Evaluation Design Make use of expansion of PBC schemes over time The rollout takes place at the District level Treatment and control facilities were allocated as follows: Identify districts without PBC in health centers in 2005 Group the districts based on characteristics: rainfall population density livelihoods Flip a coin to assign districts to treatment and control groups.

12

13 2/7/2014 Roll-out plan Phase 0 districts (white) are those districts in which PBF was piloted NOT part of the impact evaluation Phase 1 districts (yellow) are districts in which PBF is being implemented in 2006, following the roll-out plan Phase 2 districts (green) are districts in which PBF is phased in later; these are the so-called Phase 2 or control districts following the roll-out plan. PBF is being introduced in these districts in 2008.

14 2/7/2014 Program Implementation Timeline

15 Quality assurance in comparisons Law of large numbers does not apply here… Proposed solution: Propensity scores matching of communities in treatment and comparison based on observable characteristics Over-sample similar communities in Phase I & Phase II It turned out Couldnt find enough characteristics to predict assignment to Phase I Took a leap of faith and did simple stratified sampling

16 More money vs. More incentives Incentive based payments increase the total amount of money available for health center, which can also affect services Phase II area receive equivalent amounts of transfers average of what Phase I receives Not linked to production of services Money to be allocated by the health center Preliminary finding: most of it goes to salaries

17 The baseline has 4 surveys December 2005-March 2006: General Health facility survey (166 centers) General Health household survey (2,016 HH) August – November 2006: HIV/AIDS facility survey (64 centers) HIV/AIDS household survey (1994 HH)

18 Design and sample (General health) 2159 Households 166 (All Health Facility surveyed) 18 Adm. District 30 Adm. Districts RWANDA TREATMENT DISTRICTS 83 HF HH in the catchm. Area of each HF CONTROL DISTRICTS 83 HF13-15 HH 2/7/2014

19 Follow-up surveys February-September surveys: Combined health facilities survey for General Health - HIV/Aids Household survey for General Health (panel data) Household survey for HIV/AIDS (panel data) Data cleaning ongoing. Preliminary results will be available in early 2009

20 Lessons learned (1) This impact evaluation was designed prospectively and was started early enough for baseline data to be collected, before the scale-up of the intervention. Evaluating the impact of such an intervention is almost impossible without planning the evaluation before launching the intervention.

21 Lessons learned (2) Many impact evaluations focus on small-scale, localized pilot-projects who benefit from a lot of attention and supervision. The impact evaluation of PBC in Rwanda covers almost the entire country and will assess the impact of the intervention under "real" conditions. While it might be good to start with modest pilot- projects, it is also crucial to be able to assess whether an intervention is effective when scaled-up at country level, under realistic and sustainable conditions.

22 Lessons learned (3) Such a country-wide impact evaluation requires that the impact evaluation is strongly owned by the Government. In the case of Rwanda, the Government has been in the driving seat, by asking for the impact evaluation, making sure that the general health and the HIV/AIDS sector evaluation were well synchronized and requesting and enforcing a strong coordination among the donors. The result was a impact evaluation design and a geographical and chronological roll-out plan of PBC in Rwanda that was carefully adhered to by all stakeholders.

23 Lessons learned (4) An interesting feature of the evaluation in Rwanda is that it will try to measure impact not only on processes such as number of visits to the health centers, number of vaccinations, but also by directly measuring health outcomes such as malaria and anemia tests, anthropometrics among the target population. The ultimate objective of performance-based contracting is to improve the population's health status and the impact evaluation will explicitly measure it.