1. 1 AN IMPACT EVALUATION OF PERFORMANCE BASED CONTRACTING FOR GENERAL HEALTH SERVICES IN RWANDA Evaluation team: Paulin Basinga Paul Gertler Jennifer Sturdy Christel Vermeersch Damien de Walque Project team Agnes Soucat Miriam Schneidman + team

2. 2 11/11/2015 Presentation plan  Country and program information  Impact evaluation design  Descriptive results from the baseline study

3. 3 SECTION 1: COUNTRY AND PROGRAM INFORMATION

4. 4 11/11/2015 RWANDA general & health sector  Total population : 9,038,001 (2005)  30 Administrative districts  GDP per capita of $230  33 District Hosp. and 369 Health centers  HDI:ranked 158th (2004)  MMR: 750 per 100,000 (DHS 2005)  IMR:86 per 1,000 (DHS 2005)  HIV: 3.1% (DHS 2005)

5. 5 11/11/2015 Relevance and Severity of the Health Issue Addressed  Diagnostic  Shortage of human resources for health services  Low levels of productivity and motivation  High levels of absenteeism  Low user satisfaction & poor quality of service lead to low use.  Increase morbidity and mortality  Goal  Increase number of trained medical personnel  Increase motivation  Improve quality of services  Increase personnel income  Policy Response  Performance Based Contracting & Financing

6. 6 National PBC model for Health Centers  16 Primary Health Care indicators, eg:  New Curative Consultation = $0.27  Delivery at the HC = $3.63  Completely vaccinated child = $ 1.82  14 HIV/AIDS indicators, eg:  One Pregnant woman tested (PMTCT) = $1.10  HIV+ women treated with NVP = $1.10  Separation of Functions between stakeholders

7. 7 11/11/2015 Fig: Implementing organizations

8. 8 SECTION 2 : IMPACT EVALUATION DESIGN

9. 9 11/11/2015 Hypotheses For both general health services and HIV/AIDS, we will test whether PBC:  Increases the quantity of contracted health services delivered  Improves the quality of contracted health services provided  Does not decrease the quantity or quality of non- contracted services provided,  Decreases average household out-of-pocket expenditures per service delivered  Improves the health status of the population

10. 10 11/11/2015 Evaluation Design  Make use of expansion of PBC schemes over time  The rollout takes place at the District level  Treatment and control facilities were allocated as follow:  Identify districts without PBC in health centers in 2005  Group the districts based on characteristics: rainfall population density livelihoods  Flip a coin to assign districts to treatment and control groups.

11. 11

12. 12 11/11/2015 Roll-out plan  Phase 0 districts (white) are those districts in which PBF was piloted  NOT part of the impact evaluation  Phase 1 districts (yellow) are districts in which PBF is being implemented in 2006, following the ‘roll-out plan’  Phase 2 districts (green) are districts in which PBF is phased in later; these are the so-called ‘Phase 2’ or ‘control districts’ following the roll-out plan. PBF is being introduced in these districts in 2008.

13. 13 11/11/2015 Program Implementation Timeline

14. 14 Quality assurance in comparisons  Law of large numbers does not apply here…  Proposed solution:  Propensity scores matching of communities in treatment and comparison based on observable characteristics  Over-sample “similar” communities in Phase I & Phase II  It turned out  Couldn’t find enough characteristics to predict assignment to Phase I  Took a leap of faith and did simple stratified sampling

15. 15 More money vs. More incentives  Incentive based payments increase the total amount of money available for health center, which can also affect services  Phase II area receive equivalent amounts of transfers  average of what Phase I receives  Not linked to production of services  Money to be allocated by the health center  Preliminary finding: most of it goes to salaries

16. 16 Surveys  General Health facility survey (168 centers)  General Health household survey  HIV/AIDS facility survey (64 centers)  HIV/AIDS household survey Note: HIV/AIDS study is for another presentation

17. 17 Design and sample 2159 Households 166 (All Health Facility surveyed) 18 Adm. District 30 Adm. Districts RWANDA TREATMENT DISTRICTS 83 HF 13-15 HH in the catchm. Area of each HF CONTROL DISTRICTS 83 HF13-15 HH 11/11/2015

18. 18 11/11/2015  Socio-economic information  Anemia finger prick test: children 12-71 months old  Malaria dip stick test: children under age 6  Anthropometrics: <6 years old  Mental health: mothers, pregnant women, adults over age 20  Sexual history and preventative behavior knowledge  Pre-natal care utilization and results  Parents or caretakers were asked for information regarding child (<5 years) health status General Health Household Survey: content

19. 19 11/11/2015 Health Centers Survey  All 168 centers in General Health; 64 in HIV/AIDS  General characteristics  Human resources module: Skills, experience and motivations of the staff  Services and pricing  Equipment and resources  Vignettes: Pre-natal care, child care, adult care VCT, PMTCT, AIDS detection services  Exit interviews: Pre-natal care, child care, adult care, VCT, PMTCT

20. 20 11/11/2015 Analysis Plan  All analyses will be clustered at the district level  Compare the average outcomes of facilities and individuals in the treatment group to those in the control group 24 months after the intervention began.  Use of multivariate regression (or non-parametric matching) : control confounding factors  Test for differential individual impacts by:  Gender, poverty level  Parental background (If infant : maternal education, HH wealth)

21. 21 SECTION 3: DESCRIPTIVE RESULTS FROM BASELINE SURVEY

22. 22 HEALTH FACILITIES BASELINE SURVEY

23. 23 Classification of facilities 11/11/2015

24. 24 Financial Capital Resources Main Sources of Funding at Facility Level Average Annual Funding at Facility Level RWF 2,416,597.00 RWF 727,169.90 RWF 2,069,715.00 RWF 297.47 RWF 295,745.90 Health District Religious Consultation Fees Drug Fees Lab Fees Main Sources of Funding 11/11/2015

25. 25 Human resources On average: 1 doctor for every 31,190 individuals, 1 nurse for every 4,835 individuals StaffObsvMeanStd dev.MinMax Nurses A11650.240.6605 Nurses A21645.322.74018 Social Ass A21650.160.93011 Lab tech1660.801.02010 Total staff16515.057.28343 11/11/2015

26. Availability of laboratory tests Phase I (Intervention district)Phase II (Control districts) Ttest Number of Obs. cstd Number of Obs. cstd Anemia8037.50%0.48717738429.76%0.4599571-1.046271 Pregnancy test8056.25%0.49920828434.52%0.4783014-2.84633 Blood test (types)802.50%0.15711843.57%0.18669150.396673 Blood count801.25%0.1118034843.57%0.18669150.9601514 Urine test801.25%0.1118034842.38%0.1533710.5374025 Urine dipstick8050.00%0.50315468460.71%0.49131881.379621 Stool culture808.75%0.2843491843.57%0.1866915-1.384994 Parasites test7789.61%0.30712668494.05%0.23802351.029283 Malaria (blood smear)8050.00%0.50315468453.57%0.50171820.4550284 Malaria dipstick7955.70%0.49991888472.62%0.44859062.277273 Tuberculosis test8036.25%0.48375518440.48%0.4937940.5533122 HIV test8062.50%0.48717738455.95%0.4994259-0.8493115 Gonorrhea gram stain 8018.75%0.39277498421.43%0.41279040.4252997 Syphilis TPHA8010.00%0.3018928848.33%0.2780454-0.3679879 Syphilis RPR8065.00%0.47997898447.62%0.5024324-2.263157 Pap test8038.75%0.49025318436.90%0.4854451-0.2421459 11/11/2015

27. Availability of equipment Phase I (Intervention district)Phase II (Control districts) Ttest Number of Obs. cstd Number of Obs. cstd avail_Stethoscope7898.72%0.1184100.00%01.037999 avail_Obstetric_stet79100.00%08498.81%0.10-0.9696023 avail_Bloodpressure_8097.50%0.158498.81%0.100.6224446 avail_Autoclaves8077.50%0.428472.62%0.44-0.7182782 avail_Scales_adults8062.50%0.488467.86%0.460.7166591 avail_Scales_child8070.00%0.468479.76%0.401.443468 avail_Otoscope8072.50%0.448472.62%0.440.016974 avail_Thermometers8098.75%0.1184100.00%01.024853 avail_Timer8090.00%0.308483.33%0.37-1.250493 avail_Patient_beds7989.87%0.308389.16%0.31-0.1478858 avail_Other_beds7964.56%0.488366.27%0.470.2271164 avail_Ventouse8021.25%0.418429.76%0.451.24663 avail_Aspirators7948.10%0.508340.96%0.49-0.9105169 avail_Curetage8010.00%0.308420.24%0.401.830757 avail_Speculum8083.75%0.378478.57%0.41-0.8433283 avail_Microscopes8095.00%0.218496.43%0.180.4499004 avail_Centrifuges8068.75%0.468469.05%0.460.0409054 avail_Refridge8078.75%0.418375.90%0.43-0.431276 avail_Freezers7931.65%0.468435.71%0.480.5461755 avail_Stove8013.75%0.348314.46%0.350.1289797 avail_Coldbox8098.75%0.118391.57%0.27-2.139053 avail_Sharpdisposal7981.01%0.398490.48%0.291.73995 avail_Examinationtab8097.50%0.158494.05%0.23-1.090562 avail_Ambulance8016.25%0.378411.90%0.32-0.7976888 11/11/2015

28. Availability of drugs Phase I (Intervention district)Phase II (Control districts) Ttest Number of Obs. cstd Number of Obs. cstd Amoxycilline8096.25%0.198496.43%0.180.060 Cotrimoxazole80100.00%08496.43%0.18-1.710 Ampicilline8063.75%0.488472.62%0.441.218 Peni_procaine8096.25%0.198495.24%0.21-0.318 Peni_benzathine8097.50%0.158494.05%0.23-1.090 Gentamycine8052.50%0.508445.24%0.50-0.926 Ceftrioxone806.25%0.24844.76%0.21-0.415 Ocytocine7884.62%0.368473.81%0.44-1.692 ….. ….….. Nevirapine7652.63%0.508339.76%0.49-1.630 Efavirenz7819.23%0.39827.32%0.26-2.252 Zidovudine7815.38%0.36824.88%0.21-2.234 Laminvudine7817.95%0.38827.32%0.26-2.046 Didanosine773.90%0.19821.22%0.11-1.074 Stavudine7817.95%0.38827.32%0.26-2.046 Abacavir786.41%0.24821.22%0.11-1.732 ….. 11/11/2015

29. 29 Prenatal Care procedures: Comparison between health provider & patient responses 92% 86% 80% 21% 18% 13% 64% 85% 21% 17% 8% Number of previous miscarriages Number of previous pregnancies Late menstrual dates Lost/gain of weight, nausea, vomiting Medicaments that is taking right now Family history of genetic problems HealthProviderPatient 11/11/2015

30. 30 Prenatal care Physical Examination Comparison between health provider and patient responses 52% 58% 72% 81% 86% 83% 19% 17% 97% 61% 93% Weight measured Pelvic examination Height measured Abdomen examination Blood presure measured Listen featal heatbeat Health Provider Patient 11/11/2015

31. 31 Quality index: Descriptive statistics INDEXMeanSEMinMax Process quality Prenata care_Vignette0.480.0020.170.83 Child care_Vignette0.440.0020.200.71 Exit interv satisfaction0.800.0040.001.00 Structural quality HF Infrastructure0.760.0040.171.00 HF Equipment0.740.0030.321.00 HF: Laboratory0.520.0050.001.00 11/11/201531

32. 32 PATIENT SATISFACTION  Negative significant correlation between satisfaction and: The length of wait time Total time spent attending the facility, The cost of the visit The cost of the medications.  No difference between satisfaction of mutuelle members and non-members But mutuelle membership is significantly positively correlated with cost  Prenatal visit: positive correlation with the cleanliness of the facility  Child care: lower levels of satisfaction in Waiting time, Total time spent for visit, Cleanliness of the facility, Privacy during the exam, Attitude of the staff, Explanation of the child’s condition during the exam 11/11/2015

33. 33 HOUSEHOLD BASELINE SURVEY

34. 34 Sample description  2159 HH, 10,880 individuals  Gender balance (49,3% male and 50,7 female)  Average HH size is 5.71 individuals,  Age range of <12 months to 96 years old.  Half of the sample is below 12 years old  75% of the sample is under the age of 30 years old. (Sampling strategy) 11/11/2015

35. 35 HH Education by phase Phase I (Intervention district) Phase II (Control districts) Variableobs%stdobs%std.tstat Ever_attended_school325879.60.402417280.60.3951.048 No_schooling258810.50.307335611.50.3191.208 Primary258882.10.383335679.80.401-2.227 Secondary_above25886.90.25533567.90.2701.309 11/11/2015

36. 36 Household assets Phase I (Intervention district)Phase II (Controldistricts) VariableObs StdObs stdtstat Number of rooms in dwelling9603.591.0911733.391.16-4.12 Complete sofa set9601.88%0.1411732.90%0.171.53 Radio96049.90%0.50117350.04%0.500.07 Radio-cassette or music system9603.65%0.1911735.80%0.232.31 Telephone –9601.04%0.1011733.32%0.183.51 Mosquito nets96022.50%0.42117328.73%0.453.27 Sewing machine9600.83%0.0911731.19%0.110.82 A bed96063.23%0.48117358.14%0.49-2.39 Wardrobe9604.38%0.2011735.54%0.231.23 Metallic library9600.63%0.0811731.36%0.121.68 Table96063.02%0.48117362.49%0.48-0.25 Chair96085.31%0.35117383.89%0.37-0.91 Car9600.63%0.0811730.43%0.07-0.64 Motorcycle9600.42%0.0611730.51%0.070.32 A bicycle96017.92%0.38117316.88%0.37-0.63 Dwelling - High Quality Materials9600.21%0.0511730.17%0.04-0.20 Dwelling – Low Quality Materials96044.17%0.50117338.45%0.49-2.67 Uses Electrogaz for Cooking (No)96099.38%0.08117399.57%0.070.64 Main source of lighting is electricity9560.84%0.0911701.37%0.121.15 Well water95471.91%0.45117262.97%0.48-4.38 Other source of water95419.81%0.40117225.26%0.432.98 Open source of water9548.07%0.27117211.35%0.322.52 Owns flush toilet9600.10%0.0311730.26%0.050.80 Owns latrine96080.63%0.40117376.47%0.42-2.32 36

37. 37 Maternal health service utilization Age PNC provide with the last child Preventive injection for Tetanus Assisted delivery Delivery at the health facility <20 28 (98.86%)27 (77.78%) 13 (43.33)15 (50%) 20-34 1154 (96.7%)1117 (84.15%) 595 (29.04)664 (31.91) 35+ 515 (94.95%)489 (46.01%) 153 (20.45)162 (21.34) Overall 1,681 (95.84)1,227 (72.86) 837 (26.01)929 ( 28.47) 11/11/2015

38. 38 Maternal health service utilization Phase I (Intervention district) Phase II (Control districts) obs PercStd eobs PercStd etstat Assisted delivery 149224.660.01172627.170.011.61 Delivery at the health facility 150827.40.01175529.30.011.19 Times received PNC 752 2.7811.1389272.7091.159-1.274 Preventative injection for Tetanus 75773.45%0.44292772.38%0.447-0.488 11/11/2015

39. 39 Immunization Phase I (Intervention district) Phase II (Control districts) Obs PercStdObs PercStdtstat Fully immunized ( 12-23 months old) 27273.53%0.44230276.49%0.425-0.818 Fully immunized ( 12-71 months old) 122561.14%0.488145964.08%0.4801.570 11/11/2015

40. 40 Conclusion  HH Results comparables to the recent DHS  Validity of treatment – control groups  Of 110 key characteristics and output variables of HF, the sample is balanced on 104 of the indicators.  Of 80 key HH output variables, the sample is balanced on 73 of the variables.  Majority of the indicators which differ between Phase I and Phase II are results from patient exit interview 11/11/2015