1. RAcE Niger Final Evaluation Results
RAcE 2015 Programme
Multi-Country Results Dissemination Meeting
25 October 2017

2. RAcE Niger Final Evaluation Results - Overview
To demonstrate the plausible contribution of the RAcE project to changes in treatment coverage indicators and estimated mortality change, ICF assessed project and state-level data, estimated the change in child mortality in RAcE project areas using LiST, and documented contextual factors that may have influenced child health in project areas.
Here we present findings that answer two evaluation questions:
Was there a reduction in childhood mortality, and were the lives of children ages 2–59 months saved, in the RAcE project area?
What was the RAcE project’s contribution to the estimated changes in mortality?
In this presentation I will share some high level evaluation findings, focusing on specifically on findings of the LiST analysis and plausible contribution of RAcE. Due to time limitations I will not be going through all evaluation findings.
This is not an evaluation of the entirety of RAcE project inputs to strengthening health systems, including community-level health service delivery. This evaluation does not assess the second objective of the RAcE programme: stimulate policy review and regulatory update in each country on disease case management.

3. Summary of Key Evaluation Findings (1)
Caregiver’s trust in Relais Communautaire (Rcom) and belief that RComs provide quality services nearly universal.
Steady increase in the number of malaria, pneumonia, and diarrhea cases treated, corresponding to increase of active and functional RComs.
Level of education (writing proficiency) the biggest challenge to identifying, recruiting, and maintaining qualified RComs.
The majority of respondents believed that RComs provided quality services and were a convenient source of treatment. At endline, caregiver’s trust in RComs was nearly universal: 99 percent of caregivers viewed RComs as trusted health providers, and 98 percent believed that RComs provide high-quality services. At the end of the project, 88 percent of caregivers cited the RCom as a convenient source of treatment, and 74 percent of caregivers with a child who had been sick in the two weeks preceding the survey found the RCom at first visit.
These positive perceptions of RComs likely contributed to increased care-seeking from RComs for sick children under 5 years of age.
Overall, the percentage of sick children for whom advice or treatment was sought from an appropriate provider increased from 69 percent at baseline to 85 percent at endline (p<0.02). By placing RComs in hard-to-reach communities as an extension of health services, care-seeking from RComs was high at endline; the percentage of cases of illness among children ages 2–59 months taken to an RCom as the first source of care was 75 percent at endline.

4. Summary of Key Evaluation Findings (2)
Regular supervisory visits (at least once every three months) likely contributed to improvement in quality of RComs services and data.
RAcE project filled the gap of medicine and supplies through iCCM services provided by RComs.
RComs experienced minimal stockouts throughout project implementation.
MSP developed a scale-up strategy, with WHO support, to integrate iCCM services for children’s diseases in the national health system.
Throughout the project implementation period, coverage of supervision of RComs was very high—the percentage of RComs who received at least one supervisory contact during the previous three months in which records were reviewed was 100 percent for all three years. For most years, this indicator reflects supervision efforts by World Vision and MSP, combined. In the first two quarters of Year 4, 45 percent of RComs received supervision visits from MSP alone. World Vision’s provision of drugs and supplies directly to RComs during routine supervision resulted in low levels of stockouts throughout project implementation.
World Vision’s provision of drugs and supplies directly to RComs during routine supervision resulted in low levels of stockouts throughout project implementation.
To support the scale-up, a national coordination committee on community health has recently been formed to better structure iCCM health interventions. The draft iCCM scale-up strategy plan (2017–2021) is currently being reviewed by all partners and stakeholders.

5. Estimated Change in Child Mortality in RAcE Niger Project Areas
The LiST model estimated results based on the total population of the four RAcE project districts eligible for iCCM (994,904).
Estimated change in U5MR in the project area:
17 deaths per 1,000 live births
13 percent decrease in U5MR from 2013 to 2016.
Table 1. Estimated mortality rates modeled in LiST for each project year.
RAcE Niger
Year
Under-five mortality rate (deaths per 1,000 live births)
2013
137
2014
132.46
2015
125.87
2016
119.77
The LiST model estimated results based on the total population of the four RAcE project districts not covered by health services, and consequently eligible for iCCM, which was estimated to be 994,904. RAcE intervention area population as reported by World Vision and included in the first grant agreement letter between WHO and World Vision signed in June The intervention area population was based on 2011 data from the MSP, including total general population of each district and the percentage of health coverage in each district.

6. Estimated Lives Saved in RAcE Niger Project Areas
An estimated total of 1,128 under-five lives saved by pneumonia, diarrhea, and malaria treatment from 2013 to 2016.
An estimated 965 lives were saved due to treatment provided by RCom.
Table 2. Estimated number of child lives saved per year by treatment interventions in RAcE Niger project areas
RAcE Niger
2013
2014
2015
2016
Total
Percentage intervention treatment by RComs
Estimated lives saved by RCom-provided treatment
Total lives saved among children 1–59 months (all interventions)
200
475
793
1,468
Intervention
Estimated lives saved
ORS 92
188
288
568
86%
488
Zinc for treatment of diarrhea 29 59 91
179
89%
159
Oral antibiotics for pneumonia 9 18 27 54
55% 30
ACTs 55
109
163
327
88%
1,128 -
965
Because the LiST model uses the coverage estimates for the baseline year to estimate lives saved for the model’s projection period, the estimated lives saved for the baseline year is by default zero lives saved.

7. LiST Model Limitations
The accuracy of the model results is limited by the data input to the model.
LiST does not account for the mode of delivery or source of care (with the exception of facility birth).
LiST model does not account for changes in diagnostics, the quality of care, timeliness of pneumonia and diarrhea treatment, nor referrals made or completed. 
Limitations of some of the data inputs:
For other interventions, except immunization, the model and results assume that areas had the same intervention coverage reported in service statistic data for RAcE districts in Dosso and Tahoua. However, RAcE intervention areas likely had lower coverage than that presented by service statistic data used in the model. Statistics from the MSP used to determine the RAcE project area population estimated that approximately 40 to 58 percent of the population in the districts in which RAcE was implemented was covered by health facility services; this ranged from 40 percent in Dogondoutchi to 58 percent in Keita.
WHO and UNICEF national immunization coverage estimates likely overestimate coverage in the hard-to-reach RAcE project areas.

8. Plausible Contribution of RAcE
Observed increases in iCCM-related indicators are most likely due to RAcE project interventions.
The results from this evaluation suggest that it is likely that the RAcE project contributed substantially to the estimated decrease in under- five child mortality between 2013 and 2016.
The estimated change in under-five mortality in the project areas during RAcE implementation was 17 deaths per 1,000 live births, a 12.6 percent decrease over the course of the project. Approximately 965 child lives were saved due to iCCM treatment provided by RComs through the RAcE project. The results from the evaluation suggest that it is likely that RAcE contributed significantly to the estimated decrease in under-five mortality.

9. Conclusion The LiST model estimates that from 2013 to 2016:
13 percent decrease in child mortality in 4 RAcE Niger districts
Net 1,931 lives were saved among children under five
359 lives lost due to decreases or stagnation in intervention coverage
2,290 lives saved due to increases in intervention coverage
  1,128 under-five lives (49%) saved by pneumonia, diarrhea, and malaria treatment.
ICF concludes that:
An estimated 965 under-five lives were saved (42%) due to RCom- provided treatment
It is highly plausible that the RAcE Niger project contributed substantially to the observed mortality reduction, namely, to nearly 50 percent of the estimated total child lives saved.

10. Acknowledgements
ICF would like to thank World Vision and the Niger MSP for sharing their data, time, thoughts, and experiences in implementing the RAcE project in Niger.
We would also like to thank the RCom in Niger, who work hard to provide services to caregivers and children in communities, and the caregivers who give so much to ensure and improve the health of their children.
This work was made possible by the WHO through funding by Global Affairs Canada.

11. Thank You!