1. RAcE Malawi Final Evaluation Results
RAcE 2015 Programme
Multi-Country Results Dissemination Meeting
25 October 2017

2. RAcE Malawi Final Evaluation Results - Overview
To demonstrate the plausible contribution of the RAcE project to changes in treatment coverage indicators and estimated mortality change, ICF assessed project and district data, estimated the change in child mortality in RAcE project areas using LiST, and documented contextual factors that may have influenced child health in project areas.
Here we present findings that answer two evaluation questions:
Was there a reduction in childhood mortality, and were the lives of children ages 2–59 months saved, in the RAcE project area?
What was the RAcE project’s contribution to the estimated changes in mortality?
In this presentation I will share some high level evaluation findings, focusing on specifically on findings of the LiST analysis and plausible contribution of RAcE. Due to time limitations I will not be going through all evaluation findings.
This is not an evaluation of the entirety of RAcE project inputs to strengthening health systems, including community-level health service delivery. This evaluation does not assess the second objective of the RAcE programme: stimulate policy review and regulatory update in each country on disease case management.

3. Summary of Key Evaluation Findings (1)
Many HSAs do not reside in their catchment areas, some of whom are consequently not trained in iCCM.
Health surveillance assistants’ (HSAs’) availability is limited.
Care-seeking overall did not change over the course of the project (~70%)
Only about half of all sick child cases taken to an HSA at endline; a majority of caregivers sought care from other providers instead of HSAs.
HSAs managed more than 1 million cases of malaria, more than 500,000 cases of cough with fast breathing, and nearly 300,000 cases of diarrhea in children under 5 years from 2013 through 2016.
The percentage of sick child cases for which caregivers sought care from an appropriate provider (approximately 70 percent) did not change over the course of the project, but those taken to an HSA as a first source of care increased, from 26 percent at baseline to 34 percent at endline. Among those cases that sought any care, approximately half sought care from an HSA as the first source. This increase was mainly due to the increase in seeking care from an HSA for cough with difficult or fast breathing.

4. Summary of Key Evaluation Findings (2)
RAcE supported the introduction of mRDTs and replacement of cotrimoxazole with amoxicillin at village clinics.
Increases in cases of fever tested for malaria likely due to the RAcE project’s efforts to introduce mRDTs at the community level.
Increases in cases of cough with difficult or fast breathing assessed for high respiratory rate likely due reinforcement via RAcE project’s refresher trainings following treatment protocol updates.
Contributions of other projects* to the improvements in care-seeking and sick child case management are likely small.
RAcE project facilitated the incorporation of facility-level iCCM data in the district health information system-2, the national health management information system.
*Concern Universal, Population Services International, Save the Children’s Quic Study (mRDT pilot)
There were increases in malaria and respiratory rate assessments performed by HSAs during the project implementation period, as well as in confirmed malaria treatments. The project rolled out mRDTs at the community level and trained HSAs to provide ACTs to only children who have confirmed malaria. The project also began to equip HSAs with amoxicillin—rather than cotrimoxazole—as first-line treatment for suspected pneumonia and trained HSAs on the administration of the new drug, which included proper respiratory rate assessment. Diarrhea treatment with zinc, however, was low at baseline and did not improve over the project implementation period. This is an area of concern because the treatment protocol for diarrhea to is to administer both ORS and zinc. This trend may have been noticed earlier in the project if iCCM reporting forms included data on ORS and zinc provision, but the forms collect information on the number of diarrhea cases managed only.
The contributions of other projects to the improvements in care-seeking and sick child case management are likely small. Concern Universal trained 10 HSAs in iCCM in Ntcheu district, but the RAcE project subsequently equipped and supported them. PSI implemented activities that supported iCCM more widely in Ntcheu district, so the dispersible amoxicillin orientation, HSA review meetings, district team supervisions, and mentorship training may have had some impact on the assessment and treatment indicators that increased during the RAcE project implementation period. However, any other iCCM activities implemented in the four original RAcE project districts were done at a much smaller scale, and none of the activities implemented in Lilongwe district would have affected the results discussed above because Lilongwe district was not included in the household survey. Lastly, although the RAcE project was the first project to roll out mRDTs at the community level in Malawi, other projects, such as the Save the Children’s Quic Study, paved the way by piloting mRDTs and informing Malawi’s current mRDT policy.

5. Estimated Change in Child Mortality in RAcE Malawi Project Areas
The LiST model estimated results based on the total population in HTRAs of the four original RAcE Malawi project districts at the start of the project in (1,906,136).
Estimated change in U5MR in the project area:
6 deaths per 1,000 live births
5 percent decrease in U5MR from 2013 to 2016.
Table 1. Estimated mortality rates modeled in LiST for each project year.
RAcE Malawi
Year
Under-five mortality rate (deaths per 1,000 live births)
2013
124.26
2014
122.24
2015
120.27
2016
118.54
Baseline child mortality, total fertility rate, and contraceptive prevalence inputs for 2013 were those measured by the 2011 DHS. The model assumes that mortality, fertility, and contraceptive prevalence did not change significantly in the RAcE project areas from 2011 to However, if the actual 2013 child mortality rates were lower than those used in the model, due to progress in improving child health over the two-year period of 2011–2013, the change in mortality estimated by the model may be an overestimate.

6. Estimated Lives Saved in RAcE Malawi Project Areas
An estimated total of 572 under-five lives saved by pneumonia, diarrhea, and malaria treatment from 2013 to 2016.
An estimated 216 lives were saved due to treatment provided by HSAs.
Table 2. Estimated number of child lives saved per year by treatment interventions in Malawi project areas
RAcE Malawi
2013*
2014
2015
2016
Total
Percentage intervention treatment by HSAs
Estimated lives saved by HSA-provided treatment
Total lives saved among children 1–59 months (all interventions)
189
292
395
477
1,353
Intervention
Estimated lives saved
ORS for diarrhea 3 6 10 19
54%
Zinc for diarrhea 7 15 23 45
52%
Oral antibiotics for pneumonia 81
167
260
508
36%
183
ACT for malaria
49%
Total lives saved by year 91
188
293
572 -
216
Because the LiST model uses the coverage estimates for the baseline year (2013 for iCCM interventions) to estimate lives saved for the model’s projection period, the estimated lives saved for the iCCM baseline year is by default zero. There were iCCM activities in some project areas prior to the 2013 baseline year; this model does not take into account the effect of these activities.
At the beginning of the RAcE project, HSAs did not have mRDTs at their village clinics; they treated malaria presumptively—providing ACTs to all children ages 5–59 months who had fever. mRDTs were rolled out in the RAcE project districts in 2014 and 2015, and treatment protocols were changed so that HSAs were treating only cases of fever that had a positive mRDT. Therefore, the percentage of children with fever who were treated with ACT decreased over the course of the project; however, the percentage of children with confirmed malaria who were treated with ACT did not change significantly. Treatment by facility-based providers should have been confirmed malaria treatment, following positive blood test with microscopy or mRDT, at both baseline and endline. Due to this change in HSA treatment policy, the percentage of cases of fever treated with ACT within 48 hours measured at baseline was flatlined (held constant) in the model to avoid an artificial decrease in treatment coverage resulting in lives lost; the model consequently shows no estimated impact from malaria treatment.
The probable overestimate of lives saved from over-projecting the baseline population and probable underestimate of lives saved by not accounting for the population covered when the project expanded to additional districts likely—at least in part—nullified each other. However, the number of lives saved by HSA-provided treatment is likely an overestimate. The proportion of treatment provided by HSAs used to make the lives saved adjustment is representative of only the 33 household survey clusters confirmed to have HSAs present and active at endline. By adjusting the total number of estimated lives saved in the original project area by the proportion treated by HSAs in a geographic sub-set of the original project area, we are incorrectly making the assumption that HSAs were providing this proportion of treatment across the project area. HSAs were only providing this proportion of treatment in about 55 percent of the original project area due to the HSA deployment and residency issues discussed in Section 1.3, RAcE Project Background in Malawi.

7. LiST Model Limitations
The accuracy of the model results is limited by the data input to the model.
LiST does not account for the mode of delivery or source of care (with the exception of facility birth).
LiST model does not account for changes in diagnostics, the quality of care, timeliness of pneumonia and diarrhea treatment, nor referrals made or completed. 
Holding fever treatment constant does not account for lives saved by the improvement in malaria case management following the introduction of RDTs and may therefore underestimate the total lives saved by the project.
The number of estimated lives saves is limited by the following population data constraints:
The project area population was likely smaller in 2010, the baseline model year, than that used in the model. By using the 2013 project area population as the model baseline (2010) population, the population projections made by the model for 2013 based on fertility rates and other demographic data were likely larger than actual 2013 population. Consequently, the model may have yielded slightly higher estimates of number of lives saved.
The baseline population used in the model does not account for the expansion of RAcE Malawi from four districts to eight districts in The estimated number of lives saved is based on only the original project area population in 2013 and may therefore underestimate the total lives saved by the project.
Holding fever treatment constant from baseline to endline does not account for lives saved by the improvement in malaria case management following the introduction of RDTs and may therefore underestimate the total lives saved by the project.
The baseline population used in the LiST model affects population projection and consequent estimation of lives saved. It does not, however, change the estimated impact on mortality. Estimated change in under-five mortality is thus the best indicator of estimated impact.

8. Plausible Contribution of RAcE
Observed increases in assessment of fever and cough with difficult or fast breathing are most likely due to RAcE project interventions including contributions to mRDT introduction.
Other projects in the RAcE project districts may have contributed to the observed outcomes because they supported HSAs’ iCCM work through trainings, supervisions, and review meetings, and to broader benefits to the iCCM programme.
The results from this evaluation suggest that RAcE contributions strengthened the health system and supported efforts promoting HSAs as a source of care available to communities.
The LiST analysis showed a 4.6 percent decrease in mortality—from 124 to 118 deaths per 1,000 live births between 2013 and In addition, the analysis estimated that 216 lives were saved due to pneumonia and diarrhea treatment provided by HSAs in the four original project districts.
The estimated decrease in under-five mortality was small in the four original RAcE project districts, but RAcE contributions strengthened the health system and highlighted the efforts that Save the Children made toward promoting HSAs as a source of care available to communities; introducing mRDTs to ensure that antimalarials are more appropriately used to treat only confirmed cases of malaria; assessing children who had cough with difficult or fast breathing for high respiratory rate before antibiotics are given for cases of suspected pneumonia; and trying to ensure that the iCCM data collected and reported are available, valid, reliable, and consistent.
The projects implemented by PSI and Concern Universal in Ntcheu district, as well as any other small district-level projects in the four original RAcE project districts, may have contributed to the observed outcomes because they supported HSAs’ iCCM work through trainings, supervisions, and review meetings. The RAcE project, however, equipped many of the HSAs trained or otherwise supported through other projects with essential iCCM supplies. In addition, those other projects were more localized than RAcE. The iCCM-related activities implemented in Lilongwe district did not affect the endline survey results but likely contributed to Save the Children’s routine monitoring indicators. Lastly, although the Global Fund’s Malaria Project has focused on non-iCCM activities in all of the RAcE project districts, some of those activities may have benefited the iCCM programme, such as through improved community awareness of iCCM services and community mobilization through care groups.

9. Conclusion The LiST model estimates that from 2013 to 2016:
5 percent decrease in child mortality in RAcE project districts.
Net 3,161 lives were saved among children under five
1,020 lives lost due to decreases or stagnation in intervention coverage
4,181 lives saved due to increases in intervention coverage
572 under-five lives (14%) saved by pneumonia, diarrhea, and malaria treatment.
ICF concludes that:
An estimated 216 under-five lives were saved (5%) due to HSA- provided treatment
It is highly plausible that the RAcE Malawi project strengthened iCCM services and support.

10. Acknowledgements
ICF would like to thank Save the Children and the Malawi MOH for sharing their data, time, thoughts, and experiences in implementing the RAcE project in Malawi.
We would also like to thank HSAs in Malawi, who work hard to provide services to caregivers and children in communities, and the caregivers who give so much to ensure and improve the health of their children.
This work was made possible by the WHO through funding by Global Affairs Canada.

11. Thank You!