1. LLIN Universal Coverage (UC) in Uganda: Process Evaluation (PE) Methodology Presenter at KET: Caroline Asiimwe LLIN Process Evaluation Team (Adoke Yeka, Caroline Asiimwe, Thomas Park, Wayne Stinson, Jenifer Peters) 1 April 08, 2015

2. Outline Background of the LLIN Universal Campaign (UC) Purpose of Process Evaluation (PE) Guiding PE Questions Ideal Methodologies (International Standards) Realistic methodologies (National Adjustments) Pretest of PE Methodologies Implementation plan for the PE Qualitative and Quantitative Methods for the PE Opportunities & Challenges of the PE design Discussion 2

3. 3 Background (Justification for the UC) the WHO World Malaria Report on Uganda’s trends and effect of malaria morbidity & mortality Endemic in 95% of Uganda Primary contributor to poverty and low productivity Malaria control is a cornerstone of national development National Development Plan 2010/11 – 2014/15, Health Sector Strategic Plan, 2010/11 – 2014/15 LLINs play an important role in reducing malaria burden MoH target - to protect at least 85% of the population at risk of malaria using measures that include LLINs.

4. Background (timelines for UC) 4

5. Purpose of the PE The evaluation had two phases Phase I - Process evaluation; – To understand the extent to which the universal LLIN distribution campaign was implemented according to plan – To evaluate the management of the program – To review the design and instruments for the 2014 Malaria Indicator Survey (MIS) – To document lessons for future comprehensive campaigns in Uganda and worldwide Phase II – Impact Evaluation; – Determine the impact of the UC on malaria indicators – Provide accountability for the use of resources 5

6. Guiding PE Questions Relevance: Was the design and timing of the UC based on strong evidence? Effectiveness: To what extent did the UC attain its objective of national coverage Efficiency: Was the most efficient process adopted in implementing the UC? Impact: To what extent was LLIN ownership increased by the campaign? Sustainability: Is funding sufficient to carry out the UC every three years as planned 6

7. Ideal Methodology for PE (by RBM MERG) Alignment to standards drawn by Roll Back Malaria M&E Reference Group (RBM MERG) and Alliance for Malaria Prevention Review of methods for post-campaign survey methodology (e.g Malaria Indicator Survey) – multi-stage randomised cluster survey recommended Review alignment to RBM MERG process indicators specific LLIN campaign strategy adopted Review justification for the selected indicator (i.e. how the information will be used to improve campaign quality in future) Review data collection methods and timing (e.g supervisory visits, household visits and population-based surveys) Interview persons responsible for overseeing data collection and reporting Review method for sharing the results (national post-campaign summary workshops, coordinators meetings) 7

8. Ideal Methodology – PE Indicators by RBM MERG (1) Number and proportion of districts with functioning campaign coordination mechanisms in place Number and proportion of expected supervisory reports received at the district level Number and proportion of household registration form summaries completed correctly Number and proportion of planned radio spots broadcast Number and proportion of community mobilizers trained 8

9. Ideal Methodology - PE Indicators by RBM MERG (2) Number and proportion of households visited by a community mobilizer before the campaign Number and proportion of districts that received operational funds in time for the campaign Number and proportion of distribution sites properly equipped with trained staff Number and proportion of distribution sites with LLIN stocks correctly stored and accounted for Number and proportion of distribution sites that report a gap in stocks of LLINs 9

10. Realistic Methodology - Donors’ Approach to the PE Evaluation system plan: Meta evaluation planned by the donors and MoH counterparts (Group critical evaluation sessions) –Scope of Work Design: Engagement of MoH and non MoH implementing partners to scope the PE –Sampling plan: (alignment to multi-stage cluster survey, with probability to proportionate size sampling) : 1 st stage – national population, 2 nd stage – district waves; 3 rd stage – household individuals –Measures for the PE: Efficiency, Effectiveness, Relevance, Sustainability 10

11. Realistic Methodology - MoH approach to the PE MoH expectations of the PE; a) detailed description of reality vs existing guidelines of the ideal b) measurement of two dimensions - the efficacy and efficiency of the operation c) recommendations to improve operational processes Intervention areas –Financing and financial management of the LLIN campaign –Procurement, transportation and storage of LLINs –Household registration and LLIN allocation –Distribution of LLINs to beneficiaries –Training and supervision –Advocacy, social mobilization and behaviour change communication 11

12. Pre-testing the PE Framework/Scope of work A pilot was conducted to determine a feasible PE plan; –Non participant direct observation of LLIN distribution in a purposive sampled evaluation area –Content of UC –Beneficiary identification and registration –LLIN quantification –Rolling distribution versus mass coverage –Qualitative methods used to elicit (un)anticipated information, diverse perspectives and Interests of different stakeholders Determine appropriate methods for quantitative and qualitative data collection (in context of cost, turnaround time, human resource) 12

13. PE Implementation Plan by Evaluators Mix of donor and MoH (National Coordinating Committee) preferred approaches –Qualitative Questionnaire Tool developed with MoH and Donors, pre-tested in Mukono District –Review of MoH and Donor minutes and agreements –Used this information to map out target evaluation informants and areas (purposive sampling) –Conveniently selected initial key informants, sub national evaluation areas –Snowball (non probability) sampling applied to in-depth interviewees (to elicit unanticipated information and represent diverse perspectives and Interests of different stakeholders) 13

14. PE for Planning & Coordination by MoH (1) Appropriateness of UC Design –Distribution method Rolling Distribution/Waves MoH Preferred - 8 waves, each 15-16 districts, with a 24-day timeline per district) One time Mass –Categorization of UC; stand-alone (MoH Preferred) integrated with other interventions – Implementation strategy; fixed site distribution ( MoH Preferred ) house to house distribution 14

15. PE for Planning and Coordination by MoH (2) –Beneficiary Identification and registration household records (MoH Preferred), vouchers/coupons; wristband /bracelets –LLINs Quantification 1 LLIN/2 people, with rounded up/rounded down odd numbers) MoH preferred; No of sleeping spaces; fixed number of LLINs/Household 15

16. Qualitative Methods used for the PE –Interview tool used to guide interviews (Introduction, key, probing and closing questions) – Telephone structured Interviews - planners at international level (The Global Fund) –Face to Face structured interviews – planners at national level –Face to Face In-depth interviews – planners at sub-county and community level – Focus Group Discussions – maximum diversity sampling applied to community gatherings at LC meetings & health centres, with first hand experience with UC distribution points and LLIN use –Data Coding based on pre-determined themes, analysis and reporting 16

17. Quantitative Methods used for PE (1) Effectiveness: National Coverage Evaluation considered; –Number and proportion of supervisors and community mobilisers/distributors trained for UC –Number of LLINs procured and proportion supplied at designated distribution points –Number and proportion of households registered and received at least 1 LLIN per 2 people –Population targets (UBOS vs household registration) Efficiency: Was the most efficient process adopted in implementing the UC? Were most activities implemented in a cost-efficient manner – PE did not carry out a detailed cost- benefit analysis, pended to Phase 2 Impact Evaluation 17

18. Opportunities realized with the PE methodology Provided a platform for actors to discuss and debate tools and actions (Meta-evaluation) Solicited expectations of the end user for the planners to review their plans for LLINs UC implementation- 2017 Engaged other government departments to re-think public health issues and statistics (UBOS, State House, NGOs, Donors, etc) Enabled efforts to harmonize ideal and real applications of evaluation methodologies. 18

19. Challenges with implementing the PE methodology Unreliability of quantitative data (e.g. population discrepancies between UBOS and Household registered numbers) Lack of real time LLIN distribution data despite computerization of the UC system, to enable analyses Disaggregated databases (MoH, Implementing partners with different databases) Financial data hard to access or untimely Parallel roadmaps between M&E and policy planners (Evaluation methods not necessarily embedded in planning and politics) 19

20. End Contact: karolasiimwe@gmail.com 20