GSF Surveys and Evaluation 11th – 15th September, 2017 Dar es Salaam, Tanzania

Session Objectives To become familiarized with the new outcome survey proposed methodology, tools and GSF technical support system. To learn from the pilot outcome survey in Malawi as well as other recent GSF surveys (Nepal Sustainability Survey). To discuss experience of mid-term evaluations To learn more on the process of the recent Togo Joint Evaluation

Outcome Survey Objectives To provide statistically reliable data on key sanitation and hygiene household & facility level indicators to measure outcomes of GSF supported programmes in the GSF targeted areas. This also allows for the independent verification of reported program results; To serve as a sustainability check through identifying whether households in ODF declared communities have continued to use and properly maintain improved toilets and hand washing facilities and continue to comply to national ODF criteria;

Outcome Survey Objectives To understand if the results achieved have addressed equity and non-discrimination and have met the needs of marginal and vulnerable households/populations. To collect data on emerging areas or indicators that are too sensitive, challenging or not appropriate to collect through routine programme monitoring systems. For example indicators relating to satisfaction with sanitation and hygiene facilities and participation in programme interventions.

So not only measuring outcomes…. do ‘outcome surveys’ need a new name So not only measuring outcomes…. do ‘outcome surveys’ need a new name? Suggestions welcome 

Methodology: Survey Tools Household head questionnaire Female/caregiver questionnaire Persons with disability/mobility issues questionnaire Schools and health facility questionnaire Structured observations

Household Head Questionnaire Household demographics and household assets Water access, treatment and storage Sanitation facilities (with inspections) Handwashing facilities (with inspections) Participation and exposure to programme activities Latrine construction, cost, repairs Sanitation habits and social norms

Female/Caregiver Questionnaire Latrine use of all household members Participation and exposure to program activities; Sanitation habits and social norms Satisfaction with sanitation and hygiene services in the home (privacy, safety etc.) Menstrual hygiene management Disposal of child feces Questionnaire to be administered by female interviewers only

Disability Questionnaire Access to sanitation and hygiene services in the home Satisfaction with sanitation and hygiene services in the home To be asked to persons (over 18 years) with a mobility limitation

Schools and health facility questionnaire Water management Sanitation management Handwashing Programme exposure MHM Questionnaires asked to school/health facility management

Structured Observations Conducted by a female observers who spend 3 hours in a household. Observers use a structured observation guide which is based on a checklist for recording and describing the following behaviors for all household members: Latrine use Handwashing with soap and water around events of potential pathogen transmission Safe disposal of feces of children under five years of age

Preliminary structured observation results, Malawi, 2017   Defecation / toilet use (n=360) Food prep. (n=545) Feeding child <5 years (n=45) After respiratory fluid contact (n=493) Before breastfeeding (n=166) Before eating (n=695) Handwashing practice (n=360) (n=545) (n=45) (n=493) (n=166) (n=695) No handwashing 47% 55% 62% 94% 92% 73% Handwashing with water only 32% 42% 33% 5% 8% 27% Handwashing with water and any type of soap 18% 3% 4% <1% 0% Handwashing with water and ash 13% Preliminary structured observation results, Malawi, 2017

Social Norms Example questions: Do you agree with the statement: Everyone should use a toilet Where do your neighbors think you should defecate How many of your neighbors would agree with the statement: It is fine for everyone to defecate in the open. If everyone in your community practiced open defecation, would you definitely practice open defecation To assess the existence of a norm, it is important to ask people not just what their personal normative beliefs are, but what they expect other's normative beliefs to be. Empirical norms: What people expect others in the same situation to do. Normative norms: What they think other people believe ought to be done.

Latrine Social Norms Score Counts/Frequency Household Caregiver (by domains of norms) Counts/Frequency Household Caregiver   Household (N=1445) Empirical Expectation 185 (11.8%) 1 155 (9.9%) 2 1228 (78.3%) Normative Expectation 0 (0.0%) 1 (0.1%) 10 (0.6%) 3 94 (6.0%) 4 199 (12.7%) 5 1262 (80.6%) Personal Normative Belief 295 (18.6%) 198 (12.5%) 1097 (69.0%) Preliminary structured observation results, Malawi, 2017

Methodology: Sampling Key principles: Sample frame to include all communities/villages in which GSF-funded programs have been implemented (irrespective of ODF status); Random sampling of communities/villages using a probability proportional to size, multi-stage, cluster approach; Over sampling of ODF villages may be required so indicators on ODF sustainability can be reliably measured.

Methodology: Sampling Key principles of the design: Among communities/villages selected, 16 households at random will be selected from an updated household listing. For the structured household observations, a subset of 4 households will be randomly selected from the (16) interviewed households in each community. For the schools and health posts, if communities have more than one school or health post then sampling will be conducted so just one school or health post is selected per community

Methodology: Survey Management Recruit an independent research firm/agency Manage the research firm to do the following: Develop the survey plan Customize standard tools (and translate) Conduct a pre-test Conduct training Conduct field work (data entry in tablets) Analyze data and write the final report

Analytical Approach Latrine and handwashing access data analysed against JMP ladders Outcomes disaggregated against background variables (district, education of household head, wealth quintile, ethnicity) Outcomes disaggregated by year of ODF declaration

Analytical Approach Creation of composite indicators: The extent of exposure to the GSF-supported CLTS intervention The strength of ‘latrine use habit’ The strength of social norms relating to latrine use Satisfaction with sanitation facilities Improved MHM practices ODF status Slippage

GSF Technical Support Process GSF to develop following: Survey Terms of Reference Standard survey tools for customization Analysis plan and report template Technical assistance system to review following survey documents: Survey plan/sample design Survey customised tools Analysis and final report Technical assistance in country during survey design and training Final report

Lessons from the Malawi Pilot For all actors (EA, GSF and University) to be involved in the review of research firm bids and survey proposal. Research firm to be on board for at least 4- 6 months. Tools too overloaded and analysis is too heavy .

The Malawi experience

Most recent programme baseline or outcome survey Date of last survey Cambodia 2015 Ethiopia 2016 Kenya Malawi 2017 Nepal Nigeria Tanzania 2014 Uganda 2013

Evaluation GSF commissioned 10 programme mid-term evaluations between December 2013 – January 2016 Evaluations all conducted by the same external organisation

Evaluation Lessons Evaluations were not all conducted at the mid-term and findings were often too late to feed into major decision processes regarding extensions and expansions; Findings did not necessarily uncover anything “new” but provided independent evidence to advocate for strengthening particular weaknesses in programmes at all levels. Evaluation management responses demanded attention to the above areas but monitoring of management responses has not been well systemised. General ownership of the whole evaluation process by different actors has been weak.

Togo Joint Evaluation

Summary Previous GSF M&E Guidelines stipulated outcome surveys to be conducted at the mid-term and end-term period. Moving forward programmes will be required to do outcome surveys on a more regular basis. Every two years is proposed. Outcome surveys are not always linked to evaluations although if an evaluation is to take place, an outcome survey should be conducted in the 6 months prior GSF to provide a stronger technical support role to outcome surveys. Evaluation schedule and criteria still to be determined. Moving towards a joint evaluation approach where possible.