1. GSF Surveys and Evaluation 11th – 15th September, 2017 Dar es Salaam, Tanzania

2. Session Objectives
To become familiarized with the new outcome survey proposed methodology, tools and GSF technical support system.
To learn from the pilot outcome survey in Malawi as well as other recent GSF surveys (Nepal Sustainability Survey).
To discuss experience of mid-term evaluations
To learn more on the process of the recent Togo Joint Evaluation

3. Outcome Survey Objectives
To provide statistically reliable data on key sanitation and hygiene household & facility level indicators to measure outcomes of GSF supported programmes in the GSF targeted areas. This also allows for the independent verification of reported program results;
To serve as a sustainability check through identifying whether households in ODF declared communities have continued to use and properly maintain improved toilets and hand washing facilities and continue to comply to national ODF criteria;

4. Outcome Survey Objectives
To understand if the results achieved have addressed equity and non-discrimination and have met the needs of marginal and vulnerable households/populations.
To collect data on emerging areas or indicators that are too sensitive, challenging or not appropriate to collect through routine programme monitoring systems. For example indicators relating to satisfaction with sanitation and hygiene facilities and participation in programme interventions.

5. So not only measuring outcomes…. do ‘outcome surveys’ need a new name
So not only measuring outcomes…. do ‘outcome surveys’ need a new name? Suggestions welcome 

6. Methodology: Survey Tools
Household head questionnaire
Female/caregiver questionnaire
Persons with disability/mobility issues questionnaire
Schools and health facility questionnaire
Structured observations

7. Household Head Questionnaire
Household demographics and household assets
Water access, treatment and storage
Sanitation facilities (with inspections)
Handwashing facilities (with inspections)
Participation and exposure to programme activities
Latrine construction, cost, repairs
Sanitation habits and social norms

8. Female/Caregiver Questionnaire
Latrine use of all household members
Participation and exposure to program activities;
Sanitation habits and social norms
Satisfaction with sanitation and hygiene services in the home (privacy, safety etc.)
Menstrual hygiene management
Disposal of child feces
Questionnaire to be administered by female interviewers only

9. Disability Questionnaire
Access to sanitation and hygiene services in the home
Satisfaction with sanitation and hygiene services in the home
To be asked to persons (over 18 years) with a mobility limitation

10. Schools and health facility questionnaire
Water management
Sanitation management
Handwashing
Programme exposure
MHM
Questionnaires asked to school/health facility management

11. Structured Observations
Conducted by a female observers who spend 3 hours in a household.
Observers use a structured observation guide which is based on a checklist for recording and describing the following behaviors for all household members:
Latrine use
Handwashing with soap and water around events of potential pathogen transmission
Safe disposal of feces of children under five years of age

12. Preliminary structured observation results, Malawi, 2017
Defecation / toilet use (n=360)
Food prep. (n=545)
Feeding child <5 years (n=45)
After respiratory fluid contact (n=493)
Before breastfeeding (n=166)
Before eating (n=695)
Handwashing practice
(n=360)
(n=545)
(n=45)
(n=493)
(n=166)
(n=695)
No handwashing
47%
55%
62%
94%
92%
73%
Handwashing with water only
32%
42%
33% 5% 8%
27%
Handwashing with water and any type of soap
18% 3% 4%
<1% 0%
Handwashing with water and ash
13%
Preliminary structured observation results, Malawi, 2017

13. Social Norms Example questions:
Do you agree with the statement: Everyone should use a toilet
Where do your neighbors think you should defecate
How many of your neighbors would agree with the statement: It is fine for everyone to defecate in the open.
If everyone in your community practiced open defecation, would you definitely practice open defecation
To assess the existence of a norm, it is important to ask people not just what their personal normative beliefs are, but what they expect other's normative beliefs to be.
Empirical norms: What people expect others in the same situation to do.
Normative norms: What they think other people believe ought to be done.

14. Latrine Social Norms Score Counts/Frequency Household Caregiver
(by domains of norms)
Counts/Frequency Household Caregiver
Household (N=1445)
Empirical Expectation
185 (11.8%) 1
155 (9.9%) 2
1228 (78.3%)
Normative Expectation
0 (0.0%)
1 (0.1%)
10 (0.6%) 3
94 (6.0%) 4
199 (12.7%) 5
1262 (80.6%)
Personal Normative Belief
295 (18.6%)
198 (12.5%)
1097 (69.0%)
Preliminary structured
observation results, Malawi, 2017

15. Methodology: Sampling
Key principles:
Sample frame to include all communities/villages in which GSF-funded programs have been implemented (irrespective of ODF status);
Random sampling of communities/villages using a probability proportional to size, multi-stage, cluster approach;
Over sampling of ODF villages may be required so indicators on ODF sustainability can be reliably measured.

16. Methodology: Sampling
Key principles of the design:
Among communities/villages selected, 16 households at random will be selected from an updated household listing.
For the structured household observations, a subset of 4 households will be randomly selected from the (16) interviewed households in each community.
For the schools and health posts, if communities have more than one school or health post then sampling will be conducted so just one school or health post is selected per community

17. Methodology: Survey Management
Recruit an independent research firm/agency
Manage the research firm to do the following:
Develop the survey plan
Customize standard tools (and translate)
Conduct a pre-test
Conduct training
Conduct field work (data entry in tablets)
Analyze data and write the final report

18. Analytical Approach
Latrine and handwashing access data analysed against JMP ladders
Outcomes disaggregated against background variables (district, education of household head, wealth quintile, ethnicity)
Outcomes disaggregated by year of ODF declaration

19. Analytical Approach Creation of composite indicators:
The extent of exposure to the GSF-supported CLTS intervention
The strength of ‘latrine use habit’
The strength of social norms relating to latrine use
Satisfaction with sanitation facilities
Improved MHM practices
ODF status
Slippage

20. GSF Technical Support Process
GSF to develop following:
Survey Terms of Reference
Standard survey tools for customization
Analysis plan and report template
Technical assistance system to review following survey documents:
Survey plan/sample design
Survey customised tools
Analysis and final report
Technical assistance in country during survey design and training
Final report

21. Lessons from the Malawi Pilot
For all actors (EA, GSF and University) to be involved in the review of research firm bids and survey proposal.
Research firm to be on board for at least 4- 6 months.
Tools too overloaded and analysis is too heavy .

22. The Malawi experience

23. Most recent programme baseline or outcome survey
Date of last survey
Cambodia
2015
Ethiopia
2016
Kenya
Malawi
2017
Nepal
Nigeria
Tanzania
2014
Uganda
2013

24. Evaluation
GSF commissioned 10 programme mid-term evaluations between December 2013 – January 2016
Evaluations all conducted by the same external organisation

25. Evaluation
Lessons
Evaluations were not all conducted at the mid-term and findings were often too late to feed into major decision processes regarding extensions and expansions;
Findings did not necessarily uncover anything “new” but provided independent evidence to advocate for strengthening particular weaknesses in programmes at all levels.
Evaluation management responses demanded attention to the above areas but monitoring of management responses has not been well systemised.
General ownership of the whole evaluation process by different actors has been weak.

26. Togo Joint Evaluation

27. Summary
Previous GSF M&E Guidelines stipulated outcome surveys to be conducted at the mid-term and end-term period. Moving forward programmes will be required to do outcome surveys on a more regular basis. Every two years is proposed.
Outcome surveys are not always linked to evaluations although if an evaluation is to take place, an outcome survey should be conducted in the 6 months prior
GSF to provide a stronger technical support role to outcome surveys.
Evaluation schedule and criteria still to be determined. Moving towards a joint evaluation approach where possible.