1. Can Community Health Clubs change hygiene behaviour in developing countries?
Reflections on the findings of a randomized control trial in Rusizi District, Rwanda.
Cairncross S, Waterkeyn J, Uwingabire F, Pantoglou J, Waterkeyn A, Ntakarutimana A, Mukamunana A, Joseph K, Bigirimana Zachery, Dushime T.
Presented at CBEHPP Workshop. 25th May 2017
Kigali, Rwanda

2. Outline of Presentation
Timing of Activities and Data Collection
Method of Data collection
Survey Tool: Household Inventory
Results: % achieved of 10 indicators
Conclusion
Questions raised

3. MoH/AA monitoring data in Rusizi District (2013-2017)
METHOD:
Random sampling of 50 classic CHCs
Randomly sampled respondents within a classic CHC
Criteria for respondents:
CHC households with certain attendance level
5 rounds of Data collection over 3 years

4. MoH / AA DATA COLLECTION PERIOD
Timing of activities & Data Collection in the RCT in Rusizi District :
CHC Train
‘NO GO’ period
MoH / AA DATA COLLECTION PERIOD 1 2 3 4 5
IPA RCT baseline
RCT post

5. Monitoring Tool: Household Inventory
A survey tool was developed called the Household Inventory (HHI).
HHI adapted after the paper baseline for mobile application
10 main indicators each with a 5 sub Indicators
Use of a mobile data collection for the mid line and end line (Mobenzi)
For post intervention we developed an AA application using ODK (Open Data Kit)
All data downloaded into excel, transferred to SPSS and analysed
The CBEHPP website was also developed as a data base for all CHC
Any stakeholders in CBEHPP can use this portal – trained by Africa AHEAD

6. Baseline Survey of Classic CHC: Nov-Dec 2013
3,955 respondents
Baseline Survey of Classic CHC: Nov-Dec 2013
All 50 classic villages
75% of all 4,252 households within each village
5,745 households visited
Spot observation
Criteria for survey: should not have attended any CHC sessions
Enumerators were the village health worker for each village
One day training of enumerators
Use of printed paper based survey booklets
High level of human error
Cleaning data reduced cases from 5,745 to 3,955 cases

7. Midline Survey of Classic CHC: May 2014
738
respondents
Midline Survey of Classic CHC: May 2014
Random sample of 30 CHCs throughout district
30% of all CHC members randomly sampled in each CHC
Criteria for survey:
Respondent must be the CHC Member
Respondent must have attended between 1- 4 sessions
Enumerators were the EHOs, assisted by CHC chairperson and ASOC
One day training
Use of cell phone to collect data
Mobenzi server to collect and analyse data
Cleaning data reduced cases from 797 to 738 cases (30%)

8. Endline Survey of Classic CHC: Nov-Dec 2015
424
respondents
Endline Survey of Classic CHC: Nov-Dec 2015
Random sample of 24 CHCs
Random sample of 20% of all CHC members
Criteria for Survey:
Respondent must be a CHC Member
Respondent must have attended over 17 sessions
Enumerators: EHOs, assisted by the CHC chairperson and ASOC
One day training
Use of cell phone to collect data
Mobenzi server to collect and analyse data
Cleaning data reduced cases from 475 to 424 cases (16%)

9. 1st Post Intervention Survey: May 2016
408
respondents
1st Post Intervention Survey: May 2016
All 50 CHCs sampled
10 CHC members sampled in each CHC
Criteria for survey:
Respondent must be a CHC Member
Respondent must have attended over 17 sessions
Enumerators: EHOs, assisted by the CHC chairperson and ASOC
One day training of enumerators
Use of cell phone to collect data
ODK server to collect and analyse data
Cleaning data reduced case from 502 to 408

10. 2nd Post intervention Survey of Classic CHC: Jan-March 2017
604
respondents
2nd Post intervention Survey of Classic CHC:
Jan-March 2017
Purposeful selection of half of the 50 Classic CHC = 24 CHC
Criteria:
Those CHC already had at least a baseline/midline survey
A representative spread of high, medium and low mobilised villages.
Random selection of at least 20 households from each village using the CHC registration list
At least 33% of the CHC households in each village
Respondents interviewed at household must be the CHC member
Membership cards must be used to verify data

11. SURVEY TOOL: Household Inventory
IPA selected indicators
Water Source: type, protected, distance, waiting time, quantity
Drinking water: storage, hygiene, taking method, covered, treated
Safe Sanitation: ownership, type, covered, clean, open defecation
Personal Hygiene: wash place, soap available, bedroom, clothes, children’s faces
Hand Washing: ownership, type, number, functional, used
Food Hygiene: storage, covering, animal access, floor clean, pot rack
Kitchen Facilities: place, fuel, ventilation, shelves, work surfaces
Solid Waste: compound swept, animal faces, type, recycling, rubbish pit
Green Environment: grass, access, drainage, trees, nutrition garden
Disease Prevention: mosquito net, usage, condition, immunisation, malnutrition

12. RESULTS: MoH/AA monitoring data in the RCT (IPA indicators)
Y.3.
Y.2.
Y.1.

13. Increased from 39% to 87% in one year Up to 91% in 3 years
Treated Drinking Water = boiled and/or filtered and/or chemically treated
Increased from
39% to 87%
in one year
Up to 91%
in 3 years
‘Improved Drinking Water Source’. The IPA baseline survey (Clemence et al.2016) recorded only 15% of the population was using unprotected water for drinking. Both the IPA trial data (Table 2) and the MoH/AA monitoring data (Table 3) agreed that there was little/no association between use of an ‘improved water source’ and the number of CHC sessions attended. This was to be expected as there was no provision for water source upgrading in the intervention. The slight increase shown in the CHC households from 55.9% to 68.9% may indicate choice of a safer water source, or it could be attributed to a new piped water supply scheme being constructed throughout the district during the time of the intervention. Two years after the end of training improved water source had improved to 81% who were using an improved water source, and yet no hardware was provided by us.
‘Adequate water treatment’: Sinaharoy et al. found that ‘Adequate water treatment’ did show significantly different rates of adoption by the different arms of the trial. This was also supported by our own finding that treatment of drinking water rose from 39.2% to 86.6% within a year; and kept increasing to 91% two years after the end of training. However, this rise could also be attributed to a large distribution of new water filters to the intervention area (CHC and control alike) which may have confounded findings (Barstow et al. 2016) in some way not explained by the data.

14. Improved Sanitation = covered latrine
Increased from
37% to 42%
in one year
Up to 68.9%
in 3 years
‘Improved Latrine’: Whilst IPA used this term to mean a latrine with a slab on the floor (presumably a cement slab) and a roof, our definition of an improved latrine was one with a well fitted cover whether it had a cement or mud/pole floor, as without this addition there is no control of the fecal-oral route taken by flies from the latrine pit to food. Coverage with this critical feature increased from 43% to 80% in Classic villages two years after the intervention ended, according to the MoH/AA monitoring data.

15. Improved Sanitation = Ventilated and Improved Latrine
Increased from
6% to 13%
in 3 years
‘Structurally complete latrine’: Although in the ‘classic’ arm IPA found no association of latrine ownership with attendance at meetings, we found an increase in structurally complete latrines from 89.8% beginner CHC (1-4 sessions) to 97.2% in advanced CHC households with >17 sessions by end of training, with a slight regression to 87% two years later. Regression is expected to some extent although rebuilding of latrines is more likely in CHC than non CHC households (Tobergte et al. 2016).
‘Use of soap’. The IPA baseline noted that 80% of respondents reported using soap and water during the previous handwashing. We observed use of ‘soap for handwashing’ by requesting children to demonstrate handwashing at each house, rose significantly from 49.6% at base line, to 85.8%, and continued to climb to 95% two years after end of training For a handwashing facility to be considered complete under the IPA criteria, the soap had to be left outside at the handwashing station; this was deemed foolish by households who could ill afford this luxury, as the soap would be stolen, eaten by goats or birds, or wasted by children. In many cases soap is kept carefully indoors, which may account for the lack of evidence of ‘handwashing station with water and soap’.

16. Increased from 9% to 77% in 3 years
Handwashing = use of a tippy tap / step & wash
Increased from
9% to 77%
in 3 years
‘Handwashing station: The trial team found no significant association between handwashing station and the study arm or with attendance at meetings. At IPA’s baseline, 50% reported washing their hands within the last hour, but only 468 out of 8,696 (5%) of respondents could point to a specific ‘handwashing station’. The interpretation of a ‘handwashing station’ may confuse the issue. Our baseline, like that of IPA showed a deceptively high level of handwashing ie. 100% However, if ‘Handwashing facilities’ is desegregated into various methods of handwashing, our baseline shows that 90.5% used the traditional method of a bowl of water, or poured water over hands from a bottle/jerry can. Therefore, a more accurate test of compliance with recommended practices is to measure the uptake of a ‘tippy tap’ (a plastic jerry can foot-operated facility) or the Rwandan equivalent (the manufactured Canacla - a step and wash technology) which increased from only 9.5% to 75.7% within a year. Contrary to the minimal changes found by IPA, our records found this significant increase was matched with a corresponding drop of traditional methods using a basin/bottle to 16.5% only six months after the end of the intervention. After the intervention during a period of no monitoring the coverage of handwashing facilities did drop to 53% (which may have coincided with the period of IPA data collection) but rose again to 77% in the next year when monitoring was resumed. The need for continuous follow up to maintain handwashing has been noted in other papers, although CHCs do show higher levels of handwashing than other methods (Whaley & Webster, 2011)

17. +80% In CHC Households BEHAVIOUR CHANGE TARGETS ACHIEVED BY 2017
1. Safe drinking water access and handling
91%
2. Hygienic latrines are used in CHC households
82%
3. Hand washing with soap is practiced
86%
4. Zero open defecation is achieved
99%
5. Safe disposal of children faeces
99%
6. Bath shelters are used
84%
‘Sanitary disposal of children’s faeces.’ We found that this practice was also unexpectedly prevalent at baseline (97.2%), which would tally with the wide coverage of latrines, as this would enable easy sanitary disposal of children’s faeces in the household latrine. The IPA baseline also records that 65% of child faeces were disposed in the latrine whilst 24% of children took themselves to the latrine, leaving only 1% practicing open child defecation. This indicator was not the most salient measure of hygiene behaviour change as there was little scope for change, the target of 80% for the whole period having been already achieved before the intervention. Our records show that by the end of the year all households were compliant with this recommended practice and that the same level of 99% was retained two years after training ended.
‘Use of soap’. The IPA baseline noted that 80% of respondents reported using soap and water during the previous handwashing. We observed use of ‘soap for handwashing’ by requesting children to demonstrate handwashing at each house, rose significantly from 49.6% at base line, to 85.8%, and continued to climb to 95% two years after end of training For a handwashing facility to be considered complete under the IPA criteria, the soap had to be left outside at the handwashing station; this was deemed foolish by households who could ill afford this luxury, as the soap would be stolen, eaten by goats or birds, or wasted by children. In many cases soap is kept carefully indoors, which may account for the lack of evidence of ‘handwashing station with water and soap’.
7. Solid Waste management
89%
8. Kitchen Yards are kept swept clean
96%
In CHC Households

18. Conclusion
Effective monitoring by communities themselves supported by a strong monitoring capacity of Ministry of Health may provide a more accurate assessment of hygiene behaviour change rather than externally controlled evaluations such as that performed in Rusizi district by IPA.
The Ministry of Health is more persuaded by the observations of its own councilors than by the counter-intuitive and, we believe, misleading RCT results by external evaluators.
The CHC approach used within a structured national programme is indeed a replicable intervention that can be used not only throughout Rwanda but also in other African Countries.
Kigali Action Plan, (AU,2015) recommends scaling up CHCs, based on the successful experience in Rwanda, to the ten poorest African countries to enable communities to better manage their own health.

19. QUESTIONS RAISED:
How can the negative RCT findings be so out of step with positive experience on the ground?
Choice of 4 out of 6 indicators were not appropriate
IPA results do not ring true as seen on the ground
No context was provided to provide an explanation for aparantly negative findings.
IPA reached a premature conclusion
Published findings with MoH knowledge or consent
Why was no context provided for the negative RCT findings, which were then published without MoH knowledge or consent?
Why were the indicators chosen for the trial so inappropriate?
i.e. two were already above 90% at baseline!
Why was the research component dictating time frames for the implementation? i.e. the training took place in the rains and time allowed was shortened.
Why was there so little interface with local authorities and partner NGO during the trial? i.e. research was started without local permissions
Choice of indicators
The baseline provided the rationale for a more appropriate set of indicators, the data collection tool used seemed not to have been adapted to the Rwandan context, with indicators selected for publication being more appropriate for countries such as India where only 24% of the county’s rural population have access to improved sanitation facilities, and where some members of the same team had just completed a similar RCT for the Total Sanitation Campaign (Boissen et al. 2014) with the main focus being on open defecation.
Of the six intermediate outcomes selected to represent the impact of the CHC training, three indicators (complete latrine, disposal of child faeces and handwashing) were questionable as the baseline was over 80%.
Two other indicators (water source and water treatment) were confounded by other interventions in the area.
This leaves handwashing as the one appropriate indicator selected by the RCT team. However, it is surmised that the definition of the term ‘hand washing station’ confused IPA enumerators, which skewed the data.
Are there ethical implications in the recommendation to stop CBEHPP, a national programme which is reaching out to millions of people across Rwanda?
What is the experience of other implementing NGOs in CBEHPP – do they tally with RCT findings?

20. Thank You Acknowledgements:
This intervention was funded by Bill & Melinda Gates Foundation ( )
Thanks to all communities of Rusizi District, village leaders and District Authorities and all MoH and Africa AHEAD staff who made this intervention possible.
To watch a short 5 minute video of one Nyambeho village,
of the best CHCs in Rusizi: Please use the following link: 
For a less high quality version of the video watch on You tube
For more information and to access this paper
For training in the CBEHPP Tools and Monitoring Website
Thank You