A methodology for evaluating education and training activities A CASE STUDY IN ETHIOPIA 20 TH OCTOBER 2015

2 HOW CAN WE EVALUATE EDUCATION AND TRAINING ACTIVITIES? Have capacities of government and NGOs increased? Has health and well- being improved? Has water and sanitation improved?

3 OBJECTIVES AND STUDY PHASES OBJECTIVE: To develop a methodology for evaluating the effectiveness of WASH training programs, and to test and refine this methodology in a field setting. STUDY PHASES: 1.Review methods for measuring and reporting results of education and training in WASH 2.Investigating metrics for evaluating education and training 3.Developing evaluation methodology for education and training, and piloting in Nepal and Peru 4.Modifying methodology, and piloting a new methodology in Ethiopia

4 Steps in the methodology: 1.Develop a theory of change for the education program 2.Develop indicators based on the Kirkpatrick model 3.Develop data collection tools 4.Collect data 5.Analyze and interpret data 6.Feed results back to planning process This methodology was piloted in an evaluation of a WASH training program for Health workers in Ethiopia. EVALUATION METHODOLOGY

5 STEP 1: THEORY OF CHANGE FOR EDUCATION PROGRAM 2 day WASH Awareness Training Delivered to Health Workers Health workers learn knowledge, attitudes, skills Health workers are more effective at spreading WASH messages in communities Community members learn about WASH options Community members practice better WASH

6 STEP 2:INDICATORS BASED ON KIRKPATRICK TOOL Kirkpatrick stage Examples of indicators Reaction  How relevant the training was  How useful the tools used in the training were Learning  Key knowledge, skills and attitudes related to WASH that HEWs had learned Behaviour  Changes to teaching methods of HEWs in communities as a result of the workshop Results  Changes in the community WASH situation where the trained HEWs are working.

7 STEPS Develop data collection tools  Interview protocols: Health Workers, Community members 4.Collect data  Total 40 interviews, 1 week 5.Analyze and interpret data  Created graphs, developed findings and recommendations 6.Feed results back to planning process  Used the findings to make improvements to the education program

8 FINDINGS AND RECOMMENDATIONS Kirk. Level Findings and recommendations Reaction  Workshop met expectations, and was relevant  Workshop tools (pictures, posters, games) were effective.  Need to translate materials into regional dialects Learning  Learned transmission and blocking, and critical hand washing times.  Refresher training needed on HWTS topics  Improvement needed on tippy tap construction skills Behaviour  Health workers used new methods to teach WASH in communities  Health workers need to prioritize WASH topics Results  Community members had basic knowledge of WASH  Latrine ownership and use was high  More emphasis on HWTS needed at community level

9 STRENGTHS AND WEAKNESSES OF EVALUATION METHODOLOGY StrengthsWeaknesses  Theory of change was useful in determining what to measure.  Easy to communicate and understand.  Efficient to apply.  Provided useful information for improvement  No baseline for this case  Analysis of qualitative data can be subject to interpretive bias

10 NEXT STEPS  Further develop evaluation support materials for clients and partners to implement similar evaluations  Develop a modified “light” methodology, appropriate for organizations where staff have less time to dedicate to the evaluation ANY QUESTIONS??