May 8, 2012 MWP-K Learning Event Monitoring, evaluation, and learning (MEL) framework for the Millennium Water Program, Kenya

Background Collaboration between:  Millennium Water Alliance  MWP-Kenya partners (CARE, CRS, FH, Living Water, Lifewater, World Vision)  Center for Global Safe Water, Emory University Objectives:  Improve effectiveness and increase sustainability through more data-driven decision making  Build foundation of learning and collaboration through MEL to support future continuation of MWP-Kenya

Challenges Addressed by MEL Framework Monitoring usually limited to project activities and outputs (e.g. # latrines built, # trainings completed) Final evaluation is only time seized to assess achievement of outcomes (e.g. behavior change) Want to use monitoring to track progress toward intended behavior change outcomes to enable a change of course if needed

Challenges Addressed by MEL Framework Minimal information sharing & reflection on monitoring data or other lessons throughout project life cycle. Want to intentionally budget time and persons responsible for guiding data sharing and reflection to maximize lessons learned to improve practice

Challenges Addressed by MEL Framework Sustainability seldom evaluated after project life cycle due to close of project, budget limitations, etc. Want to incorporate M&E indicators into tools used during project life cycle to assess likelihood of future sustainability (are factors in place to improve longevity of benefits after project is finished?)

Components of MEL for MWP Kenya 1. Key program indicators 2. Evaluation: baseline and follow-up 3. Monitoring: systematic, regular data collection  Includes process and tools to guide “Data for decision- making”: Sharing, feedback, and action;  Monitoring tools to assess progress towards outcomes 4. Focused topical learning  Includes partners’ leadership in special learning topics;  Emory support for some sub-studies (e.g. sanitation assessment)

1. Key Program Indicators Emory assisted partners in developing logical framework in May 2010 Indicators include:  Outputs (activities completed by partners)  Outcomes = central focus (changes that beneficiaries adopt in response to program)  Includes indicators that suggest future sustainability  E.g. School has specific WASH budget; school management committee is involved in WASH program

2. Evaluation Emory facilitated baseline assessment planning and implementation:  Led development of evaluation tools and protocols  Household, school, and health facility surveys  Analyzed data Each organization collected data prior to implementation

3. Monitoring Output-level indicators  Responsibility of each partner  Tracked through internal monitoring system Outcome-level indicators  Measured throughout course of project  To engage implementers to critically reflect on and address programmatic challenges in real-time  Monitoring activities conducted quarterly or once per school term

Data for Decision-Making Sharing of experiences and lessons learned between partners Data Reflection Tool (DaRT)  Summarizes findings from each monitoring and learning activity  Project expectations  Progress toward achieving indicators  Plans to address lack of progress  Potential threats to sustainability  Shared and discussed within partnership on a quarterly basis before Program Management Group meeting

Data for Decision-Making Quarterly Work Flow Complete monitoring activities quarterly Summarize findings into Data Reflection Tool (DART) MWP Coordinator reviews DARTs Meeting: Internal Learning/ Reflection Meeting: Internal Learning/ Reflection Meeting: Pre-PMG Learning/ Reflection Meeting: Pre-PMG Learning/ Reflection Meeting: PMG Meeting: PMG 1.Flag issues for follow- up 2. Use to guide agenda for Pre-PMG meeting 3.Distribute copies at pre-PMG meeting Discussion of: 1.What’s going well/ not as well as expected 2.Possible course changes 3.Additional info needs Discussion of: 1.Highlights from DARTs 2.Lessons learned, best practices 3.Group feedback 4.Issues to raise at PMG Discussion of: 1.Issues flagged at Pre-PMG meeting 2.Lessons learned & best practices Send Decide: Needed course changes Additional learning needs Decide: Needed course changes Additional learning needs = Responsibility of each organization = Responsibility of Coordinator = Responsibility of all partners together

Summary of Monitoring Tools ToolObjective Rapid household observations Observe and inquire about WASH practices in households in the intervention area to track changes toward achievement of intended outcomes and understand the trends in behavior change. Water point observations and management committee key informants interview Observe repair or maintenance needs; observe practice of integrated water resources management (IWRM); understand patterns of use for the water point, financial management practices, maintenance practices, and management challenges. Water point user interviews Determine functionality, use patterns, and perceptions of water point and management committee from community perspective. Provides a double check on some information gathered in the management committee interview. School monitoring visit Determine whether and how the school enacts the intervention on a day-to-day basis and assess threats to sustainability that might need to be addressed. School management committee key informant interview Understand the degree to which the SMC is involved in supporting school WASH activities, and encourage their support by inviting them to join you in your monitoring visit observations. Pupil interviews Gain pupil perspective on WASH provision and education at school and preferences. Provides a double check on some information gathered in the school assessment.

4. Focused Topical Learning Proposed Learning Leader Topic CARE Improving pupil handwashing behavior change in schools CRS Improving utilization of household latrines FH Improving community handwashing behavior in ASAL, nomadic areas LWI Improving financial management of water sources WVI Improving water supply sustainability CARE Improving pupil handwashing behavior change in schools Partners expressed interest in specific topics for further investigation As learning leader, each org. should organize one meeting with partners to discuss lessons on topic

Summary Thoughtful, timely collection of relevant data Data used to drive decision-making:  Guide targeting, technology choice, and program approaches to improve effectiveness and sustainability  Feed back findings into continued program activities in a timely and meaningful way  Engage donors, beneficiaries, and government stakeholders in the water, sanitation, and hygiene (WASH) sector Embrace benefits of working in partnership  Collective learning and reflection on best practices & lessons learned