Outcomes Working Group Webinar 3: Indicators to measure outcomes Presenters: Bobbi Gray, Research and Evaluation Specialist, Freedom from Hunger Anne Hastings, Executive Director, Microfinance CEO Working Group Working Group Facilitator: Frances Sinha, Director EDA Rural Systems (India) and board member of SPTF 4 February 2015

Agenda Introduction ( 5mins ) Bobbi (FFH): criteria to develop indicators related to health outcomes; application by 4 MFIs, lessons ( 15-20mins ) Anne (MCWG): developing a system for standardized measurement, review of outcomes studies (69), selecting themes and indicators for microfinance (15-20mins )  DISCUSSION

Recap Previous webinar : Theory of Change framework: ▫Defining short-term/intermediary and long term changes ▫Assumptions – links from different services/uses to changes ▫Attrition factor - not all participants will stay the course Next logical step to identify relevant practical indicators to measure changes - and the steps that lead to changes: Criteria for identifying practical, relevant indicators Selection relevant to microfinance What is the experience? Can we build consensus on a ‘standard menu’ ?

Indicators The basics Specific, relevant - define objectives Practical to measure - credible Unambiguous - clear Can compare, benchmark Limited number – necessarily selective Trade-offs in e.g. what is practical to measure (household income or expenditure) and/or unambiguous (women’s role in decision making)

Health Outcome Performance Indicators Project

6 Theories of Change: Improved Health Access to and use of financial services: loans; insurance; savings; payments; health loans and savings Increased income Consumption smoothing Coping with shocks Seek prompt medical treatment Seek preventive health care Access to and use of health services: education, provision of curative and preventive health services Improved health knowledge and seeking prompt medical treatment and preventive health care Integrated health and financial services: direct provision and linkages between sectors Cross-sectoral efficiency gains in provision of financial and health services to poor populations Seek prompt medical treatment Seek preventive health care Improved health outcomes

7 Choosing Health Indicators Criteria FeasibilityUsabilityUsability/ Reliability Likelihood of inclusion Measurabl e by an FSP Can be reported in client survey Can change in short- term Addresses relevant measures for FSPs Cannot rely on specific interventio ns to change outcomes Be applicable for both genders Can be benchmarke d to other data (MDGs, regional data, etc.) Reliability PPI/PATYes MaybeYes High Food security index Yes MaybeHigh Use of preventive health services Yes Maybe YesMaybe High/ Moderate Access to safe drinking water (MDG 7) Yes MaybeYesMaybeYes MaybeHigh/ Moderate

8 Current Pilot Partners

9 Survey Adaptations  Poverty measurement – Use of country-specific PPIs  Food security and nutrition –added a focus on food items in India to reflect stronger focus on nutrition  Preventive health care – focuses on institutional births in India; annual exams and Pap tests in Peru; various annual exams in the Philippines and use of health insurance (PhilHealth)  Curative Health Care: Questions same in all three countries – forgoing medical treatment and purchase of medicines due to cost  Water and Sanitation– Focuses on defecating in the open and treating water to make it safer to drink in India; Open defecation, water sources, and water treatment in Peru; Water sources and treatment of water in the Philippines  Attitudes: Only measured in Peru and Philippines, accesses levels of confidence related to ability to cover future medical costs and seek adequate medical care

10  Standardization of indicators may be difficult.  Proceed with caution in the interpretation of results.  Baseline values will be important to establish.  Baseline values with high levels of performance may not be useful to track.  The value of statistical analysis – ex. correlations between indicators of interest with poverty – can help refine our “theory of change” as well as determine which indicators may be the most useful to help us understand changes in client outcomes.  Who to track and for how long? This is a very important question to answer as it influences which indicators will be the most useful.  This process requires patience. Lessons Learned

Collaborating on Outcomes Accion BRAC & BRAC International CARE Access Africa FINCA International Freedom from Hunger Grameen Foundation Opportunity International Pro Mujer VisionFund International Women’s World Banking

The Goal of Our Work on Outcomes To develop, test and disseminate a common core measurement and monitoring system designed to:  Be cost-effectively embedded within an MFI’s operations  Use industry-standard indicators and metrics (PPI)  Provide affiliated MFIs with actionable client outcome data that can be tracked over time  Allow MFIs the flexibility to measure those changes in client outcomes they wish to affect using the same indicators for the same outcomes  Enhance the Working Group’s ability to demonstrate and improve the benefits of Responsible Microfinance for the clients being served  Contribute to the sector’s understanding of client outcomes

The Proposed Method 1.Select the indicators to test 2.Target countries and the MFIs 3.Pilot the indicators  Social performance (SP) leads will work with their partners to collect, analyze and report on the data  SP leads and MFI partners would convene to discuss experiences, identify lessons learned and develop recommendations for future use  SP leads will develop and publish a series of briefs by outcome area with theory of change and recom- mended indicators  Package and disseminate the final indicators along with supporting documentation

Lessons from the Working Group’s Review of Outcomes Phase 1: Catalogued 69 different outcomes research from all 8 members and their affiliates ▫Wide variation in product studied, methodology, indicators, quality of research, etc. ▫Difficult to draw definitive conclusions when comparing outcomes across programmes

Different methods

Different themes

Some Generalizations from Phase I Microfinance … ▫ Leads to a number of positive outcomes in various contexts ▫ Does not lead to too many negative outcomes ▫ Is appreciated by clients ▫ Is strengthened by staff ▫ Can be a good conduit for other services It may… ▫ Smooth consumption ▫ Promote empowerment ▫ Be most helpful for the least poor  We also know…  Credit helps but is not transformative  Coping strategies are sometimes used to make loan payments  That we don’t know much about drop outs

A Closer Look at Education Studies 1.There was a big range of indicators being used across and within the networks. 2.There was little consistency across institutions. 3.Most studies only looked at one dimension of education.

Phase 2: Identifying Potential Indicators for 7 Types of Client Outcomes 1.Food security 2.Coping strategies/shocks 3.Economic poverty (income or financial status) 4.Health 5.Assets, housing and business 6.Social capital and empowerment 7.Child and youth education

Phase 2: Summary of Indicators Food security % of households that are food insecure (levels measured vary by tool) Coping strategies/shocks % of households that had to [make a specific adjustment or level of change] to cope with the effects of [household or community shock] Economic poverty (income or financial status) % of households living above/below a given poverty line Health % of households with [level of access] to health care [services/supplies]

Phase 2: Summary of Indicators Assets, housing and business % of households that were able to [purchase assets/improve their homes] in the past year Social capital and empowerment % of [women] expressing confidence in making decisions regarding [specific category] % of clients who perceive [specify benefit or value] in group participation Child and youth education % of children in the household who are regularly attending school % of households with increased ability to pay for school fees

Discussion 1.Any questions/clarifications? 2.What is your experience in indicator selection? 3.Can we aim at standardisation - synchronisation? 4.Our working group: contributing to guidelines for selection (and a menu) of indicators to measure client outcomes?

Thank you For follow up, please contact: Please note: presentations and recordings from all Outcomes Working Group Meetings are being posted to the SPTF website, working groups page: groupshttp://sptf.info/sp-task-force/working- groups