1 Assessing the Impact of Microfinance: A Methodological Study Using Evidence from India Maren Duvendack Procedural Paper Presentation 23 May 2008 Supervisors: Arjan Verschoor & Nitya Rao

2 Introduction to Microfinance What is microfinance? Provision of financial and non-financial services to low-income households Microfinance important strategy in the fight against poverty Importance of microfinance recognised by United Nations and Nobel Prize Committee No clear empirical evidence yet that microfinance has positive impacts Impact assessments crucial for donors and microfinance institutions

3 Introduction of Research Project Challenge of every impact assessment: Measurement of counterfactual Elimination of biases (i.e. selection & attrition bias) Limited number of rigorous impact studies exist Study intends to focus on methodological challenges of microfinance impact assessment studies Suggest solutions to bias problem

4 Research Questions What is the impact of microfinance on the households economic and social well-being? How are microfinance assessment studies measuring the impact of microfinance? What are the methodological challenges of microfinance impact assessments? How can a rigorous treatment of biases, in particular drop-outs, improve the accuracy of impact assessment studies?

5 Research Context Financial exclusion of Indias poor recurring problem for almost 100 years Access to finance poverty reduction, thus Indian government launched various policy initiatives aimed at financial inclusion BUT: Most government-run subsidised credit programmes had negative effects Emergence of microfinance in India mainly due to lack of effective government policies

6 Research Context Emergence of microfinance in India in the 1990s Tremendous growth of Indian microfinance in terms of outreach and loan disbursements BUT: Only 8 impact assessment studies conducted in India Studies vary significantly in terms of scope and approach They investigate one or more of the following impacts: Poverty reduction Financial services Womens empowerment Studies provide conflicting results, impact of microfinance unclear Thus, more systematic approach to impact assessments needed

7 Conceptual Approaches Core elements of conceptual frameworks in impact assessments: The impact chain model The units of assessment The impact type Agent Behaviours and practices over a period of time Modified behaviours and practices over a period of time Outcomes for the agent and/or other agents Program Intervention Modified outcomes for the agent and/or other agents The difference between outcomes is the impact Mediating Processes Impact Source: Hulme, The impact chain model:

8 Conceptual Approaches Units of assessment: Individual, enterprise, household, community and institutional level Majority of studies examine impact at multiple levels Identification of impact type: Economic, social or socio-political impacts Early impact studies mainly investigated economic impacts, using indicators such as income, assets and expenditure In the 1980s, focus on social impacts, using indicators such as education, health, housing and sanitation More recently, shift towards socio-political indicators such as womens empowerment

9 Paradigms of Impact Assessments Attribution additional challenge of impact assessments Two main paradigms can be extracted commonly used to demonstrate attribution: Scientific MethodHumanities Tradition

10 Scientific Method Typically attempts to attribute effects of an intervention to its causes by utilising either… DesignProsCons …an experimental design Free from biases Delivers robust results Extensive cooperation from MFI needed Time & cost intensive Raises ethical questions …a quasi- experimental design Attempts to mimic experimental design Most popular design among MF impact assessors Identification of identical control group difficult without introducing biases …a non- experimental design Less time & cost intensive than other two designs Not particularly practitioner- friendly due to application of econometrical techniques

11 Humanities Tradition Humanities tradition seeks to explain & interpret the underlying processes of an intervention Dual function: Triangulation to crosscheck quantitative data Provides understanding of changes in social relationships Difficulties in demonstrating attribution due to lack of control group approach Causality inferred by collecting data on causal chain by interviewing programme participants, then comparison to data from areas which did not have access to programme

12 Methodological Challenges: Biases Biases common occurrence in impact evaluations adversely effect impact results, thus solution crucial Typically the following biases occur in the context of microfinance: Selection bias: self-selection & non-random programme placement Attrition bias Only handful of rigorous impact studies exist that control for biases: Hulme and Mosley (1996) Coleman (1999) Pitt and Khandker (1998) Alexander and Karlan (2007)

13 Selection Bias – Hulme & Mosley, Coleman Hulme and Mosely (1996) study of microfinance programmes in seven different countries Controlled for self-selection bias but not non-random programme placement bias Novelty: sampling of prospective clients as a control group Mixed results, depending on programme design and country context Coleman (1999) study on Thailand, uses village-level fixed-effects to control for non-random programme placement bias Also, he uses Hulme & Mosleys (1996) approach of sampling prospective clients as a control group Difference-in-difference approach employed Little impact found, more importantly microfinance led to vicious circle of bad debts

14 Selection Bias – Pitt & Khandker Until today, most rigorous attempt at controlling for selection bias Quasi-experiment & eligibility requirements used to measure programme impact Primary eligibility criterion: landownership Treatment Village Control Village Overall findings: microcredit has positive impacts BUT: accuracy of results disputed due to lax enforcement of eligibility criteria Econometric debate between Pitt & Khandker and Morduch, not resolved until today Eligible but do not participate Participants Not eligible Would be eligible Would not be eligible Source: Armendáriz de Aghion and Morduch, 2005.

15 Selection Bias – Solution? Propensity score matching (PSM) popular method used to eliminate selection bias Works by matching participants to non-participants based on predicted probability of programme participation or the propensity score Basis for matching: observable characteristics drawback Underlying assumption: no selection bias due to unobservables Combine PSM with difference-in-difference, picks up on unobservables but baseline data set required PSM results good approximation to those obtained under experimental approach

16 Attrition Bias Drop-out rates estimated to be between 3.5% to 60% in microfinance programmes worldwide Two different types of clients exiting: Graduates Drop-outs Attrition bias neglected by majority of studies, Alexander and Karlan (2007) one of the few recognising its importance Solution to attrition bias: Better sampling Systematic interviews with drop-outs

17 Methodology – Research Design Mainly a quantitative study with selected qualitative elements Questionnaire survey of 500 households Semi-structured interviews with selected key borrowers, in particular drop-outs Study proposes to employ propensity score matching (PSM) as a means to control for selection bias as well as attrition bias novelty in the context of microfinance Requirement: sampling of participants and non- participants as well as drop-outs

18 Methodology – Overview (1) Research QuestionsStepsMethodsChallenge 1) What is the impact of microfinance on the households economic and social well-being? Household survey, n=500, data collection on economic & social indicators, e.g. income, assets, education, health, etc. Administration of questionnaires, max. 1.5 hours and as pre-coded as possible Collection of appropriate income data 2) How are microfinance assessment studies measuring the impact of microfinance? Control group needed to identify counterfactual – what would have happened had the programme not existed - requires sampling of treated (i.e. part.) and non-treated (i.e. non- part.) PSM helps to create control group which is very similar to treatment group, only difference: control group did not participate Identificatio n of counterfact ual

19 Methodology – Overview (2) Research QuestionsStepsMethodsChallenge 3) What are the methodological challenges of microfinance impact assessments? Simple comparison of impact indicators between part. and non-part. leads to distorted results due to differences in observable & unobservable characteristics PSM, eliminates selection bias due to observables Possibly conduct semi- structured interviews with selected part. and non- part. to understand role of unobservable characteristics Correction for selection bias 4) How can a rigorous treatment of biases, in particular drop-outs, improve the accuracy of impact assessment studies? Sampling of drop- outs in addition to part. and non-part. PSM Semi-structured interviews with selected key drop-outs to understand reasons for attrition Tracing drop-outs

20 Methodology – Sampling Procedure Study proposes to employ multistage cluster sampling, as illustrated by figure Sampling in stages: first, identify large areas, then narrow them down by selecting smaller areas within those larger ones Research location: Andhra Pradesh Sample selection criteria: Mature microfinance programmes preferred, at least 5 years of operation Participants: 4-5 loan cycles required Region: Telangana District: Khammam Mandal: tbd Village: tbd State: Andhra Pradesh Household: tbd

21 Methodology – Ethics Oral and/or written consent of research participants shall be obtained before embarking on data collection Data collected shall be kept confidential and will be anonymised Reliance on research assistant and translators is expected, they shall be treated with the utmost respect and their expenses shall be covered by the researcher

22 Timeline Preparation Fieldwork Preparation Procedural Paper Procedural Paper Presentation Fieldwork Recce 23 May Fieldwork in India Data Analysis Writing-Up Jan Mar May Jul Sep Nov

23 Q & A Session