1. Microfinance and Home Improvement: Using Retrospective Panel Data to Measure Program Effects on Discrete Events Bruce Wydick Professor of Economics, University of San Francisco Visiting Professor, UC Santa Barbara joint with Craig McIntosh University of California at San Diego Gonzalo Villaran University of San Francisco

2. Background: Microfinance Summit: As of January 2006, 3,133 microcredit institutions have reported reaching 113,261,390 clients, 81,949,036 of whom were among the poorest when they took their first loan. Microfinance Summit: As of January 2006, 3,133 microcredit institutions have reported reaching 113,261,390 clients, 81,949,036 of whom were among the poorest when they took their first loan. Still amazing that don’t have robust results of positive microfinance impact (Armendáriz de Aghion and Morduch, 2005). Still amazing that don’t have robust results of positive microfinance impact (Armendáriz de Aghion and Morduch, 2005). Recent renewed emphasis on program impact appraisal (e.g. Easterly, 2006; Center for Global Development, 2006) Recent renewed emphasis on program impact appraisal (e.g. Easterly, 2006; Center for Global Development, 2006)

3. An “evaluation gap” has emerged because governments, official donors, and other funders do not demand or produce enough impact evaluations and because those that are conducted are often methodologically flawed.” --Center for Global Development (Saveduff, Levine, Birdsall, 2006 with E. Duflo, P. Gertler, etc.)

4. Problems with lack of quality impact studies: 1. Accurately measuring program impacts has historically been logistically difficult, time consuming, and costly.

5. Problems with lack of quality impact studies: 1. Accurately measuring program impacts has historically been logistically difficult, time consuming, and costly. 2. Many institutions would like to evaluate the effectiveness of their programs ex-post to implementation, creating problems with the establishment of baseline surveys, control groups, and other means of identification.

6. 3. Use of instruments or program rules (e.g. Pitt and Khandker, 1998) to obtain program impacts is theoretically appealing, but practically problematic: If available, instrumental variables will differ from one situation to the next. If available, instrumental variables will differ from one situation to the next. Finding instruments in a particular context strongly correlated with program access, but uncorrelated with impact variables, requires substantial ingenuity. Finding instruments in a particular context strongly correlated with program access, but uncorrelated with impact variables, requires substantial ingenuity. Point: complicates use of a standardized instrumental variable approach.

7. 4. Matching Models -- creating artificial controls in order to identify treatment effects. (e.g. propensity scores, nearest neighbor, etc.)

8. 4. Matching Models -- creating artificial controls in order to identify treatment effects. (e.g. propensity scores, nearest neighbor, etc.) Gomez and Santor (2003) use statistical matching model to identify the effect of group lending relative to individual lending among 1389 individual and group borrowers among 1,389 borrowers in Canadian lending institution.

9. 4. Matching Models -- creating artificial controls in order to identify treatment effects. (e.g. propensity scores, nearest neighbor, etc.) Gomez and Santor (2003) use statistical matching model to identify the effect of group lending relative to individual lending among 1389 individual and group borrowers among 1,389 borrowers in Canadian lending institution. Problem: Cannot control for unobservables.

10. 5. Randomized experiments--become very popular as way of ascertaining impact of development programs. Maximum degree of exogeneity in treatment and control, allowing means of overcoming self- selection, endogeneity, and omitted variable bias (common to microfinance) Maximum degree of exogeneity in treatment and control, allowing means of overcoming self- selection, endogeneity, and omitted variable bias (common to microfinance) Most elegant way of ascertaining impacts, and least controversial. Most elegant way of ascertaining impacts, and least controversial.

11. Difficulties: a) To create control group needed for identification of treatment, necessary that treatment withheld for some who desire it so impact can be measured on treatment group relative to the control…often undesirable or infeasible

12. Difficulties: b) Some treatments (e.g. microfinance) may take years to realize full effects on household--timeframe may not intersect with time one can “hold off” control group…"bleeding" of control group.

13. Difficulties: a) To create control group needed for identification of treatment, necessary that treatment withheld for some who desire it so impact can be measured on treatment group relative to the control…often undesirable or infeasible b) Some treatments (e.g. microfinance) may take years to realize full effects on household--timeframe may not intersect with time one can “hold off” control group…"bleeding" of control group. c) To avoid bleeding of control group, often short-term, but then only capture effects of initial adopters.

14. d) Point estimates from randomized experiments are subject to influence of time-specific economic shocks occurring within the relatively narrow timeframe of the experiment…understates true standard errors. e) Because randomized field experiments typically represent a snapshot of program impact over a short time frame, they are often unable to capture important dynamics of treatment impact. Ideally, we would like to understand how an intervention affects a treatment group over time.

15. Our paper presents a methodology for ascertaining welfare changes brought about by development programs that may be applicable in a variety of contexts (explain later). Main Advantages: Uses a single wave of cross-sectional surveying.Uses a single wave of cross-sectional surveying. Impact evaluation can be undertaken ex-post.Impact evaluation can be undertaken ex-post. No firm requirement for standard control group.No firm requirement for standard control group. Allows for a dynamic analysis of impactsAllows for a dynamic analysis of impacts

16. Our methodology appropriate when… Program has existed for a number of years. Program has existed for a number of years. Has been phased in over time in different geographical regions or identifiably separate populations for reasons that are independent of dependent variables. Has been phased in over time in different geographical regions or identifiably separate populations for reasons that are independent of dependent variables. Stable populations with little geographical movement. Stable populations with little geographical movement.

17. Methodology uses a single cross-sectional survey to create a retrospective panel data set based on discrete, memorable events in the history of households. e.g. install indoor plumbing, new house, purchase of first cell phone, miscarriages, infant deaths, land purchases etc. Identification of impact rests in analyzing the timing of these events with respect to the timing of treatment. Test is for differences in the probability of these major events within window surrounding the treatment. (Post vs. Pre--under conditions of exogeneity)

18. We apply methodology to studying the effects of a microfinance program in rural Guatemala on home improvements

19. Study discrete changes in the probability of major dwelling improvements, upgrades of walls, roofs, floors, the installation of indoor toilets, and the purchase of new land + compilation of these.

20. We apply methodology to studying the effects of a microfinance program in rural Guatemala on home improvements Study discrete changes in the probability of major dwelling improvements, upgrades of walls, roofs, floors, the installation of indoor toilets, and the purchase of new land + compilation of these. Use linear probability estimator that incorporates household and year fixed-effects. (Chamberlin, 1980)

21. Sneak Preview of Results: Microfinance loans for enterprise expansion is likely to exhibit significant, positive effects on some dwelling upgrades, especially to walls & floors. Roofs uncertain. Apparently not for toilets and land.

22. Identification of impacts is achieved through the existence of counterfactual, i.e. what would have happened to treatment group in the absence of a particular treatment. Counterfactual in a randomized field experiment in microfinance is the resulting level or change in impact variables realized within a subset of borrowers in the control group who desired credit but were prevented by researchers from receiving it.

23. Counterfactual that yields identification in our methodology is difference in probability of discrete events among those who received the treatment in separate years, controlling for these differences in years with village- and year-level fixed-effects.

24. Main contributions: 1. Offer a sequence of steps that include diagnostics on the data to check for supply-side & demand-side endogeneities in the rollout of a program. 2. Establish framework for thinking about when and how retrospective panel data can be used in impact analysis. 3. When data meets certain diagnostic criteria, allows us to examine the dynamics over probability of these discrete events before and after treatment & test for significance of a type of treatment effect.

25. Steps involved in methodology: Part A: Survey

26. Steps involved in methodology: Part A: Survey Step S1: Identify a program that has been phased in over a number of years in different geographical areas or among different populations.

27. Steps involved in methodology: Part A: Survey Step S1: Identify a program that has been phased in over a number of years in different geographical areas or among different populations. Step S2: Carry out random survey of program participants who have been given access to the treatment in different time periods.

28. Steps involved in methodology: Part A: Survey Step S1: Identify a program that has been phased in over a number of years in different geographical areas or among different populations. Step S2: Carry out random survey of program participants who have been given access to the treatment in different time periods. Step S3: Identify discrete historical changes with a theoretical basis for causality from the treatment & create historical panel.

29. Steps involved in methodology: Part A: Survey Step S1: Identify a program that has been phased in over a number of years in different geographical areas or among different populations. Step S2: Carry out random survey of program participants who have been given access to the treatment in different time periods. Step S3: Identify discrete historical changes with a theoretical basis for causality from the treatment & create historical panel. E.g. fresh water → reduced infant mortality

30. Steps involved in methodology: Part A: Survey Step S1: Identify a program that has been phased in over a number of years in different geographical areas or among different populations. Step S2: Carry out random survey of program participants who have been given access to the treatment in different time periods. Step S3: Identify discrete historical changes with a theoretical basis for causality from the treatment & create historical panel. E.g. fresh water → reduced infant mortality E.g. smallpox vaccine → lower instances of smallpox

31. Steps involved in methodology: Part A: Survey Step S1: Identify a program that has been phased in over a number of years in different geographical areas or among different populations. Step S2: Carry out random survey of program participants who have been given access to the treatment in different time periods. Step S3: Identify discrete historical changes with a theoretical basis for causality from the treatment & create historical panel. E.g. fresh water → reduced infant mortality E.g. smallpox vaccine → lower instances of smallpox E.g. microcredit access → higher enterprise profits → more rapid home improvements

32. Steps involved in methodology: Part B: Econometrics

33. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program.

34. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program. Step E2: Estimation of the Retrospective Intention to Treat Effect

35. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program. Step E2: Estimation of the Retrospective Intention to Treat Effect Step E3: Testing for Demand-Side Endogeneity

36. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program. Step E2: Estimation of the Retrospective Intention to Treat Effect Step E3: Testing for Demand-Side Endogeneity Step E4: Estimation of the Take-up Effect

37. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program. Step E2: Estimation of the Retrospective Intention to Treat Effect Step E3: Testing for Demand-Side Endogeneity Step E4: Estimation of the Take-up Effect Step E5: Treatment Window Regression and F-test of Take-up Effects

38. 2005 (BASIS/USAID funded) survey of 218 households located in 14 different villages near Quetzaltenango, Guatemala. 2005 (BASIS/USAID funded) survey of 218 households located in 14 different villages near Quetzaltenango, Guatemala. MFI: Fe y Alegria: 3,000 new clients/year MFI: Fe y Alegria: 3,000 new clients/year Questionnaire ascertained changes in different categories of dwelling improvement: upgrades to walls, roofs, floors, plumbing, and increases in land. Questionnaire ascertained changes in different categories of dwelling improvement: upgrades to walls, roofs, floors, plumbing, and increases in land. Each borrower was asked about changes in these variables during the history of the household, and the timing of these changes.Each borrower was asked about changes in these variables during the history of the household, and the timing of these changes.

52. Empirical Steps: Step E1: Check for supply-side endogeneity in the rollout of a program. Diagnostics: 1A: Is there endogeneity in the levels of the pre- treatment outcome? (Regress average pre ‑ treatment outcome on the year in which credit was offered to the village.)

53. Empirical Steps: Step E1: Check for supply-side endogeneity in the rollout of a program. Diagnostics: 1A: Is there endogeneity in the levels of the pre- treatment outcome? (Regress average pre ‑ treatment outcome on the year in which credit was offered to the village.) 1B: Is there endogeneity in the pre-treatment trend? (Regress average of the 1st difference of the pre- treatment outcome on year credit offered.)

54. Empirical Steps: Step E1: Check for supply-side endogeneity in the rollout of a program. Diagnostics: 1A: Is there endogeneity in the levels of the pre- treatment outcome? (Regress average pre ‑ treatment outcome on the year in which credit was offered to the village.) 1B: Is there endogeneity in the pre-treatment trend? (Regress average of the 1st difference of the pre- treatment outcome on year credit offered.) 1C. Is the rollout endogenous to shocks? (Run fixed effects using only pre-treatment data with dummy for 1st lead of year credit offered.)

56. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program. ✔ Step E2: Estimation of a Retrospective Intention to Treat Effect

59. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program. ✔ Step E2: Estimation of the Retrospective Intention to Treat Effect ✔ Step E3: Testing for Demand-Side Endogeneity

60. We estimate the following equation: (1) (1) where y it is a bivariate dependent variable that is equal to 1 if household i has housing upgrade in year t. v j is a village-level fixed effect,  t is a year-level fixed effect, X n are controls u it is an error term, and treatment dummy variable T is equal to 1 if household i first received a microfinance loan (or began receiving remittances) periods “ago,” and zero otherwise.

61. Data Set Example:

62. ...which can be adjusted for demand-side endogeneity: (2) (2) where y it is a bivariate dependent variable that is equal to 1 if household i has housing upgrade in year t. v j is a village-level fixed effect,  t is a year-level fixed effect, X n are controls u it is an error term, and treatment dummy variable T is equal to 1 if household i first received a microfinance loan (or began receiving remittances) periods “ago,” and zero otherwise. pre-treatment interactive term

63. Reason: Demand-side Endogeneity Suppose borrowing is an endogenous decision because people borrow in good economic times → creates upward bias, δ’s > 0 Suppose borrowing is an endogenous decision because people borrow when they are in difficult economic times → creates downward bias, δ’s < 0

65. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program. ✔ Step E2: Estimation of the Retrospective Intention to Treat Effect ✔ Step E3: Testing for Demand-Side Endogeneity ✔ Step E4: Estimation of the Take-up Effect

68. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program. ✔ Step E2: Estimation of the Retrospective Intention to Treat Effect ✔ Step E3: Testing for Demand-Side Endogeneity ✔ Step E4: Estimation of the Take-up Effect ✔ Step E5: Treatment Window Regression and F-test of Take-up Effects

77. Conclusions: Presented a methodology for ascertaining the impact of development programs such as microfinance that offers several advantages: 1.Can be used within existing client base.

78. Conclusions: Presented a methodology for ascertaining the impact of development programs such as microfinance that offers several advantages: 1.Can be used within existing client base. 2.Data can be collected in single x-sectional survey

79. Conclusions: Presented a methodology for ascertaining the impact of development programs such as microfinance that offers several advantages: 1.Can be used within existing client base. 2.Data can be collected in single x-sectional survey 3.Illustrates timing and dynamics of impact

80. Conclusions: Presented a methodology for ascertaining the impact of development programs such as microfinance that offers several advantages: 1.Can be used within existing client base. 2.Data can be collected in single x-sectional survey 3.Illustrates timing and dynamics of impact Other (easier) applications: Fresh water systems, Nutrition programs, Cash transfers, Vaccinations Electrification…