Impact Evaluation Methods July 9, 2011 Dhaka Sharon Barnhardt, Assistant Professor Institute for Financial Management and Research (IFMR)

Impact evaluation methods Non- or Quasi-Experimental Methods a. Pre-Post Simple Difference Differences-in-Differences Multivariate Regression Statistical Matching Interrupted Time Series Instrumental Variables Regression Discontinuity

Methods to estimate impacts Let’s look at different ways of estimating the impacts using the data from the schools that got a balsakhi Pre – Post (Before vs. After) Simple difference Difference-in-difference Simple or Multivariate regression Randomized Experiment Program participants before program Individuals who did not participate (data collected after program) Same as above, plus: (data collected before and after) Same as above plus: Also have additional “explanatory” variables

1 - Pre-post (Before vs. After) Average change in the outcome of interest before and after the programme Example: when establishing a causal link is not feasible measuring the impact of a govt. run literacy campaign state-wide; where it is difficult to construct a comparison group since everyone in the state is mandated to receive the “treatment” at the same time (Sakshar Bharat Campaign in India) Issues: does not take into account time-trend “response-shift bias” – change in the participant’s metric for answering questions from the pre to the post test Response shift bias is described as a “change in the participant’s metric for answering questions from the pre test to the post test due to a new understanding of a concept being taught (Klatt and Taylor-Powell, 2005).” For example, if a program is designed to teach parents effective discipline for children and some participants don’t know that spanking a child is considered inappropriate, they may report at the beginning of the program (pre test) that they are engaging in age appropriate discipline practices at home. However, if over the course of the program they begin to understand the relationship between age of a child and the “appropriate” discipline techniques, they may also report that they understand age appropriate discipline on the post testt – and thus, their evaluation data would reflect no program impact at all.

1 - Pre-post (Before vs. After) QUESTION: Under what conditions can this difference (26.42) be interpreted as the impact of the balsakhi program? Average post-test score for children with a balsakhi 51.22 Average pretest score for children with a balsakhi 24.80 Difference 26.42

Pre-post Method 1: Before vs. After Impact = 26.42 points? 75 50 25 26.42 points? 2002 2003

2 - Simple difference Example: Issues: A post- programme comparison of outcomes between the group that received the programme and a “comparison” group that did not Example: programme is rolled out in phases leaving a cohort for comparison, even though assignment of treatment is not random if Sakshar Bharat was rolled out in a few districts only at a time Issues: does not take into account differences that exist before the treatment (selection bias)

2 - Simple difference Compare test scores of… With test scores of… Children who got balsakhi Children who did not get balsakhi

2 - Simple difference QUESTION: Under what conditions can this difference (-5.05) be interpreted as the impact of the balsakhi program? Average score for children with a balsakhi 51.22 Average score for children without a balsakhi 56.27 Difference -5.05

What would have happened without balsakhi? Method 2: Simple Comparison Impact = -5.05 points? 75 50 25 -5.05 points? 2002 2003

3 – Difference-in-Differences or Double Difference Comparison of outcome between a treatment and comparison group (1st difference) and before and after the programme (2nd difference) Suitability: programme is rolled out in phases leaving a cohort for comparison, even though assignment of treatment is not random If Sakshar Bharat was rolled out in a few districts only at a time Issues: failure of “parallel trend assumption”, i.e. impact of time on both groups is not similar

3 – Difference-in-Differences Compare gains in test scores of… With gains in test scores of… Children who got balsakhi Children who did not get balsakhi

3 - Difference-in-differences Pretest Post-test Difference Average score for children with a balsakhi 24.80 51.22 26.42

What would have happened without balsakhi? Method 3: Difference-in-differences 75 50 25 26.42 2002 2003

3 - Difference-in-differences Pretest Post-test Difference Average score for children with a balsakhi 24.80 51.22 26.42 Average score for children without a balsakhi 36.67 56.27 19.60

What would have happened without balsakhi? Method 3: Difference-in-differences 75 50 25 19.60 6.82 points? 26.42 2002 2003

3 - Difference-in-differences QUESTION: Under what conditions can 6.82 be interpreted as the impact of the balsakhi program? Pretest Post-test Difference Average score for children with a balsakhi 24.80 51.22 26.42 Average score for children without a balsakhi 36.67 56.27 19.60 6.82

4- Simple/Multivariate Regression Change in outcome in the treatment group controlling for observable characteristics requires theorizing on what observable characteristics may impact the outcome of interest besides the programme Issues: how many characteristics can be accounted for? (omission variable bias) requires a large sample if many factors are to be controlled for

4 - Regression: controlling for pretest

Impact of Balsakhi - Summary Method Impact Estimate (1) Pre-post 26.42* (2) Simple Difference -5.05* (3) Difference-in-Difference 6.82* (4) Regression 1.92 * Statistically significant at the 5% level

Impact of Balsakhi - Summary Method Impact Estimate (1) Pre-post 26.42* (2) Simple Difference -5.05* (3) Difference-in-Difference 6.82* (4) Regression 1.92 (5) Randomized Experiment 5.87* * Statistically significant at the 5% level

Impact of Balsakhi - Summary Method Impact Estimate (1) Pre-post 26.42* (2) Simple Difference -5.05* (3) Difference-in-Difference 6.82* (4) Regression 1.92 (5) Randomized Experiment 5.87* *Statistically significant at the 5% level Bottom Line: Which method we use matters!

Another Example: Jaunpur Study 280 villages in Jaunpur, UP Intervention: Provided information to communities on education status and responsibilities of VECs Encouraged to test children and create community report cards to assess status of education Trained volunteers in villages to conduct after-school reading classes

Jaunpur Study Different impact estimates on reading level scores Method Impact Estimate (1) Pre-post 0.6* (2) Simple Difference -0.68* (3) Difference-in-Difference 0.31* (4) IV Regression 0.88* (5) Propensity Score Matching -0.03

Jaunpur Study Different impact estimates on reading level scores Method Impact Estimate (1) Pre-post 0.6* (2) Simple Difference -0.68* (3) Difference-in-Difference 0.31* (4) IV Regression 0.88* (5) Propensity Score Matching -0.03 (6) Randomized Experiment 0.06*