Different Methods of Impact Evaluation

How to measure impact? Assessing causality Event 2 (Effects) Event 1 Impact of program = outcome 1 – outcome 2 In practice, we compare two groups, one of which benefited from the program, the other one did not Event 2 (Effects) Outcome 1 Event 1 e.g. Education program/ treatment Causes Event 2 occurs if and only if Event 1 occurred before Event 1 No Education Program Event 2 Outcome 2 Causes

Constructing the counterfactual what would have happened in the absence of the program? … … for the people who benefitted from the program: we don’t observe it thus, impact evaluation will have to mimic it Counterfactual is constructed by selecting a group not affected by the program – this is the main challenge of impact evaluation Methods differ by the way the counterfactual is constructed

Non-Experimental Methods Differences between outcomes for treated and non-treated (in our ex: reached / not) are the sum of Inherent differences: “SELECTION” BIAS TREATMENT EFFECT: that we want to isolate from 1 The essence of non-experimental methods is to find a way to redress the selection bias ex-post. The critical objective of impact evaluation is to establish a credible comparison group – a group of individuals who in the absence of the program would have had outcomes similar to those who were exposed to the program. However, in reality it is generally the case that individuals who participate in a program and those who were not are different: programs are placed in specific areas (for example, poorer or richer areas) individuals are screened for participation in the program (for example, on the basis of poverty or on the basis of their motivation) and, in addition, the decision to participate is often voluntary. For all of these reasons, those who were not exposed to a program are often not a good comparison group for those who were, and any differences between the groups can be attributed to two factors: pre-existing differences (selection bias) and the impact of the program. Since we have no reliable way to estimate the size of the selection bias, we typically cannot decompose the overall difference into a treatment effect and a bias term. Selection bias disappears in the case of randomization

Non-Experimental Methods Simple Difference Multivariate (Multiple) Regression “Difference in difference” (Multiple Regression with Panel Data) Matching Randomization/RCT’s (advanced topic covered separately) The critical objective of impact evaluation is to establish a credible comparison group – a group of individuals who in the absence of the program would have had outcomes similar to those who were exposed to the program. However, in reality it is generally the case that individuals who participate in a program and those who were not are different: programs are placed in specific areas (for example, poorer or richer areas) individuals are screened for participation in the program (for example, on the basis of poverty or on the basis of their motivation) and, in addition, the decision to participate is often voluntary. For all of these reasons, those who were not exposed to a program are often not a good comparison group for those who were, and any differences between the groups can be attributed to two factors: pre-existing differences (selection bias) and the impact of the program. Since we have no reliable way to estimate the size of the selection bias, we typically cannot decompose the overall difference into a treatment effect and a bias term. Selection bias disappears in the case of randomization

Simple difference Simple difference is a first measure – why is it not sufficient? There may be differences between the two groups (age, location, gender, initial endowment, bargaining power) So we may want to control for these differences

Multi-variate regression Suppose we can observe these differences. Age composition of the group, initial infrastructure in the school, level of education, … We can include all these variables in a regression: Y = a.T + b.Age + c.Infr + d.Edu +… A regression provides the linear combination of observable variables that best “mimics” the outcome Each coefficient represents the effect of each variable Will give us the effect of the treatment everything else being equal, or more exactly every other observable characteristics being equal

Multi-variate regression Problems of the regression You may want to include many many variables, to control for as many characteristics as possible Problem of sample size (degrees of freedom) More important: do you have measures of everything? Bargaining power, Pro-activeness, Intrinsic motivation, hopelessness There are unobservables

Panel data Simple difference: before/after Counterfactual = same group before the program Can we trust this? Assumption = would have remained the same Ex: Police project Double difference Control for the situation before the program Ex: Group 1 = Treatment Group; Group 2 = Control Group 2006: Group 1 : 30 Group 2 : 60 2009: Group 1 : 50 (+66%) Group 2 : 90 (+50%) Effect = +16% Assumption: they would have grown at the same pace Not sure…

Matching We compare pairs of 2 individuals for which the values taken by ALL variables are the same

Matching Variation: Propensity Score Matching: all the variables do not need to be exactly the same, but you look for individuals which have the same “profile” Problems This matching method requires a big dataset: find pairs on a sufficient number of variables What about unobservables?

An example Case study: US Congress elections, 2002 60 000 phone calls to potential voters to encourage them to vote; 25 000 reached Outcome: did they actually go vote? 1st method: compare the 25 000 (reached) vs. the 35 000 (not reached) 2nd method: introduce co-variates in a regression 3rd method: introduce baseline data (vote in 1998) 4th method: do a matching

1st comparison: we suspect a selection bias Reached Not Reached Difference Female 56.2% 53.8% 2.4 pp* Newly Regist. 7.3% 9.6% -2.3 pp* From Iowa 54.7% 46.7% 8.0 pp* Voted in 2000 71.7% 63.3% 8.3 pp* Voted in 1998 46.6% 37.6% 9.0 pp* Selection Bias: differences in observable / unobservable characteristics → differences in outcome not due to the treatment 13 13

Non-Experimental Methods Estimated Impact 1 – Simple Difference 10.8 pp * 2 – Multiple regression 6.1 pp * 3 – Multiple regression with panel data (diff-in-diff) 4.5 pp * 4 – Matching 2.8 pp * 5 – Randomized Experiment 0.4 pp

For more details, please read Case Study: “Get out the vote For more details, please read Case Study: “Get out the vote? Do phone calls encourage voting” under References