1. CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010

2. Motivation 2  Goal of any evaluation is to estimate the causal effect of intervention X on outcome Y.  Example: does an education intervention improve test scores (learning)? Reducing class size Teacher training In-school nutrition

3. Causation is not correlation! 3  Any two variable (X and Y) can move together 1.Male teachers & academic performance of students. 2.Health and income.  But, they may have nothing to do with each other.  Other explanations?

4. Evaluation problem: Potential Outcomes Approach  Ideal way to evaluate the impact of an intervention:  observe agent in and out of program, at a point in time.  But, think about the only way in which we can evaluate the impacts of an intervention:  observe agent in or out of program, at any point in time.

5. How to assess causality?  Let Y= outcome of interest (test score) P= participation in program = 1 if in = 0 if out  Formally, program impact is: α = (Y | P=1) - (Y | P=0)  Program Impact: difference in outcomes for individuals in and out of program. Outcome w/ program Outcome w/out program

6. Another Way to Think of Evaluation Problem  The problem we face is that:  (Y | P=0) is not observed for program participants.  (Y | P=1) is not observed for non-participants  Missing Data Problem:  Counterfactual not observed.  what would have happened to agent without the intervention?

7. Solving the evaluation problem  Generate the counterfactual  find a control or comparison observation for agent facing the intervention.  Criteria for selecting comparison observation: 1. Observationally similar, at baseline (and after intervention). 2. Face same contemporaneous “shocks” as the treatment group.

8. “Counterfeit” Counterfactuals 1.Before and after:  Same individual before the treatment 2.Non-Participants:  Those who choose not to enroll in program  Those who were not offered the program

9. “Counterfeit” Counterfactual Number 1: Before and After 9  Consider how you might evaluate an agricultural assistance program.  Suppose program offers free/subsidized fertilizer.  Compare rice yields before and after  Q: If you find no change in rice yield, can you conclude the program failed?  What else changed? Drought? Lots of rainfall?

10. Scholarship Program and School Enrollment, Before and After Time Y After A B t-1 t O Before  Ultimate goal is to estimate α (Y it | P=1) - (Y i,t | P=0)  Estimate the impact on treated individuals: " A-O " =(Y i,t | P=1) - (Y i,t-1 | P=1)  Second, estimate counterfactual " B-O " =(Y i,t | P=0) - (Y i,t-1 | P=0)  “Impact” = A-B α’ α’

11. Scholarship Program and School Enrollment, Before and After Time Y After A B t-1 t O Before But, impact " A-B " may misrepresent true counterfactual.  Suppose C is the correct counterfactual.  Here, the impact of the intervention is " A-C ". α ’’ C

12. “Counterfeit” Counterfactual Number 2: Non-Participants…. 12  Compare non-participants to participants  Counterfactual: non-participant outcomes  Impact estimate: α i = (Y it | P=1) - (Y j,t | P=0)  Assumption: (Y j,t | P=0) = (Y i,t | P=0)  Issue: why did the j’s not participate?

13. Non-participants Example : Job Training and Employment 13  Compare employment & earning of individuals who sign up for training to those who do not.  Who signs up?  Those who are most likely to benefit, i.e. those with more ability  Would have higher earnings than non-participants without job training  Poor estimate of counterfactual

14. Non-participants Example 2: Health Insurance and Demand for Medical Care 14  Compare health care utilization (# doctor visits) of those who got insurance to those who did not.  But, who buys insurance?  those who expect large medical expenditures (unhealthy)  Those who do not buy insurance have less need for medical care.  Poor estimate of counterfactual

15. The problem is selection bias. 15  Selection bias: People choose to participate in program for specific reasons.  Problem occurs when reasons for participation are related to the outcome of interest:  Job Training: ability and earning  Health Insurance: health status and medical-care utilization.  Cannot separately identify impact of the program from these other factors/reasons

16. Need to know… 16  Know all reasons why someone gets the program and others not  reasons why individuals are in the treatment versus control group  If reasons correlated w/ outcome  cannot identify/separate program impact from other explanations of differences in outcomes

17. Possible Solutions… 17  We need to understand the data generation process  How beneficiaries are selected and how benefits are assigned  Guarantee comparability of treatment and control groups, so ONLY unaccounted for difference is the intervention.