1. Copyright © 2015 Inter-American Development Bank. This work is licensed under a Creative Commons IGO 3.0 Attribution-Non Commercial-No Derivatives (CC-IGO BY-NC-ND 3.0 IGO) license (http://creativecommons.org/licenses/by-nc-nd/3.0/igo/legalcode) and may be reproduced with attribution to the IDB and for any non-commercial purpose. No derivative work is allowed.http://creativecommons.org/licenses/by-nc-nd/3.0/igo/legalcode Any dispute related to the use of the works of the IDB that cannot be settled amicably shall be submitted to arbitration pursuant to the UNCITRAL rules. The use of the IDB’s name for any purpose other than for attribution, and the use of IDB’s logo shall be subject to a separate written license agreement between the IDB and the user and is not authorized as part of this CC-IGO license. Note that link provided above includes additional terms and conditions of the license. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the Inter-American Development Bank, its Board of Directors, or the countries they represent.

2. Causal Inference Francisco A. Gallego PUC-Chile and J-PAL LAC

3. Map Motivation Impact Evaluations and Counterfactuals Causal Analysis Causal Analysis and Treatment Effects 3

4. Map Motivation 4

5. Empirical questions are more difficult to answer in the social sciences. In public policy, these involve cause-effect relationships like: –Does school decentralization result in improved education quality? –Does a year of training result in higher incomes? And, more importantly maybe, what type of training results in higher increases in income? –Do conditional cash transfers result in improved health and education among children? Or, do they work because of the conditions or because of the additional cash? 5

6. Motivation Answering these questions is important because: –They help answer policy concerns Do the programs reduce poverty? Can they reduce poverty more rapidly with the same resources? –Problems faced by decision-makers –Theoretical considerations in the social sciences 6

7. Map Motivation Impact Evaluation and Counterfactuals 7

8. 8 How can we answer these questions?: Impact Evaluation A program’s impact is the difference between: 1.Results that program participants obtain after some time in the program and 2.The results that those same participants would have obtained at the same time had they not participated in the program

9. 9 We take the difference between What happened (with the program) and -What would have happened (without the program) =Program’s IMPACT This last scenario is called the counterfactual Impact Evaluation

10. 10 Impact: What is it? Time Main Result Impact Counterfactual Intervention

11. Time Intervention 11 How to Evaluate Impact? Counterfactual Impact Main Result

12. Time Intervention 12 How to Evaluate Impact? Impact Counterfactual Main Result

13. 13 The counterfactual represents the state of the world that program participants would have experienced in the absence of the program Problem: The counterfactual can’t be observed Solution: We have to “replicate” or “build” the counterfactual Counterfactual

14. Map Motivation Impact Evaluation and Counterfactuals Causal Analysis 14

15. Standard Statistical Analysis Tools: probability and other estimation techniques Objective: infer parameters from a distribution based on samples from that distribution Usage: With the help of parameters we can: –Infer the association between variables, –Estimate the probability of past or future events occurring –Update program Condition: experimental conditions cannot be changed 15

16. Causal Analysis Goes one step ahead of standard statistical analysis Objective: infer aspects of data generation process Usage: With the help of such aspects, we can –Deduce the probability of occurrence if the context does not change (static analysis) –Predict how variables change if the context changes (dynamic analysis) 16

17. Causal Analysis The idea of the dynamics of events when conditions change includes: –Predict the effect of interventions –Predict the effect of spontaneous changes –Identify the causes of events Distinction between correlation and causation –New language –Causation is key for policy-making decisions If we do XXX, will we achieve YYY? It’s not enough with knowing that XXX is associated with YYY… 17

18. Neyman-Rubin Causal Model (1923, 1974) Potential results models: Let U represent the population –Let u represent each unit in U. For each u U: –Y(u): response variable –A: attributes of units in U. Idea: expose each unit to the action of a cause Rubin assumes that causes are actions that, hypothetically, could be treatment 18

19. Causal Model and Treatment Assume, for simplicity, that there are only 2 causes or treatment levels. Let D be a variable that indicates the cause to which unit in U is exposed: In a controlled study, D is constructed by the researchers In an uncontrolled study, D is determined by factors beyond the researcher’s control 19 If the unit u is exposed to the treatment If the unit u is exposed to the control

20. Map Motivation Impact Evalution and Counterfactuals Causal Analysis Causal Analysis and Treatment Effects 20

21. Causal Model and Treatments Y is potentially affected by t or c. That is, there can be 2 response variables for the same u:. –Y 1 (u): value of the response if the unit u was exposed to the treatment. –Y 0 (u) : value of the response if the unit u was exposed to the control. Therefore, the result of each individual can be described as (if there are no externalities): 21

22. 22 We take the difference between What happened (with the program) and -What would have happened (without the program) =IMPACT of the program This last scenario is called the counterfactual How does this translate to the language of the Neyman-Rubin Causal Model? Impact Evaluation and Causal Models

23. Causal Model and Impact Evaluation For each unit u, the treatment causes: Fundamental problem of causal inference: –For the same u, the same Y 1 (u) and Y 0 (u) cannot be observed That is, we don’t have a counterfactual for each u –An individual cannot be simultaneously receiving and not receiving the treatment… What do we do? 23

24. Causal Model and Impact Evaluation What about average treatment effects? Average Treatment Effects (ATE) on U (or a sub- population) are: That is, the statistical solution replaces the causal effect at each unit for the average causal effect in a population of U units 24

25. Causal Model and Impact Evaluation Obviously, the expectations of Y 1 (u) and Y 0 (u) cannot be calculated but they can be estimated…. Econometric impact evaluation methods try to construct (under different assumptions) estimations with consistencies of: The “goodness” of these estimators is defined precisely by the plausibility of the assumptions used 25

26. Causal Model and Impact Evaluation So let’s consider estimating: Using the following estimator: (1) it’s defined for the population (2) it is estimated using a sample of the population 26

27. Causal Model and Impact Evaluation Let  be the proportion of the population assigned to the treatment group Therefore, ATE can be discomposed as: 27

28. Causal Model and Impact Evaluation Assume that: Therefore: What we can estimate consistently: 28

29. Causal Model and Impact Evaluation Do these make sense? That is: –The average effect: Under treatment, does not differ between treatment and control groups Under control, does not differ between treatment and control groups To satisfy these conditions, it is enough that the assignment of treatment D is not correlated with potential results of Y 1 (u) and Y 0 (u) The main way to achieve this non-correlation is through the random assignment of the treatment. –We will talk about this in more detail in the next class. 29

30. Causal Model and Impact Evaluation In many cases, there is simply not enough information available about how the units in the control group would have behaved if they had received the treatment –…and vice versa: how those in the treatment group would have behaved if they had not received the treatment This is key to understand estimation biases (2). In fact, with a little algebra, it can be shown that: 30

31. Causal Model and Impact Evaluation Two big sources of selection bias: 1.Selection bias 2.Heterogeneity in treatment effects Most of the available methods focus on 1, assuming that treatment effects are homogeneous among the population (or by redefining parameters of interest in the population) 31

32. Other Parameters of Interest ATE are not always the parameter of interest for public policy matters Sometimes, for example, we are more interested the average treatment effect for the person exposed to the intervention, that is: Note that the difference between ATE and TOT only makes sense if there is heterogeneity in treatment effects in the population U (if not, all units are identical…) 32

33. Other Parameters of Interest When does our estimator (2) estimate TOT consistently? As it is evident, this estimator works well if: 33

34. Other Parameters of Interest Sometimes in public policy, we are interested in knowing the average effect of offering the program (ITT) –What happened to the average child that was in a treated school in this population? –What happened to the average person that was offered the treatment? Suppose we are using intention to treat… Is this the right number to be looking for? Is this the effect of the treatment? The logic here is exactly the same as for the previous estimators, but in this case D=1 reflects being offered the treatment and not whether people were effectively treated or not. 34