1. Non-Experimental Evaluations Methods of Economic Investigation Lecture 5 1

2. Today’s Lecture  Review the Evaluation Problem  Review the Experimental Approach  Issues of Compliance  Natural Experiments 2

3. Review: The Evaluation Problem  In words: We want to evaluated the effect of a treatment on an outcome Worried that some omitted/unobservable factor affects both who is “treated” and the outcome. So let’s do this very simply: there’s some treatment T it occurs at some time k 3

4. Homogeneous Treatment Effects  Before the treatment (so at time t<k)  After the treatment (so at time t≥ k) Relationship between characteristics X and the Outcome Y Treatment Effect 4

5. The Fundamental Identification Problem  Worry that assignment to treatment is not random Assignment process leads to non-zero correlation between treatment (T) and the error (U) Probably because participation decision is based on characteristics that affect outcome (Y) too  If you can’t observe EVERYTHING that affects both treatment probability and outcome then your OLS will not be a valid approach 5

6. Experimental Approach  Randomization implies that T is independent of X Don’t have to worry about what factors generated selection into treatment because it is by design May still include some Xs to increase efficiency Can use these X’s to show that X and d are uncorrelated 6

7. Heterogeneous Treatment Effects  For simplicity for the moment ignore the other regressors (X)  Define the population mean of the treatment effect  Last time, we showed that in this situation when the heterogeneity does not affect probability of treatment then OLS estimates ATE consistently ( ) 7 Treatment Effect for individual i

8. Defining TOT  Let’s parameterize this a bit more so that an individual’s treatment effect is the mean treatment effect and some individual specific component  Then, we can define the mean impact of the treatment on the treated (TOT) as 8 Mean deviation of the treatment effect among the treatment group population

9. OLS with Heterogeneous Treatment Effects  The OLS estimate will be:  Our treatment effect is now:  Errors are heteroskedastic (that’s ok) 9

10. Heterogeneous TE and ATE  If E(ε i d i )≠0 then the OLS estimator identifies:  Worry that individuals factors that are unobservable might be correlated with treatment 10 TOT =0 because of Random Assignment

11. Without Randomization we’re back to the basic problem in evaluation  We want to evaluate: E(Y| T=1) – E(Y | T=0)  We have some treatment effect: Y i = a + b i *T i + e i  Then we know that E(Y| T=0) = a + E(e i | T=0) E(Y | T= 1) = a + b + E(e i | T=1)  Our Estimate will then be: E(b i ) + [E(e i | T=1) – E(e i | T=0)] Selection EffectTreatment Effect 11

12. In general there are 4 kinds of Non- Experimental Evaluations 1. Controls, controls, controls… 2. “Natural Experiments” 3. Matching 4. Selection Models 5. Structural Models 12 “Reduced Form” Replicate experiment with an ‘engineered’ control group “Parametric/ Structural” Model the form of the selection effect or bias and apply a statistical correction

13. Controls method  Basic goal—conditional independence so that it’s like our conditional/stratified randomization  Focus of next week’s class: If we don’t omit anything, then we’re set Trade-off between omitted variables and “over- controlling” Even if we can’t measure everything, maybe be able to control with “fixed effects” and “interactions” (both are error component models) 13

14. Pros and Cons  Pros Transparent to others Can be convincing with robust controls and well-defined outcomes  Cons VERY data intensive Can be hard to measure everything Not very convincing in complicated settings 14

15. “Natural Experiment”  Use an explicit source of variation  Don’t model (or at least don’t use a model) of underlying behavior based on behavior/preferences  ‘Natural’ or ‘Quasi’ Experiments Used to refer to situation that is not experimental but is ‘as if’ it was Not a precise definition – saying your data is a ‘natural experiment’ makes it sound better Refers to case where variation in X is ‘good variation’ (directly or indirectly via instrument) 15

16. A Famous Example: London, 1854 16

17. The Case of the Broad Street Pump  Regular cholera epidemics in 19 th century London  Two Theories on Why: Widely believed to be caused by ‘bad air’ John Snow thought ‘bad water’ was cause  An ideal experimental design: give some people good water and some bad water Ethical Problems with this 17

18. Soho Outbreak: August/September 1854  Observation 1: People closest to Broad Street Pump most likely to die  Does this distinguish between the two theories? NO—breathe same air so does not resolve air vs. water hypothesis 18

19. Distinguishing between theories  Observation 2: Some places near the well had few deaths Nearby workhouse few deaths (own well Nearby brewery had own no deaths (workers all drank beer & own well)  Observation 3: Some people far away died Woman died in Hampstead Had the habit of having water from pump deliver by her niece in Islington 19

20. Why is this a Natural experiment?  Good variation: water supply ‘as if’ it had been randomly assigned (existed before outbreak) other factors (‘air’) held constant  Can estimate treatment effect difference in means run regression of death on water source distance to pump, other factors 20

21. What’s that got to do with it?  Strongly suggests water the cause Certainly relative to the air counter-theory Investigation of well found contamination by sewer  This is non-experimental data but analysed in a way that makes a very powerful case 21

22. Elements of Non-experimental Evaluation  Good source of variation (we’re going to call this “EXOGENOUS VARIATION”)  Good measure of treatment and control group outcomes  Good measure of treatment and control group control variables 22

23. Methods for Analysing Data from Natural Experiments  If data is ‘as if’ it were experimental then can use all techniques described for experimental data and then some: OLS  Simple conditional means difference as in experimental data Difference-in-differences (really just like OLS but with a pre-treatment regression too) Instrumental Variables 23

24. ‘Natural Experiments’ Pros and Cons  Advantages: source of variation is clear “model free”  Disadvantages: Identifying assumption may be questionable or hard to believe (internal validity) may tell you about this experiment but “local” in that it does not tell you about preference parameters (External validity) using it to make simulations as to other policy changes may be bad 24

25. Matching Methods  Non-parametric (no functional form assumption) approach  Take controls and use combined variation to reconstruct experimental conditions The goal: on everything one can observe for a treatment group individual find a ‘nearest neighbor’ 25

26. Matching Pros and Cons  Pros Very flexible in determining “propensity” for treatment Can use all the variation across all variables (relative to simple control methods)  Cons Need a lot of common support for you control and treatment groups (i.e. need good matches) Heavy data requirements Heterogeneous treatments hard to measure 26

27. Selection Correction  Parametrically model the way selection occurs Famous Example: Female Labor Force Participation  A special case of “structural estimation” but can be used in combination with any methods when you’re worried about a specific form of selection bias 27

28. A quick word on Structural Estimation  In some cases, we can use theory to model the endogeneity. Often this is based on utility or objective function (e.g. profit) maximization theory  In the original structural literature, there was little attention to identification and the results may be identified by nonlinearities and parametrization. This is no longer the norm, with more attention being placed on identification.  Can also use this more loosely to constrain your estimation to realm of the likely or feasible 28

29. Structural Estimation Pros and Cons  Advantages: once you recover the parameters of the utility function (or other pref), you can use those parameters to simulate what will happen if policy changes.  Disadvantages: have to implement possibly untestable assumptions about economic and statistical model often generate wide range of estimates Can be very sensitive to specification and assumptions—not great external validity 29

30. Conclusion  Natural experiments useful source of knowledge  Often requires use of IV  Instrument exogeneity and relevance need justification  Weak instruments potentially serious  Good practice to present first-stage regression  Finding more robust alternative to IV an active research area 30