1. Experimental Evaluations Methods of Economic Investigation Lecture 4

2. Why are we doing this?  Experiments are becoming more popular in economics Development Economics, “Field Experiments” Behavioral Economics, “Laboratory Experiments”  Sometimes experiments don’t go as planned Need some econometrics to prove no problems Need some econometrics to fix the problems  Good baseline for understanding attribution of estimated differences Can compare other forms of evaluation to the “experimental ideal”

3. Some Basic Terminology  Start with example where X is binary (though simple to generalize): X=0 is control group X=1 is treatment group  Causal effect sometimes called treatment effect  Randomization implies everyone has same probability of treatment We can change this a bit with weights…

4. Why is Randomization Good?  If X allocated at random then know that X is independent of all pre-treatment variables in whole wide world If you don’t have a random sample, then this will apply to internal comparisons but will make external comparisons more difficult Implies there cannot be a problem of omitted variables, reverse causality etc On average, only reason for difference between treatment and control group is different receipt of treatment

5. Pre-treatment characteristics must be independent of randomized treatment  Define the joint distribution of X and W as f(X,W)  Can decompose this into: f(X,W)=f X │ W (X │ W)f W (W)  Now random assignment means f X │ W (X │ W)=f X (X)  This implies: f(X,W)=f X (X)f W (W)  This implies X and W independent

6. What are we estimating  In words: We want to know if, on average, there is a difference in the outcome of interest between the control group and the treatment groups  In Math: we want an estimate of:

7. Estimating Treatment Effects  Take mean of outcome variable in treatment group  Take mean of outcome variable in control group  Take difference between the two  This is how you learned it in theory and it’s right BUT: Does not generalize to where X is not binary Does not directly compute standard errors

8. Estimating Treatment Effects: A Regression Approach  Run regression: y i =β 0 +β 1 X i +ε i  The OLS estimator of β 1 is an unbiased estimator of the causal effect of X on y:  To see why: Recall that the OLS estimates E(y|X) E(y|X=0)= β 0 so OLS estimate of intercept is consistent estimate of E(y │ X=0) E(y|X=1)= β 0 +β 1 so β 1 is consistent estimate of E(y │ X=1) -E(y │ X=0)  Hence can read off estimate of treatment effect from coefficient on X  Approach easily generalizes to where X is not binary  Also gives estimate of standard error

9. Computing Standard Errors  Unless told otherwise regression package will compute standard errors assuming errors are homoskedastic i.e.  Even if only interested in effect of treatment on mean X may affect other aspects of distribution e.g. variance  This will cause heteroskedasticity This is a second order issue: your coefficient estimates are “right” (consistent) HOWEVER you can’t do any inference because your OLS standard errors are inconsistent

10. ‘Robust’ Standard Errors  Also called: Huber-White standard errors Heteroskedastic-consistent standard errors  Statistics Approach Get variance of estimate of mean of treatment and control group Sum to give estimate of variance of difference in means

11. A Regression-Based Approach  Can estimate this by using sample equivalents  Note that this is same as OLS standard errors if X and ε are independent

12. A Regression-Based Approach  Have to interpret residual variance differentyl – not common to all individuals but the mean across individuals  With one regressor can write robust standard error as:  Simple to use in practice e.g. in STATA:.reg y x, robust

13. Summary So Far  Econometrics very easy if all data comes from randomized controlled experiment and everything went as planned Just need to collect data on treatment/control and outcome variables Compare means of outcomes of treatment and control groups  Everything doesn’t always go as planned Treatment effects are small Randomization fails Non-compliance to treatment

14. How to avoid Experimental problems?  Get info on other regressors Can get consistent estimate of treatment effect without worrying about other variables  But there are reasons to include other regressors: Improved efficiency Check for randomization Improve randomization Control for conditional randomization Heterogeneity in treatment effects

15. The Uses of Other Regressors I: Improved Efficiency  Don’t just want consistent estimate of causal effect – also want low standard error (or high precision or efficiency). Especially important if treatment effects are small  Standard formula for standard error of OLS estimate of β is σ 2 (X’X) -1  σ 2 comes from variance of residual in regression – (1-R 2 )* Var(y)

16. The asymptotic variance of βˆ is lower when W is included  Proof: (Will only do case where X and W are one-dimensional)  When W is included variance of the estimate of the treatment effect will by first diagonal element of:

17. Proof (continued)  Now:  Using trick from end of notes on causal effects we can write this as:

18. Proof (continued)  Inverting leads to  By randomization X and W are independent so:  The only difference is in the error variance – this must be smaller when W is included as R 2 rises

19. The Uses of Other Regressors II: Check for Randomization  Randomization can go wrong Poor implementation of research design Bad luck  If randomization done well then W should be independent of X – this is testable: Test for differences in W in treatment/control groups Probit model for X on W

20. The Uses of Other Regressors III: Improve Randomization  Can also use W at stage of assigning treatment  Can guarantee that in your sample X and W are independent instead of it being just probabilistic

21. The Uses of Other Regressors: Adjust for Conditional Randomization  Conditional randomization is where probability of treatment is different for people with different values of W, but random conditional on W  Why have conditional randomization? May have no choice May want to do it (c.f. stratification)  MUST include W to get consistent estimates of treatment effects

22. Controlling for Conditional Randomization  What we know about our treatment X is by construction correlated with W But, conditional on W, X independent of other factors  But must get functional form of relationship between y and W correct – matching procedures  This is not the case with (unconditional) randomization

23. The Uses of Other Regressors: Heterogeneity in Treatment Effects  So far have assumed causal (treatment) effect the same for everyone but there’s no reason to believe this  May use other variables to test if treatment effect is different between two groups

24. Estimating Heterogeneous Treatment Effecs  Start with case of no other regressors, suppose that there are different beta’s for everyone, so that: y i =β 0 +β 1i X i +ε i  Random assignment implies X independent of β 1i  Sometimes called random coefficients model

25. What treatment effect to estimate?  Would like to estimate causal effect for everyone – this is not possible because we only have one realization of the treatment effect of each person  Instead, we estimate some average this is called the Average Treatment Effect (ATE)  We can estimate ATE with OLS because:

26. Proposition 2.5 OLS estimates ATE  Proof for single regressor:

27. Observable Heterogeneity  Full outcomes notation: Outcome if in control group: y 0i =γ 0 ’W i +u 0i Outcome if in treatment group: y 1i =γ 1 ’W i +u 1i  Treatment effect is (y 1i -y 0i ) and can be written as: (y 1i -y 0i )=(γ 1 - γ 0 )’W i +u 1i -u 0i  Note treatment effect has observable and unobservable component  Can estimate as: Two separate equations One single equation

28. Combining treatment and control groups into single regression  We can write:  Combining outcomes equations leads to:  Regression includes W and interactions of W with X – these are observable part of treatment effect  Note: error likely to be heteroskedastic

29. Units of Measurement  Causal effect measured in units of ‘experiment’ – not very helpful  Often want to convert causal effects to more meaningful units Health Example last week: How much does an extra $ in income get you in better health? How to interpret improvements in X percent or X standard deviations

30. Simple estimator of this would be:  where S is the factor we change in the treatment  Takes the treatment effect on outcome variable and divides by treatment effect  Not hard to compute but how to get standard error? We can do it in a regression Can use your X as an instrument (more on this when we do IV)

31. Uses of Extra Regressors: Partial Compliance  So far: in control group implies no treatment in treatment group implies get treatment  Often things are not as clean as this Treatment is an opportunity—not everyone takes it up Close substitutes available to those in control group Implementation not perfect so some people get into or out of treatment despite RA

32. Some Terminology: ITT  Z denotes whether in control or treatment group – Intention To Treat (ITT)  X denotes whether actually get treatment  With perfect compliance: Pr(X=1 │ Z=1)=1 Pr(X=1 │ Z=0)=0  With imperfect compliance: 1>Pr(X=1 │ Z=1)>Pr(X=1 │ Z=0)> 1>Pr(X=0 │ Z=0)>Pr(X=0 │ Z=1)>0

33. What Do We Want to Estimate?  ‘Intention-to-Treat’: ITT=E(y|Z=1)-E(y|Z=0)  This can be estimated in usual way in the regression  Pro’s Don’t worry about selection in compliance Get an average effect of your intervention  Cons Don’t know what the effect of the actual treatment is—combined effect of treatment and non-compliance

34. More Terminology: TOT  What if we only looked at those who took up the treatment then we would estimate Treatment Effect on Treated (TOT)  Pros Looks directly at treatment  Cons May be biased by the selection of individuals into treatment and control groups

35. Estimating TOT  Can’t use simple regression of y on Z  But should recognize TOT as Wald estimator  Can estimated by regressing y on X using Z as instrument  Relationship between TOT and ITT:

36. Next Steps…  What to do when we can’t do experiments because They didn’t work out like planned We couldn’t do one on this particular issue  Natural Experiments Use variation in the world Several different methods over the next few classes