1. Evaluating Job Training Programs: What have we learned? Haeil Jung and Maureen Pirog School of Public and Environmental Affairs Indiana University Bloomington Nov 7 th 2009

2. What Policy Question to Ask and How to Answer

3. Policy Questions and Parameters of Interest I Average Treatment Effect: Universal Programs  α ATE = E(∆|X)=E(Y 1 -Y 0 |X) Treatment on the Treated: Programs with Self- Selection  α TT = E(∆|X, D=1)=E(Y 1 -Y 0 |X, D=1) Local Average Treatment Effect: Program Expansion by Providing Inducements  α LATE = E(Y 1 -Y 0 |X, D(z)=1, D(z’)=0) Intent To Treat: The Effect of Program Availability  α ITT = E(Y 1 |D=1, R=1) - E(Y 1 |D=1, R=0) in the presence of attrition in the treatment group

4. Policy Questions and Parameters of Interest II ATE for all who are eligible LATE for those who are eligible and are induced to participate because of new incentives to participate TT for those who are eligible and participate ITT for those who are eligible and are accepted to participate

5. Treatment on the Treated (TT) = α TT – It answers how the program changes the outcome of participants compared to what they would have experienced if they had not participated – This parameter is informative about the gross gain accruing to the economy from the existence of a program at a certain level compared to the alternative of shutting it down – It gives policy makers clear advice about whether they should keep the program or not Policy Questions and Parameters of Interest II

6. Decomposition of Conventional Selection Bias Y i = β 0 + α TT D i +Xβ + υ i E(Y 1 |D=1) –E(Y 0 |D=0)=E(Y 1 -Y 0 |D=1) +[E(Y 0 |D=1)-E(Y 0 |D=0)] The conventional selection bias (B) can be decomposed into three sources of bias (Heckman et al., 1997)  B 1 = bias that arises from comparing non-comparable people: ex) discrepancy in years of age in the treatment and comparison group  B 2 = bias that arises from weighting comparable people incomparably: ex) different distribution over the same range of age in the treatment and comparison groups  B 3 = the “true” selection bias (selection on unobservables): ex) participation based on individual returns to the program that is unknown to researchers Estimator= a α T T Conventional Selection Bias = B Heckman et al., 1997

7. Conventional Selection Bias Over Different Comparison Groups No-shows in the treatment group – Bias 3 only The eligible but non-participating group (ENP): same local labor market and questionnaire – Biases 1 and 2 exist – Same amount of Bias 3 as no-shows A comparison group from Survey of Income and Program Participation (SIPP) – Biases 1 and 2 exist – Large Bias 3 than ENP and no-shows

8. If Random Experimentation is Not Possible, What Lessons Have We Learned about Designing a Quasi-Experiment? Use the same questionnaire to obtain individual labor outcomes or demographic information Match individuals in the treatment and comparison groups in the same local labor markets Find a comparison group for which observed characteristics largely overlap that of program participants

9. Sources of “True” Self-Selection Bias in Employment and Training Programs Y it = β 0 + α i D i + X it β + θ i + U it where U it = ρU it-1 + ε it Three sources coming from individuals’ self- selection behavior (selection on unobservables)  α i = α 0 + ν i : individuals select into the program because they know they will earn higher returns from the program (overestimation of true α TT )  θ i : individuals select into the program because their latent or forgone earnings are low at the time of program entrance (underestimation of true α TT )  U it = ρU it-1 + ε it : individuals’ earnings are depending on previous earnings that are low at the time of program entrance (underestimation of true α TT ) Heckman et al. (1999)

10. Estimated Selection Bias by Different Empirical Methods over Simulation Cross-sectional, difference-in-differences, and AR(1) regression estimators seem to work better under all three sources of selection bias – Three sources of bias offset to each other – Estimates with small bias can be misleading Instrumental variable estimator and Heckman Selection correction model seem to work best under θ i and U it = ρU it-1 + ε it – Selection caused by α i = α 0 + ν i heavily against these two models

11. “How do non-experimental estimators work? It depends.” Random assignment still works best Propensity Score Matching: – Require a large sample – Selection on observables only – Sensitive to various matching methods – No clear guidance for superior matching procedures Difference-in-Differences: – Require panel data – Selection on unobservables (time constant heterogeneity only) – Sensitive to choice of time before and after program Regression Discontinuity: – Require a clear cut participation rule – Require a large sample around the threshold – Works like random assignment under the above two conditions – Effects are obtained only for individuals around the threshold for participation Pirog et al. (2009), Battistin and Rettore (2007), and Heckman et al. (1999)

12. It is all about Data, Methods, and Self-Selection Behavior Data availability: Cross-Sectional, Repeated Cross- Sectional, or Panel Data Size of Data: Large sample or small sample Quality of Data: Same Location and Same Survey Instruments for the Treatment and Control Groups Self-Selection on Observables Self-Selection on Unobservables – Heterogeneous response to the program – Time constant individual heterogeneity – Autocorrelation between earnings in different time periods Pirog et al. (2009), Heckman and Vytlacil (2007), and Heckman et al. (1999)

13. Discussion and Conclusion Random assignment still works best to estimate Treatment on the Treated – There is no “silver bullet” for program evaluation How to reduce conventional bias – Use the same questionnaire to obtain individual labor outcomes or demographic information for the treatment and comparison groups – Match individuals in the treatment and comparison groups in the local labor markets Find a comparison group for which observed characteristics largely overlap that of program participants Be aware of interactions among biases from different sources

14. Thank You