The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ The Fundamental Evaluation Problem and its Solution

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Evaluation Problem 2.Treatment Effects 3.Selection Bias 4.Solution Approaches Hagen, Tobias und Bernd Fitzenberger (2004), Mikroökonometrische Methoden zur Ex-post-Evaluation, in: Hagen, Tobias und Alexander Spermann, Hartz-Gesetze – Methodische Ansätze zu einer Evaluierung, ZEW-Wirtschaftsanalysen, 74, S.45-72

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Example: Evaluation of programs of active labour market policy (e.g. job creation measures)  Investigation of a program‘s effect on a certain outcome variable (e.g. the employment probability)

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Measurement of the program‘s success by the share of individuals entering employment during a certain period of time is not sufficient. Problem: The causal effect of the program is not measured – employment take-up could happen without program participation

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Causal effect: Employment probability of participants versus employment probability of participants had they not participated Problem: „counterfactual situation“ – participants cannot simultaneously be non- participants!

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Solution: Estimation of the hypothetical employment probability participants in case of non- participation by using the employment probability for non-participants. Use of a „comparison or control group“, to be able to estimate the success of participation.

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ In case participants and control group differ with respect to observable or non- observable characteristics that do have an influence on outcome variables Selection Bias

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Question: What is the effect of a program on the outcome variable y? y 1 : Outcome variable in case of participation y 0 : Outcome variable in case of non-participation C : Dummy variable, set to 1 in case of participation

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ The actual observed outcome variable for an individual i results from: The program effect is:

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Problem: It is not possible to calculate an individual causal effect. No individual can be in two different states of participation at the same point in time.

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ However, it is possible to estimate The mean effect of participation on the group of participants („Average Effect of Treatment on the Treated“ – ATT). The mean effect of participation expected for an individual drawn randomly from a population of participants and non-participants („Average Treatment Effect“ – ATE).

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ E[y 1 ] only is observable for participants and E[y 0 ] only is observable for non- participants.

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Case differentiation: 1.Participants and non-participants („control group“) differ neither with respect to observed nor to unobserved characteristics  consistent estimates of expected value of the outcome variable using the sample mean:

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ WhereasT: Number of participants NT: Number of non- participants

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Participants and non-participants differ in regard to observed and unobserved characteristics  Selection Bias  Difference of sample means does not lead to consistent estimators

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ X-Heterogeneity: Heterogeneity of the treatment effect that can be explained by differences in observed variables. U-Heterogeneity: Heterogeneity of the treatment effect that can be explained by differences in unobserved variables.

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Definition homogeneous treatment effect: Treatment has the same effect on individuals with different observed attributes, i.e. no X-heterogeneity. Measure has the same effect on individuals with different unobserved attributes, i.e. no U-heterogeneity. Treatment effect is identical for all individuals and ATT=ATE.

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ X-Heterogeneity / U-Heterogeneity:  Heterogeneous treatment effect  Selection bias 1.Selection bias due to observed variables 2.Selection bias due to unobserved variables

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ „selection on observables“ „selection on unobservables“ Regression Methods „Propensity- Score- Matching“ Difference-in- Difference- Estimators (DiD) Selection Models Instrumental Variable Approaches (IV)

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Social experiments 2.Natural experiments 3. Quasi experiments 4. Non experimental statistical / econometric methods

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Social Experiments Example: Introduction of a care budget Basic Information (Arntz/Spermann, ZEW DP 04-84): 7 sites in East and West Germany goal: 2000 participants; assigned randomly to 1000 in the program & 1000 in the control group duration:

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Matching transfer plus case management Treatment: Outcome: Duration in home care Life satisfaction Quality of care Home Care arrangements

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Discussion of biases: Randomization bias Treatment dropout bias Control group substitution bias Attrition bias General equilibrium effects Preliminary results: Swiss Journal 2006

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ No randomization bias No structural change of participants and non- participants due to the fact that they participate in a social experiment

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ No Treatment Group Dropout Bias R=1  T=1 R=0 indicator variable for control group R=1 indicator variable for program group T=0 indicator variable for non-receipt of program T=1 indicator variable for program receipt Persons who were assigned to the program group have to receive the program

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ III) No Control Group Substitution Bias R=0  T=0 Persons of the control group do not participate in comparable programs

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ No General Equlibrium Effects No indirect effect of the program that could change the direct effect of the program

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ No Attrition Bias Program or control group members may not get lost during the experiment

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Natural Experiments See ZEW-DP (Boockmann/Zwick/Ammermüller/Maier) on „Eingliederungszuschüssen an ältere Arbeitnehmer“ (Hartz I-III evaluation) See later presentation on Difference-in- Difference estimators

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Quasi Experiment  Spermann/Strotmann (ZEW DP 05-68) about the „ Targeted Negative Income Tax (TNIT) experiment“ in Mannheim Main features: 1.Target group: Means-tested unemployed 2.Employee subsidy (earnings supplement) 3.Time-restriction which varies between household types

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Example for Quasi Experiments: Evaluation Design for TNIT Study Site-randomized control group in the same local labour market Program district in the northern part of Mannheim Comparison district in a comparable southern part on Mannheim Program and comparison group are comparable due to t-tests on important observables

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Example for Quasi Experiments: Treatment, Outcome, & Identification Information about the potential earnings supplement in case of participation in the private labour market Check by survey: Did the program group understand the program (proxy for receipt of treatment) Result: Program group understood the basic idea Treatment:

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Example for Quasi Experiments: Treatment, Outcome, & Identification Participation: available Income: available Hours of work: not available Duration of jobs after time limit: not available Outcome:

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Identification: Is the selection on observable assumption plausible? HIT (1997) and HIST (1998) set up criteria for comparison group data quality: 1.Same data source for program and comparison group: fulfilled 2.Program and comparison group reside in the same local labour market: fulfilled 3.Data contain a rich set of covariates: only partly fulfilled, do not observe individual employment history and pre-program data Quality checks for matching are not feasible Solution: We restrict to Probit and Tobit, Propensity Score Matching confirmed results

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ If one accepts the plausibility of the selection on-observables assumption,  Then average marginal effect Probit estimation of the program dummy could be interpreted as ATE

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Results: Descriptive Statistics

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Probit Models for overall employment (average marginal effects)

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Notes: Administrative data, Mannheim 2000, p-values in parentheses. ***/**/* indicate statistical significance at the 1, 5 and 10 percent level, respectively.

The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/ Non-experimental Methods  „ Matching“  Instrumental Variables  Panel Data  etc.