The Use of Counterfactual Impact Evaluation Methods in Cohesion Policy Joint meeting of ESF Evaluation Partnership and DG REGIO Evaluation Network in Gdańsk (Poland) on 8 July 2011 The Use of Counterfactual Impact Evaluation Methods in Cohesion Policy Bernhard Boockmann Institute for Applied Economic Research (IAW), Tübingen (Germany)

Structure of the Presentation Motivation and key questions Which Cohesion policies provide suitable treatments for counterfactual evaluation? Isolating the impact of Cohesion policy from the effect of other interventions Strategies for reliable and robust identification of treatment effects Meeting data requirements Conclusions

Motivation and Key Questions Counterfactual evaluation is an established methodology for deriving policy impacts and provides a basis for cost- benefit analysis. The key questions for the paper are the following: Which particular difficulties arise in the application of counterfactual evaluation in the context of Cohesion policy? What limits the applicability of counterfactual evaluations in Cohesion policy? How can these difficulties and limitations be overcome?

Structure of the Discussion Micro evaluation, Rubin model Experimental approach Non-experimental approach Treatment and outcome definition Identification strategy Data

Which Cohesion Policies Provide Suitable Treatments for Counterfactual Evaluation?

Conditions for Suitable Treatments The treatment must be clearly defined and separable from other treatments; it must be homogenous, so that all treated individuals in the population receive the same treatment. The treatment must be unambiguously directed to a group of individuals, while there are other individuals who are not treated (i.e., treatment is not universal). Feedback, spillover and other indirect effects from the treated to the non-treated must be excluded. The effect of the treatment must be measurable and non- negligible in magnitude.

Example: OP for Saxony (Germany), 2007-2013 The OP centres on four priority axes. There is a large range of interventions : Measures for unemployed individuals (such as classroom and on-the-job training programmes). Measures directed at particular target groups (programmes for the illiterate, persons starting their own companies, young people). Measures at the level of projects and activities (such as support for mentoring and programmes for the strengthening of local actors).

Example: OP for Saxony (Germany), 2007-2013 Some interventions do not provide a suitable treatment Condition violated Intervention 1 Qualification measures specific to particular occupations (e.g. health care) 2 Financial support given to initiatives and networks 3 Displacement effects relevant for programmes directed at the unemployed with particular disadvantages 4 Programmes for the development of soft skills and intercultural competencies

Example: OP for Saxony (Germany), 2007-2013 Interventions are not suited for counterfactuals impact assessment and must be examined with other methods, such as simulation, macro-evaluation or qualitative approaches. There remain important interventions from the operational programme that could quite successfully be evaluated through counterfactual methods, such as general qualification measures for the unemployed, hiring subsidies or support to start-ups by the unemployed.

Isolating the Impact of Cohesion Policy from the Effects of Other Interventions ESF interventions are closely connected with the policies of the member states, filling gaps or complementing existing programmes. Can the additional impact of the ESF be ascertained using counterfactual evaluation problems? Answer: yes, by multiple treatment approaches. However, applying them is costly, as several measures must be analysed jointly. Costs may be reduced by conducting studies jointly for national and Cohesion policy interventions.

Strategies for Reliable and Robust Identification of Treatment Effects All non-experimental methods in counterfactual require identifying assumptions. It is not sufficient simply to have “a control group”. E.g., matching requires that outcome and treatment must not be correlated once the covariates have been taken into account.

Example: Identifying assumption for the matching estimator Language-training programme for non-native parts of the population (e.g. Dmitrijeva 2008). If participation depends on language skills before the start of the programme (e.g., the programme is directed towards those with the lowest skills), and initial language skills are unobserved, results may be biased. This is because participants may consist of those most in need of training (who possess systematically lower initial language skills than non-participants). Hence, initial language skills should be in the data.

Meeting Data Requirements Advantages of administrative data over survey data: Large number of observations. No unit non-response or panel attrition. Longitudinal dimension. Exactness, especially regarding past events and histories. Less costly in comparison with survey data.

Member States Making Administrative Data Available for Evaluation Research

Meeting Data Requirements Problems with administrative data: Survey data is easier to handle. There are confidentiality problems and lack of information for certain groups. There is too little individual information available (see language-training example above). Wunsch and Lechner (2011): Inclusion of information on individual health and on the last employer is particularly important.

Meeting Data Requirements A possible (but costly) way out: Merge survey data with administrative data. Example for an ESF programme evaluation: Deeke and Kruppe 2006. Multiple-purpose linked administrative and survey data such as the IZA evaluation dataset (Caliendo et al. 2010) reduce some of the costs of conducting reliable counterfactual evaluations.

Conclusions (1) A choice must be made regarding which interventions to evaluate using counterfactual impact evaluation. Alternative approaches must be used when causal impact evaluation is not feasible. The methodology is apt in principle to identify the added value of Cohesion policy. However, this requires taking into account national or regional programmes. These should be evaluated in conjunction with Cohesion policy interventions.

Conclusions (2) Provision of administrative data is of key importance. Survey data is insufficient for state-of-the-art evaluations. ESF funding should be used to create an appropriate data base if no data base is available. Until administrative data with sufficient quality of information are available, surveys provide a useful (but costly) alternative to complement administrative data.

Conclusions (3) It is important to establish quality standards for causal impact evaluation All data should be made available for replication purposes. Include cost-benefit analysis based on estimated impacts. Ensure that the results are discussed in scientific debate.