1. Counterfactual impact evaluation of cohesion policy Examples of innovation support from two regions D. Czarnitzki A, Cindy Lopes Bento A,C, Thorsten Doherr B A) K.U.Leuven B) ZEW, Mannheim C) CEPS/INSTEAD, Luxembourg Warsaw, 12 December 2011

2. Introduction Task in this research project: Explore to what extent publicly available beneficiary data of European Cohesion Policy can be used for a quantitative study of policy impacts at the firm-level Focus: innovation activities of firms Requirements: Linking beneficiary data to firm-level information Amadeus Database Patent database Other resources containing data about innovation activities at the firm level Obtain control group of non-funded firms

3. Challenge Beneficiary lists typically only include –Recipient name –Project title –Year of funding approval –Approved amount of funding Recipient names have to be searched in other databases using text field algorithms Potential hits have to be manually checked –For each study two text field searches necessary: –Recipients have to be identified in Amadeus database –Identified Recipients and control group (obtained from Amadeus) have to be searched in patent database or other related data source containing information on innovation activities

4. Example of recipient data

5. Countries examined Of 11 countries/regions investigated: Poland, Slovakia, Slovenia, Flanders, Wales and London were eliminated because of small numbers of projects in these regions/countries. Spain was eliminated as in the data we had available firm names were not included. FR data were tested, but was impossible to tell treatment date Only CZ and DE retained

6. Country case I: Czech Republic Recipient data: –26,075 projects –Time period: 2006 – 2011 –Total amount of 10,747,210,000 –Avg. amount per project 412,265 How many (different) firms? –unknown!!! Name list is searched in Amadeus (firm database) –contains 14,609 different firms

7. Country case I: Czech Republic Data: Total # of firms contained in Amadeus: 14,609 Total # of firms retained : 12,887 Methodology: Difference-in-difference Compare annual patent applications per firm in pre- treatment (1997-2003) and treatment phases (2008/9) 1,433 treated firms 11,454 control firms

8. Country case I: Czech Republic Patenting fell by 63% in controls, only 14% in treated Highly statistically significant (chi² =12.07, p<0.01) Understates impact, since some firms still in their pre- treatment phase (data lag) Patenting here is really a proxy for a wider range of innovative activities Next example has data for wide range of innovation activities…

9. Country case II: Germany Recipient data: –47,616 projects (out of those 33,201 in Eastern Germany) –Time period: 2006 – 2011 –Total amount of 9,060,653,000 –Avg. amount per project in Eastern (Western) Germany: 92,400 ( 415,923). Name list is searched in Mannheim Innovation Panel Outcome variables: –R&D investment (R&D intensity = R&D / Sales) –R&D employment divided by total employment –Total innovation investment / Sales –Investment into physical assets (relative to capital stock) –Innovation types

10. Link to the Mannheim Innovation Panel MIP = German part of the the CIS Annual survey; asks about 5,000 firms about the innovation activities We can link 5,606 different grants to the MIP. These correspond to 1,904 different firms. –Restrict time period to 2007-2010: We lose firms as they are in the MIP but not in the relevant years. –After removing observations with missing values in variables of interest, we can use a final sample of 623 supported firms. –Control group: 21,226 observations Estimator: Nearest Neighbor matching

11. Germany

12. Non-subsidized firmsSubsidized firms p-value of t-test on mean difference N=21,226N= 623 VariableMeanStd. Dev.MeanStd. Dev. Covariates LOGEMP3.6331.6643.9931.239p<0.001 LABPROD0.2452.3470.1460.215p<0.001 EAST0.3130.4640.8680.338p<0.001 GP0.2840.4510.2740.447p=0.591 FOREIGN0.0640.2450.0610.24p=0.748 EXPO0.5490.4980.7780.416p<0.001 Outcome variables RDINT1.5275.7855.4312.278p<0.001 RDEMP3.60810.58411.95719.154p<0.001 INNOINT3.3589.1199.21615.917p<0.001 INVINT37.7293.86351.554124.60p=0.047 PD0.3510.4770.5840.493p<0.001 PC0.2940.4560.4350.496p<0.001 PA0.1390.3460.1080.31p=0.013 PN0.3970.4890.6320.483p<0.001 [1] [1] Due to missing values, the number of observations is of 16,748 for the non-subsidized firms and of 488 for the subsidized firms for R&D employment. [2] [2] Due to missing values, the number of observations is of 9,291 for the non-subsidized firms and of 330 for the subsidized firms for innovation intensity

13. Germany Selected control group, N=623 Subsidized firms, N=623 p-value on the t-test on mean difference Variables MeanStd.dev.MeanStd.dev. Covariates LOGEMP 3.9711.5613.991.239p=0.798 EAST.870.336.868.338p=0.937 FOREIGN.072.259.061.239p=0.457 EXPO.775.418.778.415p=0.898 GP.274.447.274.447p=1.000 LABPROD.148.228.146.215p=0.853 Outcome variables RDINT 3.4427.8825.43012.278p<0.001 INNOINT 6.90313.5859.21515.917p=0.009 RDEMP 9.17716.27511.95719.154p=0.028 PD.495.499.584.493p=0.003 PC.373.493.435.496p=0.039 PA.167.373.107.310p=0.004 PN.526.501.632.482p<0.001 INVINT 37.45108.03047.091109.286p=0.410

14. Germany Robustness tests: Restrict sample to innovating companies –as purpose of project is not supported systematically: innovation vs. something else –Main results reported earlier hold BUT: Once we control for subsidies received from German Federal Government all positive effects reduce somewhat in terms of magnitude and also statistical significance reduces slightly. Cohesion Fund reciepients are also more likely to receive other subsidies!!!

15. Germany Robustness test: does the size of the grant matter?

16. Lessons learned Reporting standards should be improved. Otherwise a quantitative evaluation lacks credibility or produces no results because of noisy data. What should be reported at the minimum? –Funding start and end date in addition to amount –Type of recipient (firm vs. other) –Purpose of grant –Recipient name AND location –All in database compatible formats And…. –If possible, historical data should be stored centrally, e.g. by EC. –Longer time lag between evaluation and program completion should be applied.

17. Q&A: Discussion Contact: Prof. Dr. Dirk Czarnitzki K.U.Leuven Dept. of Managerial Economics, Strategy and Innovation Phone: +32 16 326 906 Fax: +32 16 326 732 E-Mail: dirk.czarnitzki@econ.kuleuven.bedirk.czarnitzki@econ.kuleuven.be