The Dutch Innovation voucher Hungary May 18, 2011 Two Examples of Dutch Policy Evaluation: randomization and regression discontinuity Marc Van der Steeg CPB Netherlands Bureau for Economic Policy Analysis The Hague, The Netherlands

The Dutch Innovation voucher Hungary May 18, 2011 The evaluation problem A two-way relation ► causal: policy leads to more innovation ► correlation: innovative firms make more use of innovation policy instruments How to disentangle these two relations? ► add covariates to the regression equation ► do highbrow econometrics (e.g. matching) ► or... Controlled experiment ► experimental group and control group ► random allocation ► difference is causal impact

The Dutch Innovation voucher Hungary May 18, 2011 The Dutch innovation voucher pilot 2004 Goal: “Lead them to water and pay them to drink” (Angrist et al., 2006) ► Introduce SMEs to public research institutes ► Market-oriented incentives for research institutes

The Dutch Innovation voucher Hungary May 18, 2011 Characteristics voucher program credit note, worth max EUR for SME’s only no own contribution required application-oriented research questions placed with a defined group of institutes no restrictions on level of question or technology valid for half a year 100 vouchers available LOTTERY if demand exceeds supply

The Dutch Innovation voucher Hungary May 18, 2011 Example of voucher project “Biodiesel from Africa” ► Seeds of tropical plant Jatropha can be used for production of biodiesel ► However, current oilpresses less efficient than for production of biodiesel from coleseed: 40 % lost ► Diligent Energy Systems used an innovation voucher and asked Technical University of Eindhoven to improve efficiency of process –PHD-student carried out experiments –Plans to do further research on adaptation of coleseed press for Jatrophra

The Dutch Innovation voucher Hungary May 18, 2011 Research questions To what extent did the voucher affect the degree of: 1.Science-industry interaction ► Direct effect ► Persistence effect 2.Innovation ► Product innovations ► Process innovations

The Dutch Innovation voucher Hungary May 18, 2011 Application process 1044 applications on September 17th, 2004 Lottery: ► 100 winners ► 944 ‘losers’

The Dutch Innovation voucher Hungary May 18, 2011 Data (1) First round of telephone interviews May, 2005 ► 100 winners ► 500 randomly selected losers ► questions about actual and counterfactual behaviour Response rate ► 71 winners (71%) ► 242 losers (48%) Second round of telephone interviews in September 2006, with questions on: ► interaction after voucher period ► realized innovations after voucher period

The Dutch Innovation voucher Hungary May 18, 2011 Data (2) No significant differences between winners and losers in background characteristics: ► firm size ► region ► sector ► previous interaction with public research institutes We can confidently attribute any differences in outcomes to the voucher policy instrument ► however, we still add all available covariates to regression equation ► this raises precision of effect estimates

The Dutch Innovation voucher Hungary May 18, 2011 Analysis (1): Direct effect Data on behaviour during voucher period: ► 62 out of 71 (= 87%) winners commissioned a project ► 20 out of 242 (= 8%) losers commissioned a project Effect estimates ► 13% of the vouchers not used (= (71-62)/71) ► 8% crowding out (= 20/242) ► 79% additional assignments (= 62/ /242) Main conclusion: 8 out of every 10 vouchers additional ► Voucher generates substantial more science-industry interaction

The Dutch Innovation voucher Hungary May 18, 2011 Stated preferences versus revealed preferences Survey questions: ► Losers: what would you have done if you would have won a voucher ► Winners: what would you have done if you would not have received a voucher ► 76% winners say: without voucher, fewer projects ► 86% losers say: with voucher, more projects Conclusion: in this case, stated counterfactual behaviour gives good proxy for real causal effect ► But need not always be the case!!!

The Dutch Innovation voucher Hungary May 18, 2011 Other findings No indications for effect on size of project ► Most SME’s commissioned project of more or less voucher value Indication for small timing effect (of 1 out of 10) ► Voucher winners claimed that voucher did not affect number of projects, but only led them to carry out a project earlier in time. Same analysis on two 2005 voucher pilots shows robustness of results ► Van der Steeg et al. (2007) –2005-1: 7 out of 10 –2005-2: 5 out of 10

The Dutch Innovation voucher Hungary May 18, 2011 Analysis (2): Persistence and innovation output effect 1.Persistence: no effect ► Voucher winners do not carry out more assignments than voucher losers in 1½ year after voucher period. ► Reasons for not carrying out new assignment: too expensive, no new questions, own research 2.Innovation output: mixed evidence ► positive effect on process improvements ► no significant effects on realization of new or better products and new processes 3.Indications for crowding out of own R&D

The Dutch Innovation voucher Hungary May 18, 2011 Note on importance of method Randomization was crucial Applicants were not at all representative for Dutch SME’s!!! ► average SME (11-50 workers): 14 % realized product innovation (CIS data) ► Voucher applicants (11-50 workers) : 75 % Failing to control for differences in innovation capacity would lead to huge overestimation of effects ► Unobservable factors can be crucial, e.g. timing of having an innovative idea may determine decision to apply for an innovation subsidy

The Dutch Innovation voucher Hungary May 18, 2011 Crucial issues for design and evaluation of voucher scheme 1.Identify measurable goals of voucher policy 2.Randomization is crucial for evaluation! 3.Sufficient vouchers and size of control group 4.Collect pre-treatment characteristics / behaviour of firms ► via application form; administrative datasets 5.Make cooperation to evaluation obligatory ► also for control group of losers in lottery! 6.Avoid abuse ► e.g. print assignment to copy shop of university 7.Do not let losers of lottery apply in next wave ► or large enough time span between waves

The Dutch Innovation voucher Hungary May 18, 2011 The Dutch voucher instrument after 2004 pilot Two new larger pilots in 2005: 1000 vouchers Definitive instrument since 2006: ► around 6000 vouchers annually ► Split in small (E2500) and large (E7500) vouchers 2010: introduction of “private” vouchers ► Possibility to commission question to private knowledge provider 2011: new government stopped voucher program (as part of large budget cuts)

The Dutch Innovation voucher Hungary May 18, 2011 Dutch cohesion policy (1) Regression discontinuity 2007 plans by government to improve 83 most disadvantaged neighborhoods Substantial budget of 350 million euro per year Neighborhood action plans with goals to improve housing, jobs, education, integration and safety.

The Dutch Innovation voucher Hungary May 18, 2011 Dutch cohesion policy (2) Setup and evaluation design Ranking of neighborhoods on various socio- economic indicators: worst 83 have been selected Evaluation design: ► Exploit ranking with cut-off at neighborhood 83 ► Exploit availability of before and after information on range of relevant outcome indicators Diff-in-diff: compare development in 1-83 with that in ‘ Local’ diff-in-diff design: versus

The Dutch Innovation voucher Hungary May 18, 2011 Same pre-trends supports common trend assumption: Liveability index (scale 1-7)

The Dutch Innovation voucher Hungary May 18, 2011 Common pre-trends also for wide range of other characteristics % of social housing % of non-western immigrants

The Dutch Innovation voucher Hungary May 18, 2011 No effect on primary school test scores after one year But we need to measure later-year effects… :

The Dutch Innovation voucher Hungary May 18, 2011 Other evaluation plans: Innovation and Science policy Innovation loans ► Selection of proposals on basis of certain criteria / judgements ► Compare performance of just selected versus just not selected companies Scholarships for talented researchers “Veni & Vidi grants” ► Selection of researchers on basis of ranking of proposals by comittee ► Compare scientific output & careers of just selected versus just not selected applicants

The Dutch Innovation voucher Hungary May 18, 2011 Contact Report on innovation vouchers: 58/ Contact: