Evaluating the effectiveness of innovation policies Lessons from the evaluation of Latin American Technology Development Funds Micheline Goedhuys

June DEIP, Amman June, Structure of presentation 1. Introduction to the policy evaluation studies:  policy background  features of TDFs  evaluation setup: outcomes to be evaluated, data sources 2. Evaluation methodologies:  the evaluation problem  addressing selection bias 3. Results from Latin American TDF evaluation: example of results, summary of results, concluding remarks

June DEIP, Amman June, A. Introduction: Policy background Constraints to performance in Latin America S&T falling behind in relative terms: small and declining share in world R&D investment, increasing gap with developed countries, falling behind other emerging economies Low participation by productive sector in R&D investment: lack of skilled workforce with technical knowledge; macro volatility, financial constraints, weak IPR, low quality of research institutes, lack of mobilized government resources, rentier mentality

June DEIP, Amman June, A. Introduction: Policy background Policy response: shift in policy From focus on promotion of scientific research activities, in public research institutes, universities and SOE To (1990-…) needs of productive sector, with instruments that foster the demand for knowledge by end users and that support the transfer of Know How to firms TDF emerged as an instrument of S&T policy

June DEIP, Amman June, A. Introduction: Policy background IDB: evaluating the impact of a sample of IDB S&T programmes and instruments frequently used: Technology Development Funds (TDF): to stimulate innovation activities in the productive sector, through R&D subsidies Competitive research grants (CRG) OVE coordinated, compiled results for TDF evaluation in Argentina, Brazil, Chile, Panama (Colombia)

June DEIP, Amman June, B. Introduction: Selected TDFs Country and PeriodNameTools Argentina FONTAR-TMP ITargeted Credit Argentina FONTAR ANRMatching Grants Brazil ADTENTargeted Credit Brazil FNDCTMatching Grants Chile FONTEC-line1Matching Grants Panama FOMOTECMatching Grants

June DEIP, Amman June, B. Introduction: features of TDFs Demand driven Subsidy Co-financing Competitive allocation of resources Execution by a specialised agency

June DEIP, Amman June, C. Introduction: evaluation setup Evaluation of TDFs at recipient (firm) level Impact on :  R&D input additionality  Behaviour additionality  Innovative output  performance: productivity, employment and growth thereof

June DEIP, Amman June,

June DEIP, Amman June, IndicatorData source Input additionality Amount invested by beneficiaries in R&D Firm balance sheets; Innovation surveys; Industrial surveys Behavioral additionality Product / process innovation, linkages with other agents in the NIS Innovation surveys Innovative Outputs Patents; Sales due to new products Patents databases; Innovation surveys PerformanceTotal factor productivity Labor productivity; Growth in sales, exports,employment Firm balance sheets; Innovation surveys; Industrial surveys; Labor surveys

June DEIP, Amman June, A. The evaluation problem (in words) To measure the impact of a program, the evaluator is interested in the counterfactual question: what would have happened to the beneficiaries,… if they had not had access to the program This is however not observed, unknown. We can only observe the performance of non- beneficiaries and compare it to the performance of beneficiaries.

June DEIP, Amman June, A. The evaluation problem (in words) This comparison however is not sufficient to tell us the impact of the program, it presents rather correlations, no causality Why not? Because there may be a range of characteristics that affect both the possibility of accessing the program AND performing well on the performance indicators (eg R&D intensity, productivity…) Eg. size of the firm, age, exporting…

June DEIP, Amman June, A. The evaluation problem (in words) This means, ‘being in the treatment group or not’ is not the result of a random draw, but there is a selection into a specific group, along both observable and non-observable characteristics The effect of selection has to be taken into account if one wants to measure the impact of the program on the performance of the firms!! More formally….

June DEIP, Amman June, A. The evaluation problem Define: Y T = the average expenses in innovation by a firm in a specific year if the firm participates in the TDF and Y C = the average expenses by the same firm if it does not participate to the program. Measuring the program impact requires a measurement of the difference (Y T - Y C ) which is the effect of having participated in the program for firm i.

June DEIP, Amman June, A. The evaluation problem Computing (Y T - Y C ) requires knowledge of the counterfactual outcome that is not empirically observable since a firm can not be observed simultaneously as a participant and as a non- participant.

June DEIP, Amman June, A. The evaluation problem by comparing data on participating and non- participating firms, we can evaluate an average effect of program participation, E[Y T - Y C ] Substracting and adding E[Y C |D=1]

June DEIP, Amman June, A. The evaluation problem Only if there is no selection bias, the average effect of program participation will give an unbiased estimate of the program impact There is no selection bias, if participating and non- participating firms are similar with respect to dimensions that are likely to affect both the level of innovation expenditures and TDF participation Eg. Size, age, exporting, solvency… affecting RD expenditures and application for grant

June DEIP, Amman June, B. The evaluation problem avoided Incorporating randomized evaluation in programme design Random assignment of treatment (participation in the program) would imply that there are no pre- existing differences between the treated and non- treated firms, selection bias is zero Hard to implement for certain types of policy instruments

June DEIP, Amman June, B. Controlling for selection bias Controlling for observable differences Develop a statistically robust control group of non- beneficiaries identify comparable participating and non- participating firms, conditional on a set of observable variables X, i.o.w.: control for the pre-existing observable differences using econometric techniques: e.g. propensity score matching

June DEIP, Amman June, B. Propensity score matching (PSM) If there is only one dimension (eg size) that affects both treatment (participation in TDF) and outcome (R&D intensity), it would be relatively simple to find pairs of matching firms. When treatment and outcome are determined by a multidimensional vector of characteristics (size, age, industry, location...), this becomes problematic. Find pairs of firms that have equal or similar probability of being treated (having TDF support)

June DEIP, Amman June, B. PSM Using probit or logit analysis on the whole sample of beneficiaries and non-beneficiaries, we calculate the probability (P) or propensity that a firm participates in a program P(D=1)=F(X) X= vector of observable characteristics Purpose: to find for each participant (D=1) at least one program non-participant that has equal/very similar chance of being participant, which is then selected into the control group.

June DEIP, Amman June, B. PSM It reduces the multidimensional problem of several matching criteria to one single measure of distance There are several measures of proximity: Eg nearest neighbour, predefined range, kernel – based matching...

June DEIP, Amman June, B. PSM Estimating the impact (Average effect of Treatment on Treated): ATT=E[E(Y1 | D = 1, p(x)) –E(Y0 | D = 0, p(x))| D=1 ] Y is the impact variable D = {0,1} is a dummy variable for the participation in the program, x is a vector of pre-treatment characteristics p(x) is the propensity score.

June DEIP, Amman June, B. Difference in difference (DID) The treated and control group of firms may also differ in non-observable characteristics, eg management skills. If panel data are available (data of pre-treatment and post-treatment time periods) the impact of unobservable differences and time shocks can be neutralised by taking the difference-in-differences of the impact variable. Important assumption: unobservables do not change over time In case of DID, the impact variable is a growth rate.

June DEIP, Amman June, Example of results Impact of ADTEN (Brazil) on (private) R&D intensity Single difference in 2000 [(RD/sales 2000 beneficiaries – RD/sales 2000 control)] after PSM 92 observations each beneficiaries 1.18% Control group 0.52% Difference: 0.66% positive and significant impact,net of subsidy

June DEIP, Amman June, Example of results Impact of FONTAR-ANR (Argentina) on (public+private) R&D intensity (=R&D expenditures/sales) Difference in difference with PSM 37 observations each [(RDint. afterANR beneficiaries –RD/sales beforeANR ben.)- RD/sales afterANR control-RD/Sales beforeANR control)] Beneficiaries ( ) = 0.12 Control group ( ) = DID 0.19 positive and significant impact, GROSS of subsidy

June DEIP, Amman June, Results: summary The impact of the programs on firm behaviour and outcomes becomes weaker and weaker as one gets further from the immediate target of the policy instrument: There is clear evidence of a positive impact on R&D, weaker evidence of some behavioural effects, and almost no evidence of an immediate positive impact on new product sales or patents. This may be expected, given the relatively short time span over which the impacts were measured.

June DEIP, Amman June, Results no clear evidence that the TDF can significantly affect firms’ productivity and competitiveness within a five-year period, although there is a suggestion of positive impacts. However, these outcomes, which are often the general objective of the programs, are more likely related to a longer run impact of policy. The evaluation does not take into account potential positive externalities that may result from the TDF.

June DEIP, Amman June, Results the evaluation design should clearly identify: rationale short, medium and long run expected outcomes; periodic collection of primary data on the programs’ beneficiaries and on a group of comparable non- beneficiaries; the repetition of evaluation on the same sample so that long run impacts can be clearly identified; the periodic repetition of the impact evaluation on new samples to identify potential needs of re- targeting of policy tools.

June DEIP, Amman June, Concluding remarks The data needs of this type of evaluation are evident Involvement and commitment of statistical offices is needed to be able to merge survey data that allow these analyses The merger and accessability of several data sources create unprecedented opportunities for the evaluation and monitoring of policy instruments Thank you!