Beyond surveys: the research frontier moves to the use of administrative data to evaluate R&D grants Oliver Herrmann Ministry of Business, Innovation & Employment of New Zealand oliver.herrmann@med.govt.nz 

Overview Survey data vs administrative data Micro data research in New Zealand Using administrative data for evaluation: an R&D example Impact evaluation, causality and methods Evaluation findings Limitations

Survey data (Last Millennium) Empirical research and evaluation have relied on survey data. Using a sample of individuals from a population with a view towards making statistical inferences about the population. Examples: Current Population Survey (US), Annual Survey of Hours and Earnings (UK), Business Operation Survey (NZ) Political science, sociology, and economics were all “revolutionized” using survey data sources. Empirical research and evaluation have relied on survey data The methodology: using a sample of individuals from a population with a view towards making statistical inferences about the population. Examples: Current Population Survey (US), Annual Survey of Hours and Earnings (UK), Business Operation Survey (NZ) During the second half of 20th century: development of modern survey methods and of statistical techniques for analysing survey data. Enormous insights in the fields of political science, sociology, and economics were all revolutionized using survey data sources. Lessons: include knowledge and additions of qualitative perspective

Administrative data (New Millennium) Governments create comprehensive administrative data that cover socio-economic behaviour from education, earnings, workplace to family composition, health and retirement. Administrative data are highly preferable to survey data because full population files are generally available and offer much larger sample sizes, administrative data have a longitudinal structure that enables evaluators to follow individuals over time and address many critical policy questions, administrative data do not suffer from high rates of non-response, attrition, and under-reporting. Overcome the linear problem

New Zealand’s Hotbed of Microdata Research Statistics NZ has undertaken a number of projects that integrate data supplied by different government agencies The Integrated Data Infrastructure (IDI) is a comprehensive database with longitudinal microdata about individuals, households, and firms The IDI pulls together a range of administrative and survey-based data sources (financial performance, employment, merchandise trade, business practices, government assistance). The IDI allows for the investigation of previously unanswerable questions. Researchers and evaluators have access to answer research, policy, and evaluation questions to support informed decision making. The Integrated Data Infrastructure (IDI) is a comprehensive person- and firm-level database for Government researchers to utilise The IDI pulls together a range of administrative and survey-based data sources, resulting in a rich and multi-dimensional In addition to contextual and other information

Using administrative data for evaluation: An example - publicly funded R&D Innovation is well known to be an important driver of economic growth and investments in R&D are among the factors that drive innovation. Governments encourage business investment in R&D, with the aim of correcting or alleviating two main market failures: Difficulties by firms to fully appropriate the returns to their investment. Difficulties in finding external finance, in particular for small start-up firms.

Policy objectives of R&D grants Increase and enhance the R&D. Increase the economic performance. In New Zealand there are different types of government R&D subsidies assistance to build R&D capability assistance for R&D projects

Impact evaluation = causal inference Identify the causal effect of a policy or intervention. We did program X, and because of it, Y happened. Use the results to decide whether the programme should be continued/terminated/expanded/modified Y happened because of X, not for some other reason. Thus it makes sense to think that if we did X again in a similar setting, Y would happen again In a more research-friendly universe, we’d be able to observe a single firm (called company ICT) after government both gave and didn’t give R&D assistance: Ytreated ICT - Yuntreated ICT

Evaluation problem (1) In reality, we observe one outcome only The outcome we do not observe is called the counterfactual. In order to estimate the impact of treatment, we need to estimate the counterfactual outcome Is the solution to get more observations? Then we can calculate: Average(treated) - Average(untreated) But what if there’s an underlying difference between the treated and untreated (selection bias)?

Evaluation problem (2) Simplest world : impact is common across all individuals/firms Impact is different across individuals/firms but doesn’t affect participation/selection => variation in impact has few policy implications Impact is different across individuals/firms and this affects participation/selection => implications for policy and for unbiased estimation

Non or Quasi-experimental approaches We use non-experimental approaches to build a counterfactual - selection on observables (matching) - selection on unobservables (Difference in differences)

Control group and eight matching variables Employment and change in employment Total productivity and change in total productivity Capital intensity and change in capital intensity Exporting industry Firms in group Firm age R&D activity Ownership status

Actual and counterfactual outcomes Changes in performance of assisted firms to matched similar New Zealand firms. The additional impact result of the public subsidy alone Comparing actual and counterfactual outcomes.

Evaluation findings Even before receiving R&D grants assisted firms have higher sales, are larger, more capital intensive, exporting and undertaking R&D. Significant impact on economic performance of the firms due to the R&D subsidies. “Capability Building” has a positive impact on employment growth, sales, value added, and productivity. Counterintuitive finding: No impacts for “Project Funding” ! Positive impacts for small firms. Positive impacts for firms that had not undertaken R&D two years prior to receiving their assistance. No impacts for large firms and no impacts for prior R&D performers.

Limitations and related evaluations Data limitations (4 years after receiving assistance) Previous evaluations (case studies and surveys) showed very positive outcomes. Before/After comparison (completion of the grant) does not inform about any additional impact. Most previous evaluations have overestimated the economic impact. Our methods address this question by comparing the performance of an assisted firm to a matched similar firm.

Additional slides

Potential outcomes framework (Rubin, 1974)

Main identification problem is the lack of a counterfactual response

Matching Match each treated participant to one or more untreated participant based on observable characteristics. After matching treated to most similar untreated, subtract the means, calculate average difference

Issues Selection/participation depends only on observable characteristics; it does not depend on unobservables Difficult to match on many observables X => condense all observables into one “propensity score,” match on that score. Need good data

Difference in differences Before – after : Growth in outcomes for assisted firms Firm with assistance Outcome, Y (e.g. sales) Time t1 t2 Zero growth O Y0 Y1 Y'

Difference in differences Comparison of outcomes post treatment Y(b) – Y(d) Firm with assistance Outcome, Y (e.g. sales) Time t1 t2 O Y0 Y1 Y' b Control firms without assistance a d c

Difference in differences = Growth in sales for assisted firms – Growth in sales for control firms Firm with assistance Outcome, Y (e.g. sales) Time t1 t2 O Y0 Y1 Y' b e Control firms without assistance a d c