1. SAI’s role in development and use of key indicators for R&D evaluation: a quantitative example and some concluding remarks INTOSAI Working Group on Key National Indicators Ville Vehkasalo & Timo Oksanen, 23.4.2013, Krakow

2. Presentation outline Our stance on indicator development Example of how to use key indicators in quantitative R&D evaluation Qualitative evaluation possibilities Concluding remarks; incorporation into the White Paper on KNI 2

3. 3 About SAI’s role in indicator development Depending on the national mandates, the SAI’s role can be active or passive – or something in between However, an active role in indicator development can endanger SAI’s independency and objectiveness The NAO of Finland has not participated in Finland’s KNI development Therefore, we have kept an outsider’s view to Finnish KNI-system

4. Example: how can we use key indicators in quantitative R&D evaluation? EU’s Regional Development Fund (ERDF) aims to achieve the following objectives in 2007–2013: 1) to enhance regional R&D and innovation capacities 2) to stimulate innovation and entrepreneurship in all sectors of the regional and local economy 3) to promote entrepreneurship, in particular by facilitating the economic exploitation of new ideas and fostering the creation of new firms. Source: Regulation (EC) No 1080/2006 4

5. Example Cost of the ERDF program in Finland, 2007– 2013: 1,7 billion euros (EU funding) The effects of ERDF program are monitored using these indicators: 1) number of new firms 2) number of jobs 3) unemployment rate 4) employment rate 5

6. Example 5) regional GDP increase relative to the whole economy 6) share of exports in firms’ turnover 7) share of R&D activities in GDP 8) average educational level. Source: ERDF Program of Southern Finland 2007–2013 6

7. Example The number of new firms is included in Finland in Figures, which contains key statistical data about Finland on 25 different statistical topics, produced by Statistics Finland This statistic is not included in Findicator, the official key indicator compilation 7

8. Example How can we measure the effects of the 2007– 2013 ERDF program in Finland? Counterfactual: what would have happened without the program? We need a control group that was not subjected to the program But in 2007–2013, the whole country is included in the ERDF program 8

9. Example However, in the earlier ERDF program, 2000– 2006, small parts of Southern Finland were not included in the program Therefore, we can compare the development in these new municipalities to those in Southern Finland that had been included earlier (old municipalities), in order to control for economy- wide fluctuations that may also affect start-ups Population changes can be accounted for by using per capita figures 9

10. Example Straightforward comparison is out of the question, as old and new municipalities have systematic differences: new firms per 1000 capita, population-weighted means year 2005year 2011 old municipalities5,045,25 new municipalities7,057,37 Even before joining the program, new areas had higher rates of firm creation 10

11. Example In order to control for unobservable characteristics, we have to use panel data: the same municipalities before and after the policy change Specifically, we use the number of new firms from 2005 (before) and 2011 (after) in each of these municipalities Small sample: only 31 new municipalities vs. 34 old ones (N = 65) 11

12. Example We use the difference from 2005 to 2011,  y = y 2011 – y 2005, as the independent variable Differencing wipes out time-invariant characteristics, such as proximity to a larger city Regression  y =  +  new_munic Coefficient estimates are: coef.robust s.e. p-value new_munic-0,095 0,3520,788 constant0,150 0,2000,455 12

13. Example new_munic estimate has wrong sign but it is statistically insignificant Previous estimates are unweighted, i.e. small and large municipalities get the same weight, or importance, in the results Alternatively we can use weights that measure the size of the municipality, for instance population levels 13

14. Example If we use 2005 population levels as weights, we get these estimates: coef.robust s.e. p-value new_munic0,110 0,1710,525 constant0,205 0,1230,101 Again, can not reject null hypothesis 14

15. Example Average change of +0,31 in the intervention group differs from zero (p = 0,014) but it would be misleading to attribute this to the program We had an average change of +0,2 in the municipalities that were included earlier, i.e. even without this “new” program The ERDF program did not cause the observed increase of 0,31 in the number of new firms 15

16. Example This example is a bit unrealistic (sample too small, etc.) but it illustrates the basic quantitative evaluation framework: 1) Gather relevant data on intervention and control groups, before and after the intervention 2) Use simple difference-in-differences regression or standard panel data methods 3) Present your results with careful interpretation 16

17. Qualitative methods Quantitative methods are useful in assessing program effectiveness In addition, there are various qualitative approaches to R&D evaluation, such as interviews and participant observation Possible explanations to why or how something happened/did not happen as planned General conclusions not possible 17

18. R&D subproject conclusions (1): Evaluating specific programs and interventions Evaluation of R&D programs is difficult, but not impossible Finding relevant data can be tricky Not possible to evaluate all programs; must have control groups Without proper analysis, indicators are of limited use in program evaluation 18

19. R&D subproject conclusions (2): Evaluating the whole R&D system as a part of modern society Problems are threefold: normative, causative and conceptual Lack of clear, strategic whole-of-society vision communicated by the government (normative) Lack of understanding and knowledge about the general impacts of R&D system on modern economies (causative) What would and could be the role of SAIs and Key National Indicators of R&D in all of this? (conceptual) 19

20. R&D subproject: Incorporation into the White Paper on KNI WG Secretariat can freely use our reports in preparing/editing the White Paper on KNI For instance, our reports could be useful in augmenting the section Principles and Guidelines, subsection Guidelines for knowledge- based economies, where the evaluation of R&D programs is already mentioned 20

21. R&D subproject: List of reports Utilising R&D knowledge at R&D policymaking in Finland: problems and promises, Helsinki 2011 (.ppt) SAI’s role in development and use of key indicators for R&D evaluation, Riga 2012 (.ppt) SAI’s role in development and use of key indicators for research and development (R&D) evaluation, 2012 (.doc) SAI’s role in development and use of key indicators for R&D evaluation: a quantitative example and some concluding remarks, Krakow 2013 (.ppt) 21

22. Thank you! ville.vehkasalo@vtv.fi timo.oksanen@vtv.fi http://www.vtv.fi/en 22