The current Finnish development of KNIs and their use in the SAI's activities Timo Oksanen & Ville Vehkasalo INTOSAI WG KNI, 24.– , Bali

Part 1: Development of KNIs 2

Finnish KNI compilation Findicator About visitors per month New theme “sustainable development indicators” (slide 4) added in October 2013 Further details: More new indicators coming soon on the participation of citizens Still only few R&D&I indicators 3

Sustainable development indicators in Findicator: 4

FIRM - Finnish Innovation and Research Measurement: description of a new model in a new service The aims of this service are as follows: show visually and in a holistic way, how Finland’s innovation system is performing in comparison with the best countries to analyze the different dimensions of research and innovation activities and especially to communicate the impacts of research and innovation inputs 5

FIRM - Finnish Innovation and Research Measurement The Research and Innovation Council of Finland has approved the basic starting points of the model, prepared by the Finnish Funding Agency for Innovation (Tekes) and the Academy of Finland in December 2012 The Council proposed that the model has to be tested with an international comparison Tekes and the Academy of Finland organised a survey in November 2013 for researchers and experts in OECD-countries to test the model and to find new development ideas 6

FIRM - Finnish Innovation and Research Measurement The Research and Innovation Council is also preparing a review Research and Innovation Policy Guidelines for 2015–2019 O ne of the central focuses of the review will be the social effects and impacts of R&D&I Learning and follow up of important international development in the field of R&D, e.g. OECD (CSTP) project “Assessing the Role of Impact in National Science and Research Systems” 7

Part 2: Use of indicators and statistics in performance auditing 8

9 Use of indicators The Finnish KNI compilation Findicator was launched in 2009 How are indicators and statistics used in today’s performance audits? We searched the pdf-documents of all performance audit reports from the years 2010–2013 using search words ”indicator” and ”statistic” We studied the context of all matches and classified the reports accordingly Search was not limited to key national indicators

Use of indicators Based on search results, we classified the reports to four categories: 1) indicators or statistics were not used 2) included some descriptive statistics 3) included approximate or informal data analysis 4) included rigorous statistical inference 10

Use of indicators In total, there were 56 published performance audit reports in 2010–2013; on average 14 reports per year In 30 reports, indicators or statistics were not used in at all Category 1 proportion: 54% 13 reports included data as descriptive statistics (graphs, tables etc.) Category 2 proportion: 23% 11

Use of indicators 5 reports included some kind of informal or approximate analysis of data (visual inspection, comparisons etc.) Category 3 proportion: 9% 8 reports included rigorous statistical inference (regression analysis, means testing etc.) Category 4 proportion: 14% 12

Performance audit reports 2010–2013 (N = 56) 13 no statistics 54% descriptive statistics 23% informal analysis 9% statistical inference 14%

Use of indicators All in all, roughly half of all published reports had used indicators or statistics in some way However, proper statistical analysis is infrequent The use of KNIs is still very rare in performance auditing, if we look at published reports Only 2 reports out of 56 (4%) included Findicator data 14

Use of indicators KNIs maybe too general? In most audits you might need more specific evidence Rigorous data analysis is demanding and most auditors lack the necessary skills Easier to use data only as descriptive statistics Traditional approach to performance auditing is quite persistent; people seldom want to change their working methods and habits 15

Conclusions Development of indicators is an ongoing process Still lots of things to do, especially in the R&D sector Use of KNIs is rare in performance audit reports Proper statistical analysis is also infrequent Old-school auditing still dominates 16

Thank you! & web contacts: 17