TAFTIE Policy Forum “Measuring innovation” Can we measure innovation? Lessons from innovation scoreboards Hugo Hollanders Brussels, November 25, 2014

What is an innovation scoreboard? “Innovation scoreboards consist of a collection of national or regional indicators of relevance to innovation” Arundel, A. and H. Hollanders, "Innovation Scoreboards: Indicators and Policy Use", in C. Nauwelaers and R. Wintjes (eds.), Innovation Policy in Europe, Edward Elgar: Cheltenham, 2008 “Innovation scoreboards are an attempt to summarize innovation indicators and to compare innovation performance of countries, regions or sectors” Hollanders, H. and N. Janz, "Scoreboard and indicator reports", in F. Gault (ed.), Handbook of Innovation Indicators and Measurement, Edward Elgar : Cheltenham, 2013

Purpose of innovation scoreboards The main purpose of an innovation scoreboard is to assist policy by summarizing a range of innovation indicators at the national, regional or sectoral level by permitting a comparison of the relative success or failure of the innovation system, or through the identification of specific aspects of the innovation system which perform well or poorly

Policy relevance Innovation scoreboards serve three policy needs They act as an ‘early-warning’ system for potential problems If used over time, they can track changes in innovation strengths and weaknesses They can attract the interest of policy-makers and as such can help build a consensus to introduce actions and policies to improve the innovation capabilities of firms and institutions Policy use Innovation scoreboards will be most useful for policy when combined with an integrated system for evaluating innovation policies

Principles of scoreboard design The indicators should Be of similar importance as measures of the drivers of innovation activity Be based on reliable statistics Hold their value over time Be of relevance to medium- and long-run policy issues

Examples of innovation scoreboards European Commission Innovation Union Scoreboard (IUS) Innovation Union Competitiveness Report OECD Science, Technology and Industry Scoreboard OECD Science, Technology and Industry Outlook INSEAD/WIPO Global Innovation Index World Economic Forum Global Competitiveness Report World Bank Knowledge Assessment Methodology Deutsche Telekom Stiftung, BDI Innovationsindikator

Examples of innovation scoreboards

IUS uses 25 indicators which are grouped into 3 main categories: Enablers capture the main drivers of innovation performance external to the firm Firm activities capture the innovation efforts at firm level Outputs capture the effects of firms’ innovation activities Average performance is measured by a composite indicator, the Summary Innovation Index 8 Innovation Union Scoreboard IUS monitors the implementation of the Europe 2020 Innovation Union flagship by providing a comparative assessment of the innovation performance of the EU28 Member States and the relative strengths and weaknesses of their innovation systems

9 IUS Measurement framework

IUS Performance groups 10

Differences in rankings

IUS International comparison Europe’s performance gap with the US and Japan is closing but the gap with South Korea is increasing Europe is

Lessons from innovation scoreboards What are the strengths and weaknesses of the innovation measurement tools and indicators we use today? Strengths Country coverage, measuring variety of innovation activities, good coverage of inputs and throughputs, use of innovation survey data Weaknesses Partial coverage of outputs, use of innovation survey data

Lessons from innovation scoreboards What’s the state of play on the development of new indicators? Community Innovation Survey 2014 Better coverage of public sector as a driver of innovation Public procurement Ad-hoc modules: 2012 on growth, 2014 on innovations with environmental benefits EC’s Innovation Output indicator Combines 5 of the IUS indicators into one indicator measuring outputs of which one is a new indicator: Employment in fast-growing firms of innovative sectors

Fast-growing firms innovative sectors 2010: High-Level Panel asked to recommend new headline innovation indicator : work by OECD/Eurostat to identify the innovativeness of sectors 2013: indicator launched by EC Employment in fast-growing firms of innovative sectors provides an indication of the dynamism of fast-growing firms in innovative sectors as compared to all fast-growing business activities. It captures the capacity of a country to transform rapidly its economy to respond to new needs and to take advantage of emerging demand

Lessons from innovation scoreboards What has not been measured yet? Outputs, current indicators focus on Employment in high-tech and knowledge-intensive activities Exports in high-tech and knowledge-intensive activities Both of which do not necessarily have to be directly linked to innovation Sales of innovative products Data are not available for all countries and there are issues about the comparability of results across countries Commercialization activities Innobarometer 2014

Commercialization of innovations

Commercialization of innovations

Thank You

Lessons from innovation scoreboards Is there a better taxonomy and nomenclature for the innovation process? How can we develop the process of finding the exact tools to observe, monitor and describe policy relevance, data quality (international availability and cross-country comparability) and robustness of results? What could be the role of TAFTIE in developing innovation indicators and ensuring the most relevant picture of impact of TAFTIE support?