1. The NETIM Monitoring Tool
Brokerage Event: Peer learning for innovation agencies
Brussels, February 15th 2017

2. NETIM Partners
ARTI – Agenzia Regionale per la Tecnologia e l’Innovazione
Regional Agency for Technology and Innovation, Lead Partner, Italy
IFKA – Iparfejlesztesi Kozhasznu Nonprofit Korlatolt Felelossegu Tarsasag
Public Benefit Non-Profit Ltd. for the Development of Industry, Partner, Hungary
FUNDECYT PCTEX – Fundacion FUNDECYT Parque Cientifico Y Tecnologico De Extremadura
Foundation FUNDECYT Science and Technological Park of Extremadura, Partner, Spain

3. NETIM = NEw Tools for Innovation Monitoring
The problem and challenges:
Monitoring innovation dynamics in the territories not easy.
Traditional indicators not always appropriate.
Availability of data at the desired disaggregation scale not granted.
Direct data collection costly.
New generation of highly integrated innovation policies (see RIS3) even more difficult to monitor and evaluate.

4. The project choices
Concentrate efforts on the monitoring of S3 policies
need to include the specialization dimension
a new challenge for regions
no consolidated approaches
Focus pilot implementation on policies that are common to more partners (Puglia and Extremadura regions):
support to R&D&I in companies
support to innovative companies start-up

5. The project approach
Disaggregation of all collected indicators across the different RIS3 priorities, according to a three-tier classification (Priority Areas, Innovation Value Chains, Innovation Fields)
Use of different indicator sets: input, output, result, transition
Monitoring based on information mandatorily provided by the participants to public calls and beneficiaries
Monitoring data collected through questionnaires, at project submission time, at project conclusion and in later follow-up phases.
Direct surveys, focus groups and other qualitative approaches used for impact evaluation

6. Designing the data collection system
The monitoring system
Different categories of information collected:
direct outputs of the funded projects
direct impacts of the funded projects on the relevant dimensions of innovation behaviour of the beneficiaries (related to the selected result and transition indicators)
any change in the relevant dimensions of innovation behaviour of the beneficiaries (related to the selected result and transition indicators)
Two entities are considered in this regard:
The project, with its expected outputs and result
the individual project proposers and beneficiaries, with their economic and innovation performance

7. Designing the data collection system
The evaluation of impacts
The monitoring data related to result and transition indicators is extended by surveying a larger population of regional actors (non beneficiaries)
Counterfactual analysis approaches such as difference of differences can be used by defining appropriate control groups.
Other soft methods are used to complement the analysis: interviews to key stakeholders and opinion leaders, focus groups discussions, etc.

8. Smart Specialization Strategy Goals and objectives Priority Areas
Innovation Value Chains
Innovation Fields
Regional Operational Program
Measures/Actions
Input Indicators
Output Indicators
Result Indicators
Additional policy specific
output and result indicators
S3 specific result indicators
Transition indicators
Projects mapping to S3 sectors/fields
Mapping criteria
Ex-ante Project Questionnaire
Projects
Project Beneficiaries
Clustered sets of indicators for S3 monitoring and impact evaluation

9. Project Beneficiaries Questionnaires for SMEs
Confronting ex-ante and ex-post indicators values for monitoring variations over time
Questionnaires for SMEs
Human resources
Certifications
R&D investments
Product and process innovations
Environmental innovations
Intellectual property rights
R&D capability
Collaboration network
Financing sources
Ex-ante data
compute variations
Ex-post data
Confronting beneficiaries data with control group data for counterfactual analysis of impacts
Control group

10. A three level segmentation of Puglia region S3
all projects submitted to S3 calls are classified according to this tree
all the information collected with reference to a given project (indicators) is then attributed to a specific priority area, innovation value chain, innovation field (indicators segmentation)

11. International openness
AREA
RESULT INDICATORS
Firms’ dynamics
Survival rate of start-ups
Growth in the number of start-ups
Growth in the number of academic spin-off
Companies birth rate in knowledge intensive sectors
Growth in the number of firms
Human resources
Growth of employees in high-tech and Knowledge intensive sectors
Percentage of researchers on the total number of employees
Growth in the average companies’ dimension
Employment growth in start-ups
Networking
Number of industry-academia research agreements
Percentage of firms having cooperating agreements with other organizations for innovative activities in the previous three years
Percentage of start-ups collaborating with large companies
Companies developing R&D in collaboration with research organization
Percentage of SMEs collaborating with large companies
International openness
Number of R&D international project proposals
Number of new firms attracted in the region
Number of new local units of existing firms in the region
Number of firms exporting for the first time
Innovative behavior
Growth in the number of international/national patents in the field
Growth in the number of trademarks
Growth in the number of designs in the field
Percentage of firms introducing a product/process/organizational innovation
Percentage of firms introducing a process innovation
Percentage of firms introducing organizational/marketing innovation
Result indicators for Puglia region S3

12. TRANSITION INDICATORS
Transition indicators for Aerospace Innovation Value Chain in Puglia
ACTORS
TRANSITION INDICATORS
SMEs + Large companies
Percentage of engineers on total employees
SMEs
Share of SME’s sales to large companies located outside the region
Number of research projects of SMEs in collaboration with large firms
Number of SMEs able to supply a complex product or manage the entire life cycle of a product
Large companies
Number of R&D employees in large firms
Percentage of outsourced service of large companies to regional SMEs
Number of research projects of large firms in collaboration with regional R&D organizations
Number of research projects of large firms in collaboration with regional SMEs
RTOs
Number of research projects of R&D organization in collaboration with regional companies

13. The Ex-ante project questionnaire
PROJECT INFORMATION
OBJECTIVES
EXPECTED RESULTS
KEYWORDS
KEY TECHNOLOGIES
KEY ENABLING TECHNOLOGY
INNOVATION FIELD
INNOVATION VALUE CHAIN
PRIORITY AREA
The Ex-ante project questionnaire
one single questionnaire for each project proposal
few questions, simple and fast to compile
guided online compilation trough web interface
the project proposers specify to which S3 specialization area the project contributes and what technologies are used in order to realize the intended innovation

14. INNOVATION CAPABILITY NETWORKING AND COLLABORATION
GENERAL COMPANY DATA
NACE code
Human Resources
INNOVATION CAPABILITY
Turnover
R&D expenditure
Innovation
Patenting
Certifications
Public funding
NETWORKING AND COLLABORATION
The Ex-ante questionnaire for applicants
one questionnaire for each member of the project consortium
the questions explore different areas of innovative behaviour

15. INNOVATION CAPABILITY NETWORKING AND COLLABORATION
GENERAL COMPANY DATA
NACE code
Human Resources
INNOVATION CAPABILITY
Turnover
R&D expenditure
Innovation
Patenting
Certifications
Public funding
NETWORKING AND COLLABORATION
The Ex-post questionnaire for beneficiaries
one questionnaire for each member of the project consortium
the same typology of information collected at submission time is evaluated at project conclusion
additional questions related to:
impact of the project on the organization
appraisal of the regional policies
future perspectives

16. NETWORKING AND COLLABORATIONS
HUMAN RESOURCES
Trained/involved researchers
Contract typologies used
INNOVATION RESULTS
Innovations
Environmental innovations
IPR
Certifications
NETWORKING AND COLLABORATIONS
New projects
Partners involved
The Ex-post project questionnaire
one single questionnaire for each project proposal
compiled at project conclusion
the project proposers describe the kind and extent of outputs and results produced by the project

17. Assuring an effective institutional framework
An effective and efficient synergy and communication must be realized among all the institutions that contribute to define, plan, manage, monitor and evaluate the public policy intervention of interest:
The policy makers
The intermediate bodies who manage the policies implementation
The monitoring and evaluation body
The external bodies providing data, surveys and analyses

18. ARTI – Agenzia Regionale Tecnologia e Innovazione
Thanks!
Giuseppe Creanza
ARTI – Agenzia Regionale Tecnologia e Innovazione
Via Giulio Petroni 15/F Bari – Italy Tel Mob Skype creanza.arti
This project has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement No The contents of this document are the sole responsibility of the NETIM partnership and can under no circumstances be regarded as reflecting the position of the European Union or of the Programme’s management structures.