1. Different sources and methods for monitoring indicators: the role of stakeholders Elisabetta Marinelli
Barcelona, 4 May, 2017

2. Introduction – stakeholders and indicators
Involved in definition of indicators/targets
Validators of the indicators and targets
Sources of information (primary data)
Focus on output and results indicators
Methodological implications (mixing methods)

3. Core indicators – Overview
Aim
Data sources
Output
Measure of the direct product of our policy intervention
Administrative data:
Information system of the Managing Authority/ RIS3 Managing Unit (quant/qual)
Result
Measure of the socio- economic dimension we are interested in
Quantitative
Official statistics
Other: i.e bibliometrics, etc.
Ad hoc surveys
Qualitative
Fieldwork

4. Applicants' input in application process
OPTION 1 VS OPTION 2
Monitoring Policy Outputs –Stakeholders as sources of information OP
Instrument 2
Instrument 3 … OP
indicators.
Instrument 2
Indicators
Instrument 3
Indicators ….
Applicants' input in application process
Opportunity to go down to priority level
RIS3 Indicators

5. Output indicators –Stakeholders as data sources
Type of info
On projects proposed
i.e. RIS3 Priority
On applicants
i.e. economic category of activity
Geo-references
Other characteristics
Methods
Quantitative
Survey (Puglia)/In-build request in the IS (Toscana)
Qualitative
text mining (Tuscany- Taxonomy or Keyword analysis of projects proposal, conversion to Cordis)
Input in application process

6. Results indicators – Stakeholders as data sources
Surveys of beneficiar.
Surveys of target groups
Other qual. methods
Question to Present regions for surveys to target groups – can you compare in your surveys beneficiaries vs non beneficiaries? Can you distinguish by priority area?

7. Results indicators – Stakeholders as data sources
Surveys of beneficiar.
Direct insights on
impact of the policy
quality of implementation
effectiveness of behavioral change
Evolution of priorities/Emerging issues
Puglia: Ex-post project & beneficiary questionnaire
Surveys of target groups
Other qual. methods
Question to Present regions for surveys to target groups – can you compare in your surveys beneficiaries vs non beneficiaries? Can you distinguish by priority area?

8. Results indicators – Role of Stakeholders
Surveys of beneficiar.
Surveys of target groups
Other qual. methods
Northern Netherlands: SME innovation survey
Novelle Aquitaine: Thematic survey
Extremadura: Encuesta Catedra de innovacion
Catalunya: Innovation barometers/ StartUp-Hub/Emerging activities
Can be used also as "context

9. Good practice on surveys
Coordination of national/regional level
Interregional/ national benchmarking
Test vs control group analysis
Check intra-regional effects
Important for large regions
Go back" to the instruments
Monitor priorities and their relationship
Link to intervention logic and S3 Priority
Geo-reference respondents
Link different classifications (e.g. NACE, CORDIS)
Beneficiaries vs. Applicants vs. other

10. Results indicators – Role of Stakeholders
Process/Interpretation (validation, communication, governance)
Structured dialogue / Focus groups /Participatory Leadership /Foresight/Ethnography/Key informant interviews
Role of moderator
Extremadura – Monitoring Working Group
Netherlands – sustainable development (Non RIS3)
Surveys of beneficiar.
Surveys of target groups
Other qual. methods
Question to Present regions for surveys to target groups – can you compare in your surveys beneficiaries vs non beneficiaries? Can you distinguish by priority area?

11. Primary data vs Official statistics
Sources
Pros
Cons
Official statistics
Reliable, cheap, comparable
Time lag
Not necessarily priority-specific
Surveys
Timely
Flexibile
Expensive
Representability
Response rate
Other qualitative data
Address small communities (not suitable to surveys)
Focus on process
Interpretation bias
Unreplicability

12. Mixing data in the monitoring system
Single method
Mixed m. Parallel
Mixed m. Sequential
Mixed m. Iterative
time, skills, budget ->
RIS3: long-term effort – it make sense to build capacity, spend time and resources

13. Reality check
Puglia
Catalunya
Experiences
Challenges
Comments