1. Interim evaluation of RIS3 in
Castilla y León
S3 Targeted Support
Barcelona, 2nd July 2019

2. Tender documents and specifications

3. Tender: Technical Specifications Dossier
Subject-matter of the contract
Interim evaluation.
Updating of the RIS
Available information
In the website:
Annual work plans & monitoring reports
Specific plans
Regional Specialisation Pattern.
The consultant never asked for more information…

4. Tender: Technical Specifications Dossier
Issues to be evaluated
Changes in the context:
National and international
Performance, legislation, strategic framework…
Thematic priorities: validity.
Through the update of the Regional Specialisation Pattern
Objectives: degree of achievement (quantitative indicators).
Actions.
Rhythm of implementation
Effectiveness – Efficiency - Sustainability (for the next programming period)

5. Tender: Technical Specifications Dossier
Activities to be carried out by the consultant
Analysis of the new context:
Socio-economic situation, R&I indicators (including Information Society)
New legislative framework (including plans), at national and European level
In R&I, Info Society and ESIF
Analysis of the RIS3 and its suitability in the new context:
Regional Specialisation Pattern
Thematic priorities
Objectives
Programmes
Governance system
Monitoring & evaluation system
Mobilised resources
Validation with Administration and stakeholders
Writing of the report.
Including conclusions and recommendations
Compared to the previous page, there is a small difference (for instance, the analysis of actions), but it all should be put together.

6. Tender: Technical Specifications Dossier
Deliverables (reports)
In 3 months:
Analysis of new context and legislative framework
Analysis of the RIS3 deployment and its suitability in the new context
Forecast of:
Stakeholders to be involved
Information sources
Evaluation methodology
Detailed list of foreseen activities, specifying contents, dates and places.
In 6 months:
Evaluation report
Regional Specialisation Pattern
Very well specified, especially the part of the methodology to be used, to make sure that it was all right.
Regional Specialisation Pattern: should have been in advance, in order to assess the validity of thematic priorities, but like this, it was enough for the Update

7. Tender: Tendering Specifications
Price
€ 155, taxes (€ 188,000 VAT included)
Technical & professional capacity required
Minimum 3-people team, with experience:
Director: 5 years in R&I policies, and 3 years in Smart Specialisation
2 Seniors: 3 & 2 years, respectively
Minimum € 100,000 invoiced in such activities in at least 1 of last 5 years.
Tendering specifications: also called “Contract documents”

8. Tender: Tendering Specifications
Assessment criteria
Quantitative (price): 30%
Not quantitative: 70%
Methodology and work plan: 40%
Criteria: conformity with the work’s objective, degree of development, concretisation.
Applied to: work plan, methodology, actions proposed, chronogram, individual dedication, deliverables
Adaptation to the regional system: 10%.
Links to national and European context: 10%.
Technical methodological improvements (10%), related to:
Analysis of new context and Strategy’s implementation.
Evaluation techniques, to determine the Strategy’s impact.
Tendering specifications: also called “Contract documents”
However, the deliverables are already specified.

9. Tender: proposal selected

10. Tender: proposal selected
Assessment criteria
Quantitative (price): too low: 105 k€ vs. 140; 2,350 hours vs. 1,600
Not quantitative:
Methodology and work plan:
Based on EC / JRC methodologies.
Interesting approaches: sectoral value chains: weak links =challenges => flagship initiatives.
Relation RIS3 – ROP.
Good participative process.
Very clear work plan.
Adaptation to the regional system: OK
Links to national and European context: OK
Technical methodological improvements:
Experiences from other regions: experts in the team.
Plan of sustainability for the period without ESIF.
Flagship initiatives happened to be pre-conceived, not derived from any analysis.
Eventually, only from the ROP perspective.
The contents were often too poor to organise meetings to present results
Only 1 expert from Galicia; only came to 2 meetings, at the beginning, with General Directors; proposed also the former Director from the Portuguese National Innovation Agency (never showed up).

11. Deficiencies in the evaluation report

12. Deficiencies in the evaluation report
Analysis of implementation
Too low number of actions examined in depth (13 / 370).
No effectiveness or efficiency analysis.
No available information, or heterogeneous.
Design & calendar of instruments.
If the actions’ managers were ready to commit to it.
Analysis of potentialities
Analysis of competitiveness of economic macroactivities.
Evolution of GDP and of exports.
Identification of strong actors in each sector.
Identification of emerging sectors.
Analysis of labour market.
Economic fabric’s absorption capacity.
Identification of opportunity niches.
Match between training / educational supply and market demand.
They said that they had internal back-office information, but we had to ask for it. True or false?
They said that it was because the information provided was insufficient.
The identification of actors was out of the sectors (pre-conceived).

13. Deficiencies in the evaluation report
Outward looking
Complementarity with national programmes, and duplicities.
Identification of trans-regional opportunities.
From the analysis of other regional RIS3 priorities.
Logic of intervention
Adaptation to the new context.
Effectiveness of actions undertaken.
Attainment of the objectives: performance, effectiveness and efficiency.
In all 3, lack of information.
Conclusions
Written before the rest of the document.
Incoherent & contradictory.
At European level, they wrote a list of the existing programmes (that was all).
It could be because they had raw information, but it seemed that they were previously conceived.

14. Lessons learnt

15. Lessons learnt Risks Murphy’s Law: If it works, DON’T TOUCH IT!!
The more detailed information in tender documents:
Proposals more similar, and less “innovative” / interesting.
Therefore, more difficult to evaluate (differentiate).
Accepting potentially reckless price reductions.
Flexibility may lead to significant deviation from the original proposal.
Problems (difficult to foresee)
Lack of initiative of the contractor:
Collection of necessary information (not available initially).
Different quality expectations: concretisation, field work…
Anti-commercial attitude.
Focus on their real skills / knowledge.
Even, they used the recordings from meetings to demonstrate what we had said (probably out of context).

16. Lessons learnt Problems from “our” side
Lack of high-level commitment-monitoring.
Sometimes, contradictions between technical-managerial-political levels.
Time constraints: accept “anything”.
Procurement regulations: you have to accept them again, if the tender is good.
Importance of monitoring as a source of relevant information.
Too much control on the evaluators’ team?
Other problems (difficult to avoid)
Criticism towards the predecessor.
The Director “sells”, and the Junior “works”.

17. for Science and Technology
Thank you
for your attention
Gregorio MUÑOZ-ABAD
Deputy Commissioner
for Science and Technology