Interim evaluation of RIS3 in Castilla y León S3 Targeted Support Barcelona, 2nd July 2019

Tender documents and specifications

Tender: Technical Specifications Dossier Subject-matter of the contract Interim evaluation. Updating of the RIS3 2018-20. Available information In the website: Annual work plans & monitoring reports Specific plans Regional Specialisation Pattern. The consultant never asked for more information…

Tender: Technical Specifications Dossier Issues to be evaluated Changes in the context: National and international Performance, legislation, strategic framework… Thematic priorities: validity. Through the update of the Regional Specialisation Pattern Objectives: degree of achievement (quantitative indicators). Actions. Rhythm of implementation Effectiveness – Efficiency - Sustainability (for the next programming period)

Tender: Technical Specifications Dossier Activities to be carried out by the consultant Analysis of the new context: Socio-economic situation, R&I indicators (including Information Society) New legislative framework (including plans), at national and European level In R&I, Info Society and ESIF Analysis of the RIS3 and its suitability in the new context: Regional Specialisation Pattern Thematic priorities Objectives Programmes Governance system Monitoring & evaluation system Mobilised resources Validation with Administration and stakeholders Writing of the report. Including conclusions and recommendations Compared to the previous page, there is a small difference (for instance, the analysis of actions), but it all should be put together.

Tender: Technical Specifications Dossier Deliverables (reports) In 3 months: Analysis of new context and legislative framework Analysis of the RIS3 deployment and its suitability in the new context Forecast of: Stakeholders to be involved Information sources Evaluation methodology Detailed list of foreseen activities, specifying contents, dates and places. In 6 months: Evaluation report Regional Specialisation Pattern Very well specified, especially the part of the methodology to be used, to make sure that it was all right. Regional Specialisation Pattern: should have been in advance, in order to assess the validity of thematic priorities, but like this, it was enough for the Update

Tender: Tendering Specifications Price € 155,371.90 + taxes (€ 188,000 VAT included) Technical & professional capacity required Minimum 3-people team, with experience: Director: 5 years in R&I policies, and 3 years in Smart Specialisation 2 Seniors: 3 & 2 years, respectively Minimum € 100,000 invoiced in such activities in at least 1 of last 5 years. Tendering specifications: also called “Contract documents”

Tender: Tendering Specifications Assessment criteria Quantitative (price): 30% Not quantitative: 70% Methodology and work plan: 40% Criteria: conformity with the work’s objective, degree of development, concretisation. Applied to: work plan, methodology, actions proposed, chronogram, individual dedication, deliverables Adaptation to the regional system: 10%. Links to national and European context: 10%. Technical methodological improvements (10%), related to: Analysis of new context and Strategy’s implementation. Evaluation techniques, to determine the Strategy’s impact. Tendering specifications: also called “Contract documents” However, the deliverables are already specified.

Tender: proposal selected

Tender: proposal selected Assessment criteria Quantitative (price): too low: 105 k€ vs. 140; 2,350 hours vs. 1,600 Not quantitative: Methodology and work plan: Based on EC / JRC methodologies. Interesting approaches: sectoral value chains: weak links =challenges => flagship initiatives. Relation RIS3 – ROP. Good participative process. Very clear work plan. Adaptation to the regional system: OK Links to national and European context: OK Technical methodological improvements: Experiences from other regions: experts in the team. Plan of sustainability for the period without ESIF. Flagship initiatives happened to be pre-conceived, not derived from any analysis. Eventually, only from the ROP perspective. The contents were often too poor to organise meetings to present results Only 1 expert from Galicia; only came to 2 meetings, at the beginning, with General Directors; proposed also the former Director from the Portuguese National Innovation Agency (never showed up).

Deficiencies in the evaluation report

Deficiencies in the evaluation report Analysis of implementation Too low number of actions examined in depth (13 / 370). No effectiveness or efficiency analysis. No available information, or heterogeneous. Design & calendar of instruments. If the actions’ managers were ready to commit to it. Analysis of potentialities Analysis of competitiveness of economic macroactivities. Evolution of GDP and of exports. Identification of strong actors in each sector. Identification of emerging sectors. Analysis of labour market. Economic fabric’s absorption capacity. Identification of opportunity niches. Match between training / educational supply and market demand. They said that they had internal back-office information, but we had to ask for it. True or false? They said that it was because the information provided was insufficient. The identification of actors was out of the sectors (pre-conceived).

Deficiencies in the evaluation report Outward looking Complementarity with national programmes, and duplicities. Identification of trans-regional opportunities. From the analysis of other regional RIS3 priorities. Logic of intervention Adaptation to the new context. Effectiveness of actions undertaken. Attainment of the objectives: performance, effectiveness and efficiency. In all 3, lack of information. Conclusions Written before the rest of the document. Incoherent & contradictory. At European level, they wrote a list of the existing programmes (that was all). It could be because they had raw information, but it seemed that they were previously conceived.

Lessons learnt

Lessons learnt Risks Murphy’s Law: If it works, DON’T TOUCH IT!! The more detailed information in tender documents: Proposals more similar, and less “innovative” / interesting. Therefore, more difficult to evaluate (differentiate). Accepting potentially reckless price reductions. Flexibility may lead to significant deviation from the original proposal. Problems (difficult to foresee) Lack of initiative of the contractor: Collection of necessary information (not available initially). Different quality expectations: concretisation, field work… Anti-commercial attitude. Focus on their real skills / knowledge. Even, they used the recordings from meetings to demonstrate what we had said (probably out of context).

Lessons learnt Problems from “our” side Lack of high-level commitment-monitoring. Sometimes, contradictions between technical-managerial-political levels. Time constraints: accept “anything”. Procurement regulations: you have to accept them again, if the tender is good. Importance of monitoring as a source of relevant information. Too much control on the evaluators’ team? Other problems (difficult to avoid) Criticism towards the predecessor. The Director “sells”, and the Junior “works”.

for Science and Technology Thank you for your attention Gregorio MUÑOZ-ABAD Deputy Commissioner for Science and Technology gregorio.munoz@jcyl.es