Governing Board Meeting Anabel de la Peña (anaisabel.delapena@inia.es) ARIMNet2 Governing Board Meeting (27-28 October 2016, Rome, Italy) Workpackage3 Anabel de la Peña (anaisabel.delapena@inia.es) INIA (SPAIN)

Monitoring, follow-up and impact assessment of calls and funded Workpackage 3 Monitoring, follow-up and impact assessment of calls and funded research projects

Objectives Analyse and evaluate the ARIMNet 1 and ARIMNet 2 (both calls) Optimize the call procedures Develop common guidelines for the monitoring to assess the impact of the calls Lessons learnt (pros and cons) for conclusions looking for best options for the future

Tasks Task 3.1 Evaluation of calls, calls procedures and impact assessment INIA (M1–48) Task 3.2 Monitoring the progress within the research projects continuously FCT (M1–48) Task 3.3 Assessment of research project results at mid-term and at the end (IRESA) (IRESA (M1–48)

Task 3.1 Survey Questionnaires 4 questionnaires addressed to: Call Office (CO) Call Board (CB) (funding bodies) Applicants (coordinators) Evaluators (EC) Questionnaires Introduction: Brief explanation of ARIMNet1 call Sections according with the phases of the call: questions with multiple answers D

ARIMNet 1 in brief Calendar of the call (May 2011 - February 2012) Consortiums of 4 – 6 partners: no limits of partners MoU signed by 14 organizations (11 countries) 3 core topics and 18 subtopics (broad scope) 86 “letter of intent” (non-compulsory), 79 proposals eligible Eligibility: ARIMNet criteria and national eligibility Evaluators: External referees + Evaluation Committee (EC) 2 EC members/proposal + 4 referees/proposal 2 Call Board meetings A submission website at Call Office website 10 projects selected

ARIMNet call 2011 THE CALL TIMING Pre- announcement (6th May 2011) Launching and guidelines on the web (20th June 2011) Letter of intent (non compulsory) (12th June 2011) Full proposals (25th September 2011) Eligibility (26 – 28th September 2011) 1st Evaluation Committee (5th October 2011) Scientific Peer Review (5th – 15th October 2011) 2nd Evaluation Committee (19 – 20th October 2011) 1st Call Board and funding decision (21st December 2011) 2nd Call Board (14th February 2012) Contract negotiations (first quarter of 2012) )

ARIMNet Call 2011 Call Board (funding bodies) Country Organization Algeria INRAA Cyprus ARI Egypt ARC France INRA, CIRAD Greece DEMETER Israel MOARD Italy MIPAAF Morocco IAV Spain INIA Tunisia IRESA Turkey NAGREF

Questionnaires Call Office 23 questions (7 sections) The preparation process: Focused on the process of preparing the ARIMNet Call 2011. Application phase: Focused on impressions of the operational processes put in place by call 2011 Eligibility phase: Focused on eligibility process of proposals under the call 2011 Scientific evaluation phase: Focused on the scientific evaluation process Selection and funding decisions: Focused on the outcomes of the funding decisions and destination of funds Negotiation phase: Focused on the process of negotiation for national contracts General issues: Focused on the general process of the call

Questionnaires Call Board 27 questions (5 sections) General Vision: Focused on the general process of ARIMNET Call 2011 Preparatory phase: Focused on the process of preparing ARIMNET Call 2011 Application phase: Focused on the process of implementation under the ARIMNET Call 2011 for proposals Selection phase: Focused on the process of project selection under the ARIMNET Call 2011 General strategic issues: Focused on the general strategic process of ARIMNET Call 2011

Questionnaires Applicants (coordinators) 15 questions (2 sections) Call procedure: Focused on the process of applying for funding under the ARIMNET Call 2011 for proposals. Communication: Focused on the process of communication between the applicants and the Call Office and Call Board when applying for funding under the ARIMNET Call 2011 for proposals List of applicants : Only the coordinators of the proposals will answer this questionnaire in order to avoid imbalance between the number of respondents per project.

(Evaluation Committee) Questionnaires Evaluators (Evaluation Committee) 13 questions (1 section) Evaluation procedure: Focused on the process of evaluation of proposals under the ARIMNet Call 2011 This questionnaire is addressed to the Evaluation Committee and is based on the Guidelines for Evaluation. In the case of the External reviewers, other questions should have been included.

Task 3.1 Timeline (work done) Tasks Responsible partner contributors 2014 Preparation of different questionnaires ( call office, call board, applicants (coordinators) and evaluation committee), and circulate them between contributors   INIA FCT, IAV April Circulate them to ARIMNet 2 for comments ALL Consolidation of the 4 questionnaires final version Final version of the 4 questionnaires to be sent to FCT Identification of target stakeholders by country for the implementation of the each questionnaire May Preparation and insertion of an online survey FCT On-line survey pre-test phase INIA, IAV Launch of the on-line survey to implement the 4 questionnaires June Analysis July- September Summary report September-October D3.1 Call procedure reports

Evaluators survey

Evaluators conclusions Positive Good internal relationship among them Good support from Call Office Adequate classification of proposals (most) Relevant the scientific and technical quality of proposals (originality, methodology, etc) Very satisfied with the global impact of the proposals (60%) Very relevant the quality of consortium (most) Relevant the project management and feasibility (all) Enough time available for the evaluation (all) Preferably physical meetings for final decisions (most)

Evaluators conclusions Negative Only 7 respondents Not good distribution of proposals: Most of evaluators assessed proposals on topic 1 “production systems” (80%) A few assessed proposals on topic 2 “food chain” (20%) Too much work: 17 proposals assessed by each expert (average) 20% judged not good the quality of the consortium 20% considered not relevant the added value to the research community

Mediterranean countries Call Board survey Mediterranean countries

Call Board conclusions Positive Suitability of the thematic research areas and topics to national priorities (all) Adequate of information requested to applicants (most) Distribution of proposals (4 referees per proposal) very good considered Enough timing for letter of intent and funding decision (most) Nomination of experts by the Call Board (most) Funded projects as response to their organization´s policy needs (all considered) Most considered “cash” as funding mechanism and only a few wanted “in kind” All considered the call as extra value to Mediterranean basin

Call Board results Negative Poor communication with Call Office (30%) Main difficulty: time and lack of alignment for the national contracts Second difficulty: availability for funding in some countries Procedure to nominate experts: long and tedious NCPs had no evidence of proposals (letter of intent non compulsory) A lot of complaints from applicants in the application phase The submission tool not useful (half) Only a ranking list Poor scientific information from experts to Call Board Dissemination results not enough

Mediterranean countries Applicants survey Mediterranean countries

Applicants conclusions Positive Enough time to prepare the proposal (most) Easily to set up the consortium (half) The electronic submission tool good assessed The ARIMNet website well considered Support from NCPs really well appreciated (most) Timing for different steps: good/poor (fifty/fifty)

Applicants conclusions Negative Main difficult issue: financial aspects (most) Promotion of the call thorough national websites: useless (more than the half) Notification letters to coordinators too generic (a quarter) Letter of intent not clear (half)

Timing for different steps Call Office survey Timing for different steps

Call Office conclusions Positive Pre-announcement good and clear LoI useful to indicate number of proposals Good communication with applicants, evaluators Call Office solved all the difficulties (in general) Inputs from Call Office in different phases (application and evaluation): very important

Call Office conclusions Negative Desirable more experience in the preparatory phase Call documents more clear: national eligibility criteria and prioritization, identity of applicants (natural person or legal entity) role of the experts (write a report per proposal) Searching of referees: time consuming Lack of alignment in contract signatures Difficult communication with some funders

Lessons learnt for future calls FUNDING DECISIONS: consideration of scientific evaluation, much more than national priorities TOPICS: restricted and specific research topics CONSORTIA: limited number of partners (minimum and maximum) DISSEMINATION OF RESULTS: improved and shared with stakeholders CALL STEPS: Two (PP and FP) for avoiding big number of proposals TIMELINE: more time for eligibility and evaluation

Lessons learnt for future calls EVALUATION: Guidelines for referees/experts with clear instructions about the content of the evaluation report Distribution of reviews from referees to experts in advance, for objections and/or comments REFEREES: Searching them in advance NATIONAL CONTRACTS: alignment for signature NEGOTIATON PHASE: foresee fall back procedures in the MoU

ARIMNet2 call 1 THE CALL TIMING Pre- announcement (30th June 2014) Launching and guidelines on the web (15th September 2014) Pre-proposals(1st December 2014) Full proposals (11th May 2015) Decision for funding (30th October 2015) Contract negotiations ( From November 2015) Start of the projects (January – April 2016) )

ARIMNet2 Call 1 Timeline (work done) Tasks Responsible partner Contributors 2016 Identification of target stakeholders by country for the implementation of the each questionnaire   INIA ALL May Preparation and insertion of an online survey FCT September On-line survey pre-test phase Launch of the on-line survey to implement the 4 questionnaires 9th September Analysis October-November Summary report December D3.1 Call procedure reports

Thanks for your attention Anabel de la Peña anaisabel.delapena@inia.es Thanks for your attention