1. Governing Board Meeting Anabel de la Peña (anaisabel.delapena@inia.es)
ARIMNet2
Governing Board Meeting
(27-28 October 2016, Rome, Italy)
Workpackage3
Anabel de la Peña
INIA (SPAIN)

2. Monitoring, follow-up and impact assessment of calls and funded
Workpackage 3
Monitoring, follow-up and impact assessment of calls and funded
research projects

3. Objectives
Analyse and evaluate the ARIMNet 1 and ARIMNet 2 (both calls)
Optimize the call procedures
Develop common guidelines for the monitoring to assess the impact of the calls
Lessons learnt (pros and cons) for conclusions looking for best options for the future

4. Tasks
Task 3.1
Evaluation of calls, calls procedures and impact assessment
INIA (M1–48)
Task 3.2
Monitoring the progress within the research projects continuously
FCT (M1–48)
Task 3.3
Assessment of research project results at mid-term and at the end (IRESA)
(IRESA (M1–48)

5. Task 3.1 Survey Questionnaires 4 questionnaires addressed to:
Call Office (CO)
Call Board (CB) (funding bodies)
Applicants (coordinators)
Evaluators (EC)
Questionnaires
Introduction: Brief explanation of ARIMNet1 call
Sections according with the phases of the call: questions with multiple answers D

6. ARIMNet 1 in brief Calendar of the call (May 2011 - February 2012)
Consortiums of 4 – 6 partners: no limits of partners
MoU signed by 14 organizations (11 countries)
3 core topics and 18 subtopics (broad scope)
86 “letter of intent” (non-compulsory), 79 proposals eligible
Eligibility: ARIMNet criteria and national eligibility
Evaluators: External referees + Evaluation Committee (EC)
2 EC members/proposal + 4 referees/proposal
2 Call Board meetings
A submission website at Call Office website
10 projects selected

7. ARIMNet call 2011 THE CALL TIMING Pre- announcement (6th May 2011)
Launching and guidelines on the web (20th June 2011)
Letter of intent (non compulsory) (12th June 2011)
Full proposals (25th September 2011)
Eligibility (26 – 28th September 2011)
1st Evaluation Committee (5th October 2011)
Scientific Peer Review (5th – 15th October 2011)
2nd Evaluation Committee (19 – 20th October 2011)
1st Call Board and funding decision (21st December 2011)
2nd Call Board (14th February 2012)
Contract negotiations (first quarter of 2012) )

8. ARIMNet Call 2011 Call Board (funding bodies) Country Organization
Algeria
INRAA
Cyprus
ARI
Egypt
ARC
France
INRA, CIRAD
Greece
DEMETER
Israel
MOARD
Italy
MIPAAF
Morocco
IAV
Spain
INIA
Tunisia
IRESA
Turkey
NAGREF

9. Questionnaires Call Office 23 questions (7 sections)
The preparation process: Focused on the process of preparing the ARIMNet Call 2011.
Application phase: Focused on impressions of the operational processes put in place by call 2011
Eligibility phase: Focused on eligibility process of proposals under the call 2011
Scientific evaluation phase: Focused on the scientific evaluation process
Selection and funding decisions: Focused on the outcomes of the funding decisions and destination of funds
Negotiation phase: Focused on the process of negotiation for national contracts
General issues: Focused on the general process of the call

10. Questionnaires Call Board 27 questions (5 sections)
General Vision: Focused on the general process of ARIMNET Call 2011
Preparatory phase: Focused on the process of preparing ARIMNET Call 2011
Application phase: Focused on the process of implementation under the ARIMNET Call 2011 for proposals
Selection phase: Focused on the process of project selection under the ARIMNET Call 2011
General strategic issues: Focused on the general strategic process of ARIMNET Call 2011

11. Questionnaires Applicants (coordinators) 15 questions (2 sections)
Call procedure: Focused on the process of applying for funding under the ARIMNET Call 2011 for proposals.
Communication: Focused on the process of communication between the applicants and the Call Office and Call Board when applying for funding under the ARIMNET Call 2011 for proposals
List of applicants : Only the coordinators of the proposals will answer this questionnaire in order to avoid imbalance between the number of respondents per project.

12. (Evaluation Committee)
Questionnaires
Evaluators
(Evaluation Committee)
13 questions (1 section)
Evaluation procedure: Focused on the process of evaluation of proposals under the ARIMNet Call 2011
This questionnaire is addressed to the Evaluation Committee and is based on the Guidelines for Evaluation. In the case of the External reviewers, other questions should have been included.

13. Task 3.1 Timeline (work done) Tasks Responsible partner contributors
2014
Preparation of different questionnaires ( call office, call board, applicants (coordinators) and evaluation committee), and circulate them between contributors
INIA
FCT, IAV
April
Circulate them to ARIMNet 2 for comments
ALL
Consolidation of the 4 questionnaires final version
Final version of the 4 questionnaires to be sent to FCT
Identification of target stakeholders by country for the implementation of the each questionnaire
May
Preparation and insertion of an online survey
FCT
On-line survey pre-test phase
INIA, IAV
Launch of the on-line survey to implement the 4 questionnaires
June
Analysis
July- September
Summary report
September-October
D3.1 Call procedure reports

14. Evaluators survey

15. Evaluators conclusions
Positive
Good internal relationship among them
Good support from Call Office
Adequate classification of proposals (most)
Relevant the scientific and technical quality of proposals (originality, methodology, etc)
Very satisfied with the global impact of the proposals (60%)
Very relevant the quality of consortium (most)
Relevant the project management and feasibility (all)
Enough time available for the evaluation (all)
Preferably physical meetings for final decisions (most)

16. Evaluators conclusions
Negative
Only 7 respondents
Not good distribution of proposals:
Most of evaluators assessed proposals on topic 1 “production systems” (80%)
A few assessed proposals on topic 2 “food chain” (20%)
Too much work: 17 proposals assessed by each expert (average)
20% judged not good the quality of the consortium
20% considered not relevant the added value to the research community

17. Mediterranean countries
Call Board survey
Mediterranean countries

18. Call Board conclusions
Positive
Suitability of the thematic research areas and topics to national priorities (all)
Adequate of information requested to applicants (most)
Distribution of proposals (4 referees per proposal) very good considered
Enough timing for letter of intent and funding decision (most)
Nomination of experts by the Call Board (most)
Funded projects as response to their organization´s policy needs (all considered)
Most considered “cash” as funding mechanism and only a few wanted “in kind”
All considered the call as extra value to Mediterranean basin

19. Call Board results Negative Poor communication with Call Office (30%)
Main difficulty: time and lack of alignment for the national contracts
Second difficulty: availability for funding in some countries
Procedure to nominate experts: long and tedious
NCPs had no evidence of proposals (letter of intent non compulsory)
A lot of complaints from applicants in the application phase
The submission tool not useful (half)
Only a ranking list
Poor scientific information from experts to Call Board
Dissemination results not enough

20. Mediterranean countries
Applicants survey
Mediterranean countries

21. Applicants conclusions
Positive
Enough time to prepare the proposal (most)
Easily to set up the consortium (half)
The electronic submission tool good assessed
The ARIMNet website well considered
Support from NCPs really well appreciated (most)
Timing for different steps: good/poor (fifty/fifty)

22. Applicants conclusions
Negative
Main difficult issue: financial aspects (most)
Promotion of the call thorough national websites: useless (more than the half)
Notification letters to coordinators too generic (a quarter)
Letter of intent not clear (half)

23. Timing for different steps
Call Office survey
Timing for different steps

24. Call Office conclusions
Positive
Pre-announcement good and clear
LoI useful to indicate number of proposals
Good communication with applicants, evaluators
Call Office solved all the difficulties (in general)
Inputs from Call Office in different phases (application and evaluation): very important

25. Call Office conclusions
Negative
Desirable more experience in the preparatory phase
Call documents more clear:
national eligibility criteria and prioritization,
identity of applicants (natural person or legal entity)
role of the experts (write a report per proposal)
Searching of referees: time consuming
Lack of alignment in contract signatures
Difficult communication with some funders

26. Lessons learnt for future calls
FUNDING DECISIONS: consideration of scientific evaluation, much more than national priorities
TOPICS: restricted and specific research topics
CONSORTIA: limited number of partners (minimum and maximum)
DISSEMINATION OF RESULTS: improved and shared with stakeholders
CALL STEPS: Two (PP and FP) for avoiding big number of proposals
TIMELINE: more time for eligibility and evaluation

27. Lessons learnt for future calls
EVALUATION:
Guidelines for referees/experts with clear instructions about the content of the evaluation report
Distribution of reviews from referees to experts in advance, for objections and/or comments
REFEREES: Searching them in advance
NATIONAL CONTRACTS: alignment for signature
NEGOTIATON PHASE: foresee fall back procedures in the MoU

28. ARIMNet2 call 1 THE CALL TIMING Pre- announcement (30th June 2014)
Launching and guidelines on the web (15th September 2014)
Pre-proposals(1st December 2014)
Full proposals (11th May 2015)
Decision for funding (30th October 2015)
Contract negotiations ( From November 2015)
Start of the projects (January – April 2016) )

29. ARIMNet2 Call 1 Timeline (work done) Tasks Responsible partner
Contributors
2016
Identification of target stakeholders by country for the implementation of the each questionnaire
INIA
ALL
May
Preparation and insertion of an online survey
FCT
September
On-line survey pre-test phase
Launch of the on-line survey to implement the 4 questionnaires
9th September
Analysis
October-November
Summary report
December
D3.1 Call procedure reports

30. Thanks for your attention
Anabel de la Peña
Thanks for your attention