The Evaluation Process, Tips from an Evaluator’s Point of View Leonardo Piccinetti EFB

OUTLINE Evaluation principles The experts, who are they ? Role of Commission staff FP7 evaluation process FP7 Evaluation Criteria

Basic principles Excellence. Projects selected for funding must demonstrate a high quality in the context of the topics and criteria set out in the calls. Transparency. Funding decisions are based on clearly described rules and procedures, and applicants should receive adequate feedback on the outcome of the evaluation of their proposals. Impartiality. All proposals submitted to a call are treated equally. They are evaluated impartially on their merits, irrespective of their origin or the identity of the applicants.

Basic principles Confidentiality. All proposals and related data, knowledge and documents communicated to the Commission are treated in confidence. Efficiency and speed. Evaluation, award and grant preparation should be as rapid as possible, commensurate with maintaining the quality of the evaluation, and respecting the legal framework. Ethical and security considerations: Any proposal which contravenes fundamental ethical principles, or which fails to comply with the relevant security procedures may be excluded at any time from the process of evaluation, selection and award

The experts, who are they ? The Commission draws on a wide pool of evaluators (database) in all scientific fields c. 50,000 in FP6 Experts/evaluators from ICPC are very welcome! Calls for “candidates” Call for applications from individuals; and from institutions Applications via CORDIS (database of experts) A mass-emailing of FP6 experts was sent A simple tick-box will ensure registration for FP7 Commission invites individuals on a call-by-call basis Not self-selection! Expertise, and experience are paramount Geography, gender and “rotation” also considered

Independent experts Expert evaluators are at the heart of the FP7 system Expert provides independent, impartial and objective advice to the Commission represents neither the employer, nor the country! Significant funding decisions will be made on the basis of expert advice The integrity of the process is crucial Experts have to read the Code of Conduct annexed to the appointment letter…and follow it!

Proposal submission and evaluation 27/03/2017 Independent experts Experts agree to terms and conditions of an “appointment letter” Typically, an individual will review 6-8 proposals “remotely”…. …then spend a couple of days in Brussels Some will participate in “hearings” with the consortia Travel and subsistence reimbursed Plus €450 honorarium per day Experts sign confidentiality and conflict of interest declaration Names published after the evaluations Presentation IP - Heysel

Actors Confidentiality The content of proposals, or the evaluation results, can’t be discussed with anyone The sole exception: in the presence of the EC moderator with experts who are evaluating the same proposal in a consensus meeting group or final panel Is not possible to distribute any documents related to the evaluation of a proposal, or take any documents from the evaluation building Note: The Commission publishes names annually, but as a group – no link between expert and proposal

Conflicts of interest (2) Types of COI set out in appointment letter Check the exact wording! Disqualifying COI Involved in preparation of proposal Stands to benefit directly Close family relationship Director/trustee/partner Employee of a partner in a proposal Member of Advisory Group Any other situation that compromises impartiality Potential COI Employed within the last 3 years by a partner in a proposal Involved in research collaboration with proposers in the previous 3 years Any other situation that casts doubt…or that could reasonably appear to do so…

Role of Commission staff Check the eligibility of the proposals Oversee work of experts Moderate discussions Organise the panel and its work Ensure coherence and consistency

Evaluation Peer-Review System Two-stage evaluation procedure Remote evaluation Evaluation on a non-anonymous basis Unless otherwise specified in call for proposal Register as an Evaluator https://cordis.europa.eu/emmfp7/

Role of Commission staff Commission staff may advise on: Background on previously supported or on-going projects Relevant supplementary information (directives, regulations, policies, etc) Evaluation rules Key points within the Work Programme, e.g. issues related to “Relevance” Commission staff may not introduce: New elements (cannot fill in “gaps” in proposals) Interpretations

Writing an FP7 proposal is NOT just a creative process for A NICE IDEA It requires to SHOW scientific, technological and depth knowledge of the subject You must present references, legislatures, previous work and experience

Be precise and to the point Use drawings to show your methodology at once Use bold types in phrases that you would like to emphasize Present cohesion and interaction among work packages

Overview of the Evaluation Process “remote” may be “remote” Submission Individual reading Consensus Panel Finalisation Full Proposal Proposal forms Final ranking list Evaluators Evaluators Evaluators Rejection list Criteria Criteria Criteria Proposals in suggested priority order Eligibility COMMISSION COMMISSION Role of experts

FP7 evaluation process IAR 1 IAR 2 IAR 3 CONSENSUS MEETING RECOMMENDANTION FOR FUNDING OR NOT IAR 2 CONSENSUS REPORT BY THE RAPORTER IAR 3 PANNEL MEETING – RESULTS ARE SEND TO APPLICANT EVERY IAR IS 3 TO 4 HOURS 0 - the proposal fails to address the issue under examination or can not be judged against the criterion due to missing or incomplete information 1 - poor 2 - fair 3 - good 4 - very good 5 - excellent

QUALITY OF COORDINATION RELEVANCE QUALITY OF THE CONSORTIUM 1. CONCEPT – CAPABILITIES OF THE PARTNERS QUALITY OF COORDINATION QUALITY OF MANAGEMENT MOBILISATION OF RESOURCES 2. ACTIVITIES – AND CAPACITY TO DO THE TASK POTENTIAL IMPACT 3. THE TARGET GROUPS THAT BENEFIT

Process Evaluation - Individual reading (Will be done remotely) The experts: Evaluate the proposal individually without discussing with the other evaluators Check whether the proposal is ‘in scope’ Complete an Individual Evaluation Report (IER) form giving comments and scores on all criteria Sign and date the form IERs should be checked and, if necessary, returned with a request to further justify the score given Scores must be in line with comments This stage can also take place remotely (at home/ workplace of the expert) …

Process Evaluation - Consensus (1) Built on the basis of the individual evaluations The aim is agreement on scores and comments Usually involves a discussion 1st part may be carried out remotely “Outlying” opinions need to be explored Not just a simple averaging exercise It is quite normal for individual views to change Moderated by a Commission staff-member helps the group reach a conclusion provides information if necessary does not contribute opinions

Process Evaluation - Consensus (2) A rapporteur is appointed, who is responsible for drafting the consensus report (CR) includes consensus marks and comments The quality of the CR is paramount It is not often changed at the panel review stage The aim is: a clear assessment of the proposal, with justification clear feedback on weaknesses & strengths To be avoided: scores that don’t correspond with the comments recommendations in view of resubmission

The Panel Review EC ask some evaluators in each sub-activity to examine and compare the CRs of every proposal that passes all thresholds Key function is to ensure consistency The Panel will recommend for a sub-activity a priority order including final marks and comments for each proposal Evaluation Summary Reports (ESR) Any new scores (if necessary) … should be carefully justified Ranking of proposals with identical consensus scores Prioritise certain criteria? Consider overall balance? Budget? Clear guidance for contract negotiation

Process Commission Follow-up Evaluation summary reports sent to applicants “initial information letter” Redress procedure Draw up final ranking lists Information to the Programme Committee Contract negotiation Formal consultation of Programme Committee (when required) Commission decisions Survey of evaluators Independent Observers’ reports New for FP7

The evaluation criteria Criteria adapted to each funding scheme and each thematic area specified in the work programme Three main criteria: S&T Quality (relevant to the topic of the call) Concept, objective, work-plan Implementation Individual participants and consortium as a whole Allocation of resources Impact Contribution to expected impacts listed in work programme Plans for dissemination/exploitation

Process FP7 Evaluation Criteria Applicable to ALL funding schemes 1. S/T quality (in relation to the topics addressed by the call) 2. Implementation 3. Impact Sound concept, and quality of objectives Appropriateness of the management structure and procedures Quality and relevant experience of the individual participants Contribution, at the European and / or international level, to the expected impacts listed in the work programme under the relevant topic / activity

Process Funding schemes Collaborative projects Support to research projects carried out by consortia with participants from different countries, aiming at developing new knowledge, new technology, products, demonstration activities or common resources for research. The size, scope and internal organisation of projects can vary from field to field and from topic to topic. Projects can range from small or medium-scale focused research actions to large-scale integrating projects for achieving a defined objective Projects may also be targeted to special groups such as SMEs, Specific International Co-operation Actions, etc. Template of the IER for your information (double click on the object) Guidance for evaluators – Call FP7-ENV-2008-1

Process FP7 Evaluation Criteria Collaborative projects 1. S/T quality (relevant to the topics addressed by the call) 2. Implementation 3. Impact Prgress beyond the state-of-the-art Quality and effectiveness of the S/T methodology and associated work plan Quality of the consortium as a whole (incl. complementarity, balance) Appropriate allocation and justification of the resources to be committed (budget, staff, equipment) Appropriateness of measures for the dissemination and/or exploitation of projects results, and management of intellectual property.

Template of the IERs for your information (double click on the object) Process Funding schemes Coordination & support actions Support to activities aimed at coordinating or supporting research activities and policies (networking, exchanges, trans-national access to research infrastructures, studies, conferences, etc). same scope and objectives as in FP6 Template of the IERs for your information (double click on the object) Guidance for evaluators – Call FP7-ENV-2008-1

Coordination & support actions 1. S/T quality (in relation to the topics addressed by the call) 2. Implementation 3. Impact CA • Contribution to the co-ordination of high quality research • Quality and effectiveness of the co-ordination mechanisms, and associated work plan Quality of the consortium as a whole (including complementarity and balance) Appropriate allocation and justification of the resources to be committed (budget, staff, equipment) Appropriateness of measures for spreading excellence, exploiting results, and dissemination knowledge, through engagement with stakeholders, and the public at large. SA • Quality and effectiveness of the support action mechanisms, and associated work plan

RELEVANCE Examines if the objectives of the work programme are met The proposer must read in between lines It must be clear what the commission wants from the call Good practice: attend info day, inside information Covering the objectives u get a 4, additional objectives might give u a 5 TIPS: Do not copy paste the objectives from the work programme U must copy and specialize them to the concept of the proposal FIND THE KEY WORDS (ie networking, knowledge transfer) relating the key word to the objective TRY TO QUANTIFY OBJECTIVES ( i.E. Not just networking – but networking of three clusters) Always make a direct reference under each objective with the related work packages of how this objective will be achieved

Proposal relevance tips Identification of impact from Call Definition of strategic goals for our proposal which clearly fulfill the expected. Definition of main actions/activities clearly related to the strategic goals Work Packages for actions/activities.

QUALITY OF CONSORTIUM Examines the experience of the consortium in relation to the objectives and the work need to be done, complementarities between partners, suitability of work undertaken, and geographical coverage TIPS Always include a description in the beginning of consortium description indicating the role of each partner and their added value to the project. It is preferred to have inside the consortium mix Member States, New member states and Associated states indicating transfer of good practices to new member and associated states. Under each partner description indicating the excellences (previous projects, research activities and experiences) and of the partner related to the proposal followed by the CVs of key persons involved and their role in the project These excellences should be summarized into an excellence key areas of the consortium in relation to the work need to be done Indicate the logic behind consortium geographical and context selection (why these partners are selected)

The coordinator There is no limitation related to the size, nature, legal status and the years of existence for the proposed coordinator. In the management forms (A1, A2 and A3) the turnover is presented. The is a indirect but very important effect to the decision of the Commission.

QUALITY OF COORDINATION Examines coherency of the proposal. If there is a methodology that defines the interrelationships between the work packages and the work tasks The plan must convince the evaluators that is a bottom down approach from the implementation plan to the work packages TIPS Start with the relation of the objectives to the work need. Define your methodology of work mentioning entities (not yet work packages) Define the interrelationships of the WPs and the work tasks inside the work packages Your methodology should have a clear description of impact creation in the target groups and European Added Value through the actions of the dissemination plan Do not include management packages, just coordination packages this would be explained in another section. YOU MUST ALWAYS START WITH THE METHODOLOGY OF WORK AND THEN WRITE THE WORKPACKAGES Work packages should mirror the work in the implementation plan with CLEAR milestones and derivables In general you could exceed the number of pages asked from the commission

General tips Every text with more than three paragraphs must begin with a 2-3 lines abstract with a clear reference to what the remaining text presents. In most cases the opinion of the evaluator is coming from this abstract. Phrases 8 words, paragraphs max 3-4 lines.

QUALITY OF MANAGEMENT Examines the experience and management mechanisms of the management unit The evaluator examines the capacity of the coordinator to handle the tasks They like to have a strong centralized management and not very complex structures They like to see decision making mechanisms from steering committees TIPS Always start indicating the central management control indicating the experience of the coordinator to undertake the tasks in similar projects Illustrate decision making mechanism (simple but partner inclusive) Show contingency plan with risks (i.e. what happens if partner withdraws) Show IPR management Show quality assurance mechanisms Show knowledge management with in partners. For each of the above relate them with management instruments (web sites, consortium agreements, meetings, etc)

MOBILISATION OF RESOURCES Examines the coherence of the financial plan to handle the tasks, relating to the allocation of work in PM within work packages and among partners. The financial plan must be delivered with the proposal although there are no predefined forms from that (except A3, deliverable lists) TIPS ALWAYS state the CVs of key persons involved with the project and their role Illustrate other resources for each organization that are allocated to the project in the partnership description (ie equipment, previous work done etc) Subcontracting is always examined in detail (why and who will do it?) Avoid large differences in work allocation between partners If a partner gets more than 30% of the budget (management and coordination) is negative Management no more than 7%

POTENTIAL IMPACT Examines the impact (results) to target group defined, and the European Added value and mainly your dissemination plan Relates to the previous work done in European and National Level TIPS Always start with the clear definition of the proposal target groups indicating the potential impact in quantitative terms if possible Use structured bulleted writing and not abstract descriptions The dissemination plan should provide evidence of European added value in quantitative terms (not general instruments i.e. web page but exclusive actions such as connection with specific initiatives, work groups etc) Always illustrate impact in relation to previous work done – specific national and European programmes indicating the approach mechanisms (even it is obvious) The main evaluation criteria is the dissemination plan. The above issues should be described as work to be done in the dissemination work package

ABSTRACT Should start with objectives followed by the consortium logic Then should describe the implementation plan with the expected results and finish with the impact through the dissemination plan

Self-assessment process Most of failed proposals are taking a low mark in 1-2 evaluation criteria (relevance is the most usual). The pre-evaluation of the proposal by internal or external executives (definitely not involved in the proposal preparation process) based on the published evaluation criteria.

Conclusion Become evaluator is the best way to learning to write proposals Understanding how EC works Networking Well paid

Thank you! Leonardo Piccinetti