Chisinau, November 6th, 2012 Dr Sebastiano FUMERO

Slides:



Advertisements
Similar presentations
ENTITIES FOR A UN SYSTEM EVALUATION FRAMEWORK 17th MEETING OF SENIOR FELLOWSHIP OFFICERS OF THE UNITED NATIONS SYSTEM AND HOST COUNTRY AGENCIES BY DAVIDE.
Advertisements

The Evaluation Process, Tips from an Evaluator’s Point of View
TEN-T Info Day for AP and MAP Calls 2012 EVALUATION PROCESS AND AWARD CRITERIA Anna Livieratou-Toll TEN-T Executive Agency Senior Policy & Programme Coordinator.
1 17/3/2009 European Commission Directorate General Information Society & Media Briefing for Remote Reading How to fill in the (IER) Individual Evaluation.
European R&D Support Programme ACCESSING EUROPEAN FUNDING FOR RESEARCH AND DEVELOPMENT.
Integrating the gender aspects in research and promoting the participation of women in Life Sciences, Genomics and Biotechnology for Health.
Improving how your organisation supports the use of research evidence to inform policymaking.
Page 1 Marie Curie Schemes Science is not the whole story! (How to write a successful Marie Curie RTN Proposal) Siobhan Harkin.
1 The FP7 Framework Programme “ERC (IDEAS)” Ayala Karniol ISERD.
FP7 redress Alan Cross Programme Committee: specific configuration Brussels, 29 November 2007.
Research and Innovation Summary of MS questions on the Commission's proposal for DG Research & Innovation Research and Innovation Rules for Participation.
1-1 PRESENTER The Role of the Framework 7 Advisor Your Name Your Websites Websites
1 CORDIS and FP7 5 steps to succeed 2 Do you need to know: When and what FP7 funding is available? Which are the first FP7 integrated projects (and who.
RELATIONSHIP BETWEEN THE MANAGING AUTHORITIES AND THE PAYING AGENCIES IN THE MANAGEMENT OF RURAL DEVELOPMENT PROGRAMMES Felix Lozano, Head of.
DR MACIEJ JUNKIERT PRACOWNIA BADAŃ NAD TRADYCJĄ EUROPEJSKĄ Guide for Applicants.
Participation Requirements for a Guideline Panel PGIN Representative.
Horizon 2020 Energy Efficiency Information Day 12 December 2014 Essentials on how to submit a good proposal EASME Project Advisors: Francesca Harris,
R.König / FFG, European and International Programmes (EIP)Page 1/18 Submission and Evaluation of Proposals Ralf König FFG - Austrian Research Promotion.
Provisional draft The ICT Theme in FP7 Submission and Evaluation (preliminary information) ICT-NCP Information Day 19 th October 2006.
UEL Guidelines for External Examiners Philip Brimson Quality Manager (Validation & Review)
How experts evaluate projects; key factors for a successful proposal
Culture Programme - Selection procedure Katharina Riediger Infoday Praha 10/06/2010.
Provisional FP7-ICT InfoDay, Torino, 11/12/ The ICT Theme in FP7 How to submit a proposal 3. Submission and selection.
Proposal evaluation process in FP7 Moldova – Research Horizon 29 January 2013 Kristin Kraav.
Info Day on New Calls and Partner Café Brussels, 10 February 2011 How to apply: Legal Framework – Beneficiaries – Application and Selection Procedure.
APRE Agency for the Promotion of European Research Lifecycle of an FP 7 project Caterina Buonocore Riga, 13th September, 2007.
FAO/WHO Codex Training Package Module 3.2 FAO/WHO CODEX TRAINING PACKAGE SECTION THREE – BASICS OF NATIONAL CODEX ACTIVITIES 3.2 How to develop national.
Technology Strategy Board Driving Innovation Participation in Framework Programme 7 Octavio Pernas, UK NCP for Health (Industry) 11 th April 2012.
IST programme 1 IST KA3: The Evaluation Introduction & Contents Principles Outline procedures Criteria and Assessment What this means for proposers.
UEL Guidelines for External Examiners Philip Brimson Quality Manager (Validation & Review)
TEN-T Experts Briefing, March Annual Call Award Criteria.
HORIZON 2020 European Commission Research and Innovation First stakeholder workshop on Horizon 2020 Implementation Brussels, 16 January 2015.
Dr. Marion Tobler, NCP Environment Evaluation Criteria and Procedure.
ERC - Advance Grant Call 2008 Pilar Lopez S2 Unit Ideas Programme Management Athens, 11 April 2008.
Being evaluators : what benefit and experience Leonardo Piccinetti EFB Ltd FP7 training Tirana, 06 October 2009.
Citizens and Governance in a Knowledge-based Society Guidelines on Proposals Presented by Henry Scott, EKT.
External examiner induction Alison Coates QA Manager (Validation & Review)
SELECTION PROCEDURE Clivio CASALI, Project Officer EM ECW Erasmus Mundus and External Cooperation Call for Proposals for mobility activities starting in.
The Role of Teaching Assistants. Session outline The Workshop includes four elements: (1)Roles and Responsibilities of Staff (2)Establishing a Professional.
Writing the Proposal: Scientific and technological objectives PHOENIX Training Course Laulasmaa, Estonia
19 November ITN = Initial Training Networks ITN is an FP7 Marie Curie “host-driven” action. European Commission has issued a new call (Call 3) for.
The partnership principle and the European Code of Conduct on Partnership.
ICT Programme Operations Unit Information and Communications Technologies How to fill in the IER form ICT Calls 2013.
Guidelines for the Organization of Practical Placements for Students (PPS) Code of Practice for Actors Gregory Makrides – European Association of Erasmus.
Marie Curie Initial Training Networks (ITN) Building knowledge about evaluation process and criteria into own proposal.
TEN-T Executive Agency and Project Management Anna LIVIERATOU-TOLL TEN-T Executive Agency Senior Programme and Policy Coordinator European Economic and.
How is a grant reviewed? Prepared by Professor Bob Bortolussi, Dalhousie University
Negotiation of Proposals Dr. Evangelos Ouzounis Directorate C DG Information Society European Commission.
Information Overview SF: Planning & Programming Workshops for EC Delegation Patrick Colgan & Ján Krištín PROGRAMMING PROCEDURES in Support of Regional.
Evaluation of proposals Alan Cross European Commission.
1 Framework Programme 7 Evaluation Criteria. 2 Proposal Eligibility Evaluation by Experts Commission ranking Ethical Review (if needed) Commission rejection.
Session 3 – Evaluation process Viera Kerpanova, Miguel Romero.
Horizon 2020 Overview Jerome de Barros NCP Health.
Date: in 12 pts Marie Skłodowska-Curie Actions Award criteria Education and Culture Policy Officers DG EAC.C3 People NCPs Training on H2020, Brussels,
Practical Aspects of participation in FP7 Tania Friederichs DG RTD International Cooperation FP7 Info Day Sarajevo, 23 April.
RCUK International Funding Name Job title Research Councils UK.
EuropeAid/152087/DD/ACT/LB Cultural activities 2016 Information Session 13 th June 2016.
UEL Guidelines for External Examiners
EIA approval process, Management plan and Monitoring
Marie Curie Individual Fellowships
What is a grant? A direct financial contribution – donation – from EU budget An action - contributing to EU policy achievement Functioning of a body acting.
Pentalateral Energy Forum & European Commission Meeting
Business environment in the EU Prepared by Dr. Endre Domonkos (PhD)
Evaluation processes Horizon 2020 Info Days November 2017
Helene Skikos DG Education and Culture
The evaluation process
Proposal Preparation &
Key steps of the evaluation process
2012 Annual Call Steps of the evaluation of proposals, role of the experts TEN-T Experts Briefing, March 2013.
Presentation transcript:

The evaluation process in the 7th Framework programme for Research and Technological Development Chisinau, November 6th, 2012 Dr Sebastiano FUMERO Head of Unit “FP7 support” European Commission Research Executive Agency

Overview How to apply The evaluation process: basic facts and figures Role of Commission/REA staff Key issues: Eligibility check Expert selection Conflicts of interest The criteria The observer Redress

FP7: how to apply Participant portal Calls for proposals Annual Work Budget, deadline, OJ ref., legal documents Links to SEP NCP Annual Work Programme Year N Guide for Applicants Call X Funding Scheme Y SEP Electronic proposal submission system

The Participant Portal (PP)

The Participant Portal (PP)

The Participant Portal (PP)

To consider prior to submission Rules on submission and evaluation The common reference for FP7 Consistency vs flexibility! Guide for applicants (annexes 1 and 2) The work programme The topics and criteria against which the proposals will be judged (all criteria are important – consider sub-criteria - think as an evaluator)

Submission Drafting the Proposal Respect page limits specified in guide for applicants Ensure you meet the minimum eligibility requirements Excellent science is a condition but not enough. Consider also: - impact, dissemination and IPS - consider project implementation and management (role of coordinator is essential) Be precise, less is sometime more… Impartial view…ask your colleagues, friend to read it before… Start with SEP asap – a missed deadline implies proposal is not admissible administrative data (part A forms) should be consistent with info in part B

Evaluation process Submission Individual evaluation Consensus Panel May be “remote” May be “remote” Submission Individual evaluation Consensus Panel Finalisation Full Proposal Proposal forms Final ranking list Evaluators Evaluators Evaluators Rejection list Criteria Criteria Criteria Proposals in suggested priority order Eligibility COMMISSION COMMISSION Experts' role

Applicants are informed Evaluation process Proposal Individual Evaluation Consensus Panel review Hearings For large projects (optional) Thresholds Eligibility Negotiation Commission ranking Commission rejection decision Remote or in Bruxelles Quick Information Letter Applicants are informed of the Commission decision Commission funding decision

Basic facts and figures Evaluation of proposals Basic facts and figures Funding decisions are based on peer review of research proposals Peer review can also “add value” to projects High quality evaluators are at the core of the system Over 118.000 experts registered for FP7 in the old database Over 15.000 experts registered in the new database (Expert Area in the Participant Portal) Over 8 300 independent experts engaged in 2011 Approx. 27 000 proposals evaluated in 2011

Evaluation Process: basic principles Evaluation of proposals Evaluation Process: basic principles EXCELLENCE TRANSPARENCY FAIRNESS & IMPARTIALITY CONFIDENTIALITY ETHICAL & SECURITY CONSIDERATIONS EFFICIENCY & SPEED

Three References Evaluation of proposals RULES Submission & Evaluation Rules on submission and evaluation The common reference for FP7 Consistency vs flexibility! Guide for applicants (annexes 1 and 2) The work programme The topics and criteria against which the proposals will be judged Guide for Applicants Call X Funding Scheme Y Annual Work Programme

Some basic misconceptions clarified Evaluation of proposals Some basic misconceptions clarified The EU’s peer review system is not a political process Lobbying has no influence Quality of the proposal is the sole criterion for success However, “quality” involves a number of factors

Role of Commission/REA staff Evaluation of proposals Role of Commission/REA staff Check the eligibility of proposals Oversee work of experts Conduct briefings Moderate discussions Organise the panel and its work Ensure coherence and consistency May advise on: Background on previously supported or on-going projects Relevant supplementary information (directives, regulations, policies, etc.) (Can even act as experts!)

Eligibility checks Evaluation of proposals Receipt of proposal before deadline Firm deadlines (SEP) Minimum number of eligible, independent partners As set out in work programme and the call Completeness of proposal Presence of all requested forms and readable, accessible and printable "In scope" vs "Out of scope" Others

Expert selection Evaluation of proposals Based on: A high level of expertise An appropriate range of competences If the above conditions can be satisfied, then also: Balance academic/industrial Gender Geography Rotation But also, of course constrained by: Availability Avoidance of conflicts of interest Uncertainty over number and exact coverage of proposals Not an easy process…!!!

FP7 Expert from this area Number of experts having supported the EC in FP7 by country Country   Number of Experts (participation) Greece EL 646 Romania RO 365 Hungary HU 311 Bulgaria BG 165 Slovenia SI 163 Slovakia SK 95 Croatia HR 76 Serbia RS 62 Macedonia MK 8 Bosnia and Herzegovina BA 5 Moldova MD 3 Number of experts registered in the EMPP by country Country   Migrated Valid Bosnia and Herzegovina BA 8 Bulgaria BG 204 162 Croatia HU 115 89 Greece HR 914 687 Hungary 297 Macedonia MK 25 21 Moldova MD 16 12 Romania RO 464 361 Serbia RS 101 78 Slovakia SK 121 94 Slovenia SI 166 123

Conflicts of interest (1) Expert selection Conflicts of interest (1) More clarity in FP7 Types of COI set out in appointment letter Disqualifying COI Involved in preparation of proposal Stands to benefit directly Close family relationship Director/trustee/partner Employee (but, see exception…) Member of advisory group Any other situation that compromises impartiality Potential COI Employed in last 3 years Involved in research collaboration in previous 3 years Any other situation that casts doubt… or that could reasonably appear to do so…

Conflicts of interest (2) Expert selection Conflicts of interest (2) Experts with a “disqualifying” COI cannot evaluate Neither in consensus group considering “problematic” proposal Nor in final panel One exception… if: The expert is employed in same organisation, but different department/lab/institute (e.g. CNRS) The constituent bodies operate with a high degree of autonomy Justified by the limited pool of qualified experts … then the Commission/REA might allow expert to participate in a panel review Should abstain if the specific proposal is discussed Exceptionally (very rare!!!), might participate in consensus group Experts with a “potential” COI Need to consider circumstances of case

The evaluation criteria 1. S/T quality (in relation to the topics addressed by the call) 2. Implementation 3. Impact Sound concept, and quality of objectives Appropriateness of the management structure and procedures Quality and relevant experience of the individual participants Contribution, at the European and / or international level, to the expected impacts listed in the work programme under the relevant topic / activity

The evaluation criteria Some exceptions Marie-Curie schemes for training and mobility of researchers Include, e.g. quality of training programme, suitability of host institution, etc. European Research Council (ERC) grants Scientific quality only criterion Excellence!!!

Each criterion is scored 0-5 Proposal scoring Each criterion is scored 0-5 Half-scores allowed Whole range should be considered Scores must pass thresholds if a proposal is to be considered for funding Thresholds apply to individual criteria: Default threshold is 3 … and to the total score Higher than the sum of the individual thresholds Default threshold is 10 Can vary from call to call!

Interpretation of the scores Proposal scoring Interpretation of the scores 0 - The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information 1 - Poor. The criterion is addressed in an inadequate manner, or there are serious inherent weaknesses. 2 - Fair. While the proposal broadly addresses the criterion, there are significant weaknesses. 3 - Good. The proposal addresses the criterion well, although improvements would be necessary. 4 - Very Good. The proposal addresses the criterion very well, although certain improvements are still possible. 5 - Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor.

Individual reading Evaluation process The experts evaluators first carry out individual readings (often done remotely) The experts: Evaluate the proposal individually (without discussing with the other evaluators) Check whether the proposal is “in scope” second check after the one done by the EC Complete an Individual Evaluation Report (IER) giving scores and comments on all criteria Scores should be in line with comments

IER Individual Evaluation Report Evaluation process May be remote Proposal 1 IER Individual Evaluation Report Expert A Consensus meeting Consensus: Scores & comments Proposal 1 IER Expert B Proposal 1 HEARING (optional) CR Consensus Report IER Expert C One proposal can be evaluated by more than 3 experts

Consensus Evaluation process Build on the basis of the individual assessments of all the evaluators Usually involves a discussion Moderated by a Commission/ REA representative Agreement on consensus scores and comments for each of the criteria One expert acts as rapporteur

Consensus reports – key points Evaluation process Consensus reports – key points The rapporteur is responsible for drafting the consensus report (CR) Includes consensus marks and comments The quality of the CR is paramount The aim is A clear assessment of the proposal, with justification Clear feedback on weaknesses and strengths To be avoided Comments that do not correspond with the scores Recommendations in view of resubmission A proof reader might be appointed for quality control

Hearings Evaluation process Co-ordinators whose proposals have passed the thresholds are invited to Brussels Intended to clarify any points raised by the experts in advance Not an occasion to “improve” the proposal Not an occasion for a multi-media show!

The final Panel Review Evaluation process Key function is to ensure consistency Final marks and comments for each proposal Evaluation Summary reports (ESR) New scores (if necessary)… carefully justified Clear guidance for contract negotiation Split proposals with identical consensus scores Approach is spelled out in WP and GFA Resolve cases where a minority view was recorded in CR [Exceptionally] recommendations for combining List of proposals for priority order

“Initial information” to applicants Sending of ESR Information to proposers “Initial information” to applicants Sending of ESR The Commission/REA does not change the ESR, except if necessary to: Improve readability [Exceptionally] To remove factual errors or inappropriate comments that may have escaped earlier proof-reading The scores are never changed The ESR is sent to the proposal co-ordinator – no commitments at this stage regarding funding This is the public face of the evaluation!

Redress? Appeal In the past, complaints arrived haphazardly Handled at different levels No systematic treatment No common record The redress procedure introduced for FP7 does not give a new right of appeal…… but it ensures a consistent and coherent approach to complaints Establishes “due process” Uphold principles of transparency and equal treatment

Redress: Principles and guidelines Appeal Redress: Principles and guidelines Redress will not “stop the train” Non-contentious proposals negotiated and selected as normal Complaints must relate to shortcomings in the handling of proposal evaluation Before a Commission decision has been made The procedure will not call into question the judgement of appropriately qualified experts

Independent observers Monitoring Independent observers Provide assurance that the process is fair And can provide constructive advice Not experts in the scientific area concerned Their reports are made available to the Programme Committee IOs are convened annually to a Round Table What are the common issues?

Commission/REA follow-up Evaluation process Commission/REA follow-up Evaluation summary reports sent to applicants Draw up final ranking lists Information to the Programme Committee Contract negotiation Formal consultation of Programme Committee (when required) Commission decisions Survey of evaluators Independent Observers’ reports

Expert questionnaire Survey For every call, experts receive a message on returning home Invited to complete an on-line survey Personal profile Evaluation process Evaluation criteria Opinion on the task and the other evaluators Logistics Comments and recommendations Early results sent to call co-ordinator after one month Full analysis at end of the year

Survey 96% of the respondents found the quality of the evaluation overall 'satisfactory' to 'excellent'

Survey 91% found the EU evaluation process similar or better than national or international schemes

Links EU research: http://ec.europa.eu/research/ 7th Framework Programme: http://ec.europa.eu/research/fp7/ Information on research activity and projects: http://cordis.europa.eu/ Questions? Contact the Research Enquiry Service http://ec.europa.eu/research/index.cfm?pg=enquiries 39 39

Thank you for your attention! Dr. Sebastiano FUMERO Head of Unit “FP7 Support” European Commission Research Executive Agency Tel: +32-2-296 96 88 sebastiano.fumero@ec.europa.eu