10 December 2015 – Parallel Session GOOD PRACTICES IN SELECTING MSCA COFUND FELLOWS Dr. Jennifer Brennan MRSC, Irish Universities Association M. Nicolas Walter, European Science Foundation “Synergies to fuel Researchers’ Careers” Luxembourg, 10 – 11 December 2015 organised by

AXA Research Fund Postdoctoral Fellowships Not M(S)CA Incoming scheme University of Turin Train2Move Programme FP7 MCA COFUND Incoming scheme IRCSET INSPIRE – FP7 MCA COFUND Outgoing scheme IRC ELEVATE – FP7 MCA COFUND Outgoing scheme 2 Cases Studied Note: In 2011, IRCSET & IRCHSS merged to form IRC

AXA Research Fund Promoting fundamental research related to risk through mobility grants at the global level Train2Move Maximising career opportunities for talented early career scientists and attracting promising researchers from all over the world IRCSET INSPIRE Offering a mobility experience for researchers based in Ireland IRC ELEVATE Offering a mobility experience (including into industry) abroad for researchers based in Ireland Objectives of the Programme

4 Boundary Conditions Disciplinary CoverageNumber of applications/awardsCo-funded by AXA Research Fund Environmental, Life or Socio Economic Risks No Technology development/applied research Conform with ethics /year for fellowships/year ( ) Success rate 6-8% N/A Train2Move 252 (2014) or 134 (2015) research topics – all disciplines except engineering and architecture Conform with ethics 156 for 14 grants in grants in 2015 Success rate 7-9% Compagnia di San Paolo (private foundation) INSPIRE All science, engineering & technology Conform with FP7 ethics 47 for 18 grants in 2009 Success rate 38% 91 for 32 grants in 2010 Success rate 35% Irish Government ELEVATE All science, engineering, technology, humanities and social sciences Conform with FP7 ethics Address potential gender dimension 74 for 15 grants in 2012 Success rate 20% 68 for 30 grants in 2013 Success rate 44% Irish Government

5 Assessment Structure - 1 Assessment ProcessAssessment CriteriaImplementing agent Outcome and Selection AXA Research Fund Two steps: Shortlisting of (90) outline proposals by panels Ranking of full proposals with inputs from external reviewers and candidate’s rebuttal Research environment (/4) Research Project (/4) CV (/4) European Science Foundation as independent third party One integrated ranked list across 3 panels Selection includes additional criteria Train2Move Two steps: Shortlisting of (42) proposals by panels Interviews and ranking of shortlisted applicants CV (/5) S&T quality (/5) Implementation (/5) Impact (/5) One integrated ranked list across 4 panels Selection follows the ranked list

6 Assessment Structure - 2 Assessment ProcessAssessment CriteriaImplementing Agent Outcome and Selection INSPIRE One step: Shortlisting and ranking of proposals by panels Track record/research potential (30) Scientific Quality (22) Proposed Impact (20) T & CD (15) Implementation (13) IRCSET with assistance from ESF in finding assessors 8 ranking lists, one per panel Budget distributed based on number of proposals per panel Selection follows the ranked list ELEVATE Call 1 – one step: Shortlisting and ranking of proposals by panels Call 2 – two steps: Assessment by external reviewers Shortlisting and ranking of proposals by two multi- disciplinary panels Track record/research potential (30) T & CD and Impact (25) Quality of Project (25) Host Org/Implementation (20) Irish Research Council Call 1 – 11 ranking lists Call ranking lists One ranking list per panel Budget distributed based on number of proposals per panel Selection follows the ranked list

Challenge in assessing applications from different parts of the world Different research experience and research environment impacts the scientific quality of applications although the potential may be present. Importance of specific and precise guidelines for applicants Importance of mentoring capability. Some positive discrimination procedures could be applied (e.g. identifying a list of ‘priority’ countries upon which identical success rate would be applied) Need to be thoroughly considered and addressed upstream, particularly if the programme also involves a capacity building element 7 Lessons Learned: Good Practice in Assessment 1

Challenge in addressing the mobility aspect Significant number of applications are not proposing real mobility as they involve applicants returning to their home countries or foreign applicants wishing to stay in the same organisation Challenging for panel members to approach Train2Move set an eligibility criteria (<12 months in Italy over the past 3 Years) – Allows to better streamline the issue INSPIRE & ELEVATE set a strict criteria about outgoing mobility 8 Lessons Learned: Good Practice in Assessment 2

Diversity Measures - ELEVATE Details regarding age, nationality and gender of applicants is hidden from evaluators so as to reduce the effects of unconscious bias. Ongoing process as Referees/Supervisors/Mentors sometimes inadvertently provide revealing information Applicants should include consideration of possible gender dimensions in their proposed research. Training and information on gender dimensions in research should be provided to applicants and other participants IRC Gender Action Plan – aim for gender balance in the membership of all assessment panels 9 Lessons Learned: Good Practice in Assessment 3

Assessing potential complementarities and synergies with the host department/institutions Motivating or making it mandatory for candidates to contact and interact with the host and possibly their local supervisors and asking them to report on this Involving representatives of the host organisation in the process and asking for direct feedback to the evaluators (letter and/or interview) INSPIRE/ELEVATE – application ineligible without written support from host at two levels, supervisor and institution Overcomes any lack of detailed knowledge of the host organisation and better consideration of the feasibility of the research project proposed. 10 Lessons Learned: Good Practice in Assessment 4

Maximising the interaction with the candidates AXA RF: Rebuttal step allowing clarification of issues in the external evaluation reports Reports provided to shortlisted candidates before they submit full applications – allow consideration of comments Career statements: allows a broader perspective on the candidates and their trajectory Train2Move: Specific questions to be addressed during the interview in the shortlisting assessment report Skype/phone interview Interactions with candidates allows clarity around the application, removes potential misunderstandings and also improved perspectives on candidates and their trajectory 11 Lessons Learned: Good Practice in Assessment 5

Ensuring a high quality evaluation (and dedicating enough time for this!) Critical that evaluators have all the same understanding of criteria E.g. the ‘Impact’ criteria often hard to approach: impact for the candidate, for the host, on the discipline? Needs to be carefully explained Disciplinary review panels should be composed of experienced researchers with a broad knowledge and understanding of the disciplines External (mail) reviewers should be precisely targeted to hold the required expertise and allow review panels to make better informed decisions Assessments by external reviewers should be checked and validated to provide a coherent level of information Reports from panel members should be checked and validated to ensure they are useful for the applicant Conflict of Interest should be dealt with in a systematic and consistent manner Importance of the panel secretaries in ensuring procedure is followed 12 Lessons Learned: Good Practice in Assessment 6

Evaluators need to be well treated! Even if some proposals may be rather short, going through them, checking some background and producing useful assessment always take more time than expected Limit the number of applications per reviewer/panel member AND/OR Allow more time to produce Support from the office is critical - Importance to provide smooth operations at the preparation/reporting phases (online platform) during the meetings (experienced staff and well defined guidelines) Some consideration could be given to compensate evaluators (IRC does) Allow reviewers to address consolidated comments (e.g. panel chairs’ declaration 13 Lessons Learned: Good Practice in Assessment 7

Assessment is an investment The application evaluation is part of the critical path towards maximising the value and impact of a fellowship programme Targeted identification of evaluators (panel members and external (mail) reviewers) is key Well defined and coherent guidelines need to be produced and respected Check and validation of assessments and reports is very important, but is time-consuming All these take a lot of time and require specific expertise 14 Lessons Learned: Good Practice in Assessment 8

Partnering with a third party with a core competency in assessment The ESF experience of partnering with fellowship programmes allows strengthening of the robustness of the assessment process while lifting some pressure on the programme management team Allows separation of the assessment from the selection process – increases the independence Allows mobilisation of multidisciplinary teams and buffers workload peaks when time is limited Allows use of existing optimised and proven procedures, infrastructure and experience No need to develop in-house know-how in case of non-recurring process Allows gain and development of in-house capacity and know-how to build up internal capacity Allows access to dedicated databases of experts 15 Lessons Learned: Good Practice in Assessment 9

Dr. Kate Ryan, Irish Research Council 16 Acknowledgements