Impressions of a Belgian evaluator for COST Patricia Mergen Royal Museum of Central Africa Meise Botanic Garden

Liaison Officer Quite a nice Fairy Tale Job !

BUT

When is the Africamuseum opening again ?

Background Information Country of Origin : Luxembourg University of Namur : Master in Biology (Topic Paleo-ecology, fossil reefs along the Meuse River) PHD in Biology (Freshwater Ecology, Distribution of Fish in Reservoir lakes in Belgium and Luxembourg ) 2002-2005 : Belgian Biodiversity Information Facility (now under Belgian Biodiversity Platform) at the ULB-VUB 2005 – : Royal Museum for Central Africa with regular sub-contracting to Botanic Garden Meise (currently ½ time)

Work experience Biodiversity Information projects and training provider Identifying funding opportunities Supporting in project proposals writing Supporting management of ongoing projects Representing the institutions in national and international bodies (advisory, think tanks, decision making ) Presentation of outcomes and activities of the institutions or projects at various events

Evaluator expertise Independent evaluator for EU FP6, FP7, H2020, Marie Curie Fellowships, COST proposals Independent reviewer to follow up ongoing EU projects in FP7, H2020 Evaluation of bilateral projects with Poland, Norway … Evaluator for foundations such as IFS (International Foundation for Sciences), JRS Biodiversity grants, GBIF calls for proposals, … Official or informal advisory in follow up committees of ongoing projects Independent expert in different groups at EU level, national or regional level (think tanks on next programs, calls, evaluation procedures … )

Evaluator expertise (cont) Main contact for the external evaluation of scientific outputs of the Royal Museum for Central Africa Close collaboration with EU National Contact points (NCPs) of Belgium and helpdesks at EU level, Testing the Participant portal …. Chair of the Belgian National Task Force for ESFRI proposal DiSSCo (Distributed System of Scientific Collections) Member of international associations such as Biodiversity Information Standards (TDWG), Consortium of European Taxonomic Facilities (CETAF), Training provider and mentor for European GBIF project BID (Biodiversity Information for Development)

General added value to be evaluator Informed of latest state of the art in your domain or related domains Better understanding of the expectations of the funding bodies Better understanding of the procedures, online tools, what to fill in, into the many forms Better understanding on how to manage a project Better understanding of financial management and reporting requirements, budget allocations and usage of the funding Better understanding of distribution of tasks and how to do the reporting of the deliverables.

Experience with COST Review Panel Member for Belgium in the domain Natural Sciences Acted as Evaluator in 2015 Had 9 proposals to check in my panel Two evaluators assigned to each proposal Other evaluators in the same panel also had 9-10 proposals to check

Remote evaluation Agree who is rapporteur You do not evaluate the proposals themselves, but check for each proposal the reviews of each of the 3 evaluators + the consensus report Produce a review panel report Clear instructions provided via Evaluators Guide and Review Panel members’ Guide, yet remaining challenging

Actually more work then evaluating proposals directly !

Remote evaluation challenges (procedures) Read 9 proposals + those in your panel (even if you do not have to evaluate them in view of the local consensus panel meeting in Brussels) For each proposal 13 questions, 3 evaluators + consensus report You have to respect the opinion of the evaluators and cannot change the points given, even if you do not agree with their analysis Check the wording, spelling if it is clear, rewrite some parts of the evaluation reports but not the meaning. Check if things are said that are not in the proposal but inferred (may need to remove) Check for improper or prone to be taken as insulting language .. For some proposals reviewers did not produce a consensus report because they could not come to an agreement, in these cases the review panel has to produce the consensus report

Onsite evaluation in Brussels Over 4 days , good collaboration with COST Association staff, very helpful. Introduction and guidelines on the procedures and ranking to achieve by the end of the meeting. Meeting in review panels by domain (Natural Sciences) and we go through all proposals one by one. The rapporteur orally summarizes the review panel report and detected issues are discussed, text modifications approved, amended or rejected. Decisions on go or no go are agreed. Hence the competition only highly ranked proposals will actually be funded in practice. At the end of the process a general debriefing is held collecting feedback from the reviewers.

General challenges Some evaluators have a hard time to follow the guidelines … Many evaluation reports are very good and clear Some are very long and difficult to summarize, go beyond the evaluation and add interpretations that are not in the proposal Some are too short or unclear (for example they answer just yes or no but without detailing much why, or say that applicants failed to meet all criteria, or forgot important stakeholders, users, but without saying which) Experts in the review panel expressed frustrations that they could not modify the marks or have a different opinion than the evaluators of the proposal, but this rule has to be accepted … In some case it would have been really useful to contact the evaluators to ask them what they meant, but this is not foreseen in the process even if it could be done without breaching the anonymity of the evaluators and reviewers Did not encounter any reports with improper language.

Why did I not review any COST since 2015 ? Conflict of Interest !!! Yeah! We are renovating , but remain very active and every 6 months both the Royal Museum for Central Africa and the Meise Botanic Garden were partners to COST proposals in Natural Sciences. Check out : www.doedat.be www.botanicalcollections.be

Thank you for your Attention ! Any Questions ?