Options for Evaluating Research: Inputs, Outputs, Outcomes Michele Garfinkel Manager, Science Policy Programme ICSTI ITOC Workshop 19 January 2015, Berlin

Today’s talk About EMBO A policy view of research assessment Stakeholder roles

About EMBO European Molecular Biology Organization (Maria Leptin, Director) Founded 1964, Heidelberg, DE Funded by the European Molecular Biology Conference –27 Member States –3 cooperation agreements Advancing policies for a world-class European research environment

Governance Three main areas: biotechnology, responsible conduct of research, scientific publishing –Technology assessment Scientific publishing –Open access –Data –Responsibilities of editors, administrators, authors Science Policy Programme

Scientific publishing The publication of scientific information is intended to move science forward. More specifically, the act of publishing is a quid pro quo in which authors receive credit and acknowledgment in exchange for disclosure of their scientific findings.

Journal name as proxy for quality Journal Impact Factor: a librarian’s number The concern is not use, but misuse –Research assessment –“JIF : a subscription for the price of the IF” Why has this been adopted for research assessment? –Cross-disciplinary –Intuitive and reflective –Prospective

Research assessment is an ecosystem Funders Researchers Journals Other assessors?

What DORA sets out Main recommendation: Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions Implementation?

What DORA sets out Research institutions and funding agencies: be clear on evaluation criteria and consider all contributions Publishers: do not use JIF as a marketing tool, make more article level metrics available, make all reference lists open, remove limits on reference list length

What DORA sets out Metrics suppliers: provide methodology and data in a useful form, account for variation in article types (reviews v. research articles) Researchers: as assessors, review for scientific content; as authors, cite appropriate (primary) literature; challenge bad practices

What DORA does not say Metrics based research assessment is wrong JIF is flawed for assessing journals Citations are a flawed metric There is a simple alternative Publishers are to blame Thomson Reuters is to blame

What DORA does not say Metrics based research assessment is wrong JIF is flawed for assessing journals Citations are a flawed metric There is a simple alternative Publishers are to blame X is to blame Altmetric Score

More institutions and funders emphasizing biosketches and 'select your 5 best papers' strategies over IF. Constructive discussions with Thomson Reuters. More interest in dialogue and a willingness to improve the JIF as a metric Competition is good for everyone Incremental advances

Engagement with funders Engaging additional research communities Study national/regional variations Editorials forthcoming –Key point: better analyses needed Policy analysis –Implementation and governance issues, metrics, stakeholders Incremental advances

This is not (just) about overworked or lazy promotion committees and rapacious journals The reward system in science is (becoming) warped Resources for thorough evaluation are not available Journal articles have become the currency of rewards rather than a contribution to knowledge It’s the system (?)

Researchers Publishers Research administrators Funders Metrics researchers Metrics providers Decision-makers Research Assessment: Stakeholders

We are great at measuring inputs (funding, numbers of students) We are good at measuring outputs (numbers of papers, some impact measures) Outcomes measurements are a problem What should we be assessing?

Papers –And how they are discovered? Data –And how they are discovered? Reviewing? Teaching? Committee work? Responsible conduct? What should we be assessing?

Workshops –Governance issues –Stakeholders Engagement with funders Ongoing work