EU Expert Group Altmetrics Next-generation altmetrics: responsible metrics and evaluation for open science EU Expert Group Altmetrics

EU expert group members James Wilsdon, University of Sheffield (chair); Judit Bar-Ilan, Bar-Ilan University; Robert Frodeman, University of Texas; Elizabeth Lex, Graz University of Technology; Isabella Peters, Leibniz Information Centre for Economics; Paul Wouters, Leiden University Team Leader-Open science policy coordination and development: Rene von Schomberg

Aims /1 assess role (alt)metrics in research evaluation consider how altmetrics can be developed for open science engage stakeholders consider implications metrics for: diversity and equality interdisciplinarity research cultures gaming :

Aims /2 examine implications of: emerging social networks research information systems citation profiles explore altmetrics for impacts, research actions, in Horizon 2020 and in next framework programme consider required data infrastructures

The Leiden Manifesto Quantitative evaluation should support expert assessment. Measure performance in accordance with the research mission. Protect excellence in locally relevant research Keep data collection and analytical processes open, transparent and simple. Allow for data verification Account for variation by field in publication and citation practices Data should be interpreted taking into account the difficulty of credit assignment in the case of multi-authored publications. Base assessment of individual researchers on qualitative judgment. False precision should be avoided (eg. the JIF).  Systemic effects of the assessment and the indicators should be taken into account and indicators should be updated regularly

Responsible metrics Robustness: basing metrics on the best possible data in terms of accuracy and scope; Humility: recognizing that quantitative evaluation should support – but not supplant – qualitative, expert assessment; Transparency: keeping data collection and analytical processes open and transparent, so that those being evaluated can test and verify the results; Diversity: accounting for variation by field, using a variety of indicators to reflect and support a plurality of research & researcher career paths; Reflexivity: recognizing the potential & systemic effects of indicators and updating them in response. The Metric Tide

Measuring is changing What counts as excellence is shaped by how we measure and define “excellence” What counts as impact is shaped by how we measure and define “impact” Qualities and interactions are the foundation for “excellence” and “impact” so we should understand those more fundamental processes first We need different indicators at different levels in the scientific system to inform wise management that strikes the right balance between trust and control Context crucial for effective data standardization

Report outline Introduction Metrics: technical state of the arts Use of metrics in policy and practice Data infrastructures and open standards Cultures of counting, ethics and research Next generation metrics: the way forward

Traditional metrics Based on citation and publication counts are not sufficient Citations take time to accumulate IF often used as a proxy for citation count h-index Disciplinary differences in publication and citation culture Ignore societal impact DORA, Leiden Manifesto, Metric Tide

Altmetrics Intend to capture and measure additional aspects of scholarly information Altmetrics Manifesto New forms of communication reflect and transmit scholarly impact Expand our view of what impact looks like Altmetric platforms Increased visibility of researchers/publications Exposing research to the public Involving the public Discussion/commenting Readers vs. authors Altmetric events can be measured/counted

Altmetrics - challenges Coverage Transparency Validity Dynamics Disciplinary differences Gaming Acceptance Research community Decision makers :