OpenUP – Revisiting Peer Review, Dissemination & Impact Measurement OPEN SCIENCE -- KNOWLEDGE FOR ALL Lisbon, 29 March 2016 Wolfram Horstmann University of Göttingen Göttingen State and University Library (With contributions by Birgit Schmidt) 1

usage Traditional disseminate-review-impact processes: How are these affected by Open Scholarship? citations (open) peer review JIF social networks SNIP SCJ ALM Altmetrics h-index blogs reporting Source Normalized Impact per Paper (SNIP) SCImago Journal Rank (SJR)

Impact assessment: Some History Impact-factor ‚obsession‘ is still ongoing Impact-factor developed 1955 onwards Introduced 1972 for librarians to select journal subscriptions As much embraced as attacked for its simplicity Main Criticisms Defines impact on a journal and not on an article level A linear scale of research quality is suggestive Prone to ‚gaming‘ Numerous other metrics & service providers now available CWTS Journal Indicators, Scimago Journal Rank, h-index, altmetrics

Impact assessment: Some history http://www.thomsonreuters.com Interviewer „I often hear ... that any prestigious journal should have a JIF of ‚2‘ or higher...“ Eugene Garfield „It is simply an arbitrary number; I can‘t justify it scientifically.“ http://www.garfield.library.upenn.edu/papers/medicalwritingv8(1)1999.html

http://www.nature.com/news/bibliometrics-the-leiden-manifesto-for-research-metrics-1.17351

Reviewing assessment practice DORA: San Francisco Declaration on Research Assessment American Society for Cell Biology, December 2012 Not to use journal-based metrics to judge the quality of a paper http://www.ascb.org/dora/ Leiden Manifesto April 2015, by a group of research assessment experts Best-practice recommendations for metrics-based research assessments Summarized in 10 principles, e.g. #1 Quantitative evaluation should support qualitative, expert assessment #4 Keep data collection and analytical processes open, transparent and simple #5 Allow those evaluated to verify data and analysis #6 Account for variation by field in publication and citation practices #10 Scrutinize indicators regularly and update them http://www.nature.com/news/bibliometrics-the-leiden-manifesto-for-research-metrics-1.17351

The Metric Tide A growing tide due to growing pressures for audit and evaluation Demand for more strategic intelligence, competition within and between institutions Research community uses metrics but with too much emphasis on narrow indicators Peer review, despite its flaws and limitations, still has widespread support across disciplines Inappropriate indicators create perverse incentives Indicators only meet potential if underpinned by an open and interoperable infrastructure Responsible metrics = robust + transparent + supplementing qualitative assessment + accounting for variation by field + recognising and anticipating systematic and potential effects of indicators + updating indicators in response https://responsiblemetrics.org/the-metric-tide/

OpenUP project facts Opening up new methods, indicators and tools for peer review, dissemination of research results, and impact measurement Project start: June 2016 (30 months) Partners Public Policy and Management Institute (PPMI) – Coordinator University of Göttingen University of Athens University of Amsterdam Know-Center, Graz Austrian Institute of Technology German Centre Higher Education Research & Science Studies (DZHW) Frontiers Media CNR

The review-disseminate-assess process What‘s missing? A common understanding of new methods, implementation good practices as well as policy implications, with a view towards Responsible Researhc and Innovation (RRI) principles) What‘s to be done in OpenUP? Facilitate an open dialogue and define a framework, validate mechanisms through pilots, offer trainings and provide practical policy guidelines and recommendations

Implementation Peer review Validation through community-driven pilots Innovative dissemi-nation Policy recommen-dations Image: http://images.inc.com/uploaded_files/image/1940x900/communication-group-1940x900_35361.jpg Metrics & Indicators

Objectives Engage the community & raise awareness for quality Responsible Research & Innovation (RRI) review-disseminate-assess processes Explore, analyse and promote (open) peer review mechanisms Explore and promote innovative methods of dissemination & communication Define research metrics and indicators for different stakeholders Validate the OpenUP framework with community-driven pilots Produce cohesive policy recommendations to complement the European RRI activities

OpenAIRE Open Access Infrastructure for Research in Europe Fostering and furthering Open Scholarship in Europe and beyond Human Network Digital Network 50 Partners: Data centres, universities, libraries, repositories from all EU countries and beyond

A European research information infrastructure A mini EU-CRIS system Discovery Metadata Validation Cleaning De-duplicating Inferring Linking Organiz-ations Projects Authors Datasets Publications Data Providers Crowdsourcing Funding Info Monitoring CRIS systems Reporting Literature Repositories Full text APIs OA Journals Classification Clustering Analysis Evaluation Data Repositories Usage data Impact … Zenodo Data Providers OpenAIRE Platform Services

New modes of research and scholarly communication COAR aims to facilitate the vision by bringing together research repositories as part of a global infrastructure; to link across continents and around the world, enabling new forms of research and supporting new models of scholarly communication. Standards, policies, infrastructure, strategy, interoperability, support, awareness, integration… Research libraries have a central role to play. Inspired by Tim Berners Lee

…will enhance the provision, visibility and use of research outputs Our vision A global knowledge infrastructure, built upon a network of open access digital repositories …will enhance the provision, visibility and use of research outputs

Factsheet • COAR e.V., a registered not-for-profit association of repository initiatives according to German law, Office seat: Göttingen, DE, Host: Göttingen State and University Library Founded in Ghent, Belgium, October 21, 2009 (28 members), evolved out of the European DRIVER Project (EC, FP7) One Annual Meeting of all members with General Assembly Members & Partners (April 2015): about 100 member institutions (out of over 35 countries in all continents); 8 partner organizations Elected Executive Board: Chairperson: Eloy Rodrigues, University of Minho, Portugal Vice Chairperson: Carmen-Gloria Labbé, Cooperación Latinoamericana de Redes Avanzadas (CLARA), Uruguay Treasurer: Márta Virágos, University and National Library of Debrecen, Hungary Oya Rieger, Cornell University, USA Wolfram Horstmann, Göttingen State and University Library, Germany Daisy Selematsela, National Research Foundation, South Africa

Summary & Conclusion Observations on research assessment Open Science A rising tide of impact maesurement Critique of unidimensional indicators New and qualitative indicators on the rise Open Science ... requires new impact measures to assess new methods ... provides the opportunities to apply them Helping initiatives underway OpenUP explores pilots, practice and policies OpenAIRE provides information infrastructure COAR extends towards global reach

Outlook: Impact is multi-dimensional https://www.sheffield.ac.uk/bbsrc-ewi/understanding-impact/index Source: BBSRC / UK