Research Indicators for Open Science Nordic Workshop on Data Citation Policies and Practices: How to Make it Happen? 23 November 2016, Helsinki Research Indicators for Open Science Anu Nuutinen Senior Science Adviser Academy of Finland anu.nuutinen(at)aka.fi © ACADEMY OF FINLAND 2016

Academy of Finland in a nutshell Four research councils Biosciences and Environment Culture and Society Natural Sciences and Engineering Health Key public funding agency for scientific research, major player in science policy in Finland * To support scientific research and research careers * To develop research environments Finnish Research Infrastructure Committee employees Strategic Research Council Funding in 2015 140 €405m* © ACADEMY OF FINLAND 2016 | FOR EXCELLENCE IN SCIENCE

Open science at the Academy of Finland Recommended archives and storage services FSD, FIN-CLARIN, CERN Zenodo, EUDAT, AVAA, Etsin, IDA Data management plans required Opening of data and methods required Depends on research ethics & law Openly available publications required Funding for publishing costs provided Academy is committed to Finnish Open Science and Research Roadmap 2014-2017* Results must be made public * To improve overall quality and impact of research and promote the good scientific practice © ACADEMY OF FINLAND 2016

Openness is a key principle in science Promotes the reproduction of scientific results. Openness, in practice, improves the overall quality and impact of research. Open science forms part of the principles of good scientific practice. The goal is to make research publications, data and material, metadata and methods widely available for further use. Academy is committed to considering the promotion of open science as one of the criteria in funding decisions. http://www.aka.fi/en/funding/responsible-research/open-science/ © ACADEMY OF FINLAND 2016

The proposed actions advancing open science must be described in the grant proposal Research plan Publication plan & description, justification, collection, use of material & ethical issues Data management plan Management, storage, access, opening, rights of the research data or brief account if there is no data Research organisations and research infrastructures are expected to support open access publishing and the delivery of open research data and methods © ACADEMY OF FINLAND 2016

Evaluation of grant proposals and funding decisions Evaluation of quality: Ethical aspects and open science (no rating) The international review panels are asked to give their assesment of the planned open science activities Are there any ethical issues involved and, if so, how are they taken into account? What is the intended level of open access to research results? Is the data management plan worked out in a sufficient way? Funding decisions The Academy pays attention to the scientific evaluation, and in addition to science policy objectives. The science policy objectives also include the promotion of open science. © ACADEMY OF FINLAND 2016

Scientific reporting after the funding period The follow up of suggested open science activities is planned to be implemented also in the Academy’s new scientific reporting At the moment bibliometric analyses are based on scientific publications; there is no comprehensive enough data available to analyse data citations Publication data can be mapped with Web-of-Science- based data BUT: The quality of reported publication data of pivotal importance Correct and complete bibliographic information! Allows a small-scale bibliometric analysis of the scientific impact of Academy funded research © ACADEMY OF FINLAND 2016

On the path towards analysing data citations What are we assessing? Amount and types of activity (Quantity, publication volume) Scientific impact (visibility, interest within the scientific community) Impact indicators alone do not provide a reliable overall picture of the level of research. Add a useful perspective to the analysis of scientific impact. Collaboration (eg. international collaboration, national collaboration, no collaboration) Quality The role of peer review © ACADEMY OF FINLAND 2016

Citation analyses: What needs to be taken into account? Disciplinary differences Various types of research data Disciplinary differences in citation practices Citation databases (citation indexing services) Overall coverage of data outputs Automated tracking of data outputs a necessary requirement Unique identifiers, DOI Credit must be assigned correctly to the researchers who produced the data output Data cleaning especially important in organisation level analyses © ACADEMY OF FINLAND 2016

Research indicators: Methodological issues Normalisation of the number of citations By research field, publication year Comparison to the international level in the research field Self citations often removed when anaysing scientific impact Counting method Whole counting of publications Fractionalisation between countries, organisations, research fields The fractional counting method leads to a more proper field normalisation of impact indicators Fairer comparisons between organisations active in different fields Importance of methodological research on research indicators © ACADEMY OF FINLAND 2016

Conclusions Openness improves the overall quality and impact of research Disciplinary differences need to be acknowledged when designing and applying reserach indicators Methodological research on research indicators is needed Responsible use of research indicators is very important Metrics has its limitations! They need to be acknowledged and taken into account when interpreting the indicators and drawing conclusions. © ACADEMY OF FINLAND 2016