1. Research Indicators for Open Science
Nordic Workshop on Data Citation Policies and Practices: How to Make it Happen? 23 November 2016, Helsinki
Research Indicators for Open Science
Anu Nuutinen
Senior Science Adviser
Academy of Finland
anu.nuutinen(at)aka.fi
© ACADEMY OF FINLAND 2016

2. Academy of Finland in a nutshell
Four research councils
Biosciences and Environment
Culture and Society
Natural Sciences and Engineering
Health
Key public funding agency for scientific research, major player in science policy in Finland
* To support scientific research and research careers
* To develop research environments
Finnish Research Infrastructure Committee
employees
Strategic Research Council
Funding in 2015
140
€405m*
© ACADEMY OF FINLAND 2016 | FOR EXCELLENCE IN SCIENCE

3. Open science at the Academy of Finland
Recommended archives and storage services
FSD, FIN-CLARIN, CERN Zenodo, EUDAT, AVAA, Etsin, IDA
Data management plans required
Opening of data and methods required
Depends on research ethics & law
Openly available publications required
Funding for publishing costs provided
Academy is committed to Finnish Open Science and Research Roadmap *
Results must be made public
* To improve overall quality and impact of research and
promote the good scientific practice
© ACADEMY OF FINLAND 2016

4. Openness is a key principle in science
Promotes the reproduction of scientific results.
Openness, in practice, improves the overall quality and impact of research.
Open science forms part of the principles of good scientific practice.
The goal is to make research publications, data and material, metadata and methods widely available for further use.
Academy is committed to considering the promotion of open science as one of the criteria in funding decisions.
© ACADEMY OF FINLAND 2016

5. The proposed actions advancing open science must be described in the grant proposal
Research plan
Publication plan & description, justification, collection, use of material & ethical issues
Data management plan
Management, storage, access, opening, rights of the research data or brief account if there is no data
Research organisations and research infrastructures are expected to support open access publishing and the delivery of open research data and methods
© ACADEMY OF FINLAND 2016

6. Evaluation of grant proposals and funding decisions
Evaluation of quality: Ethical aspects and open science (no rating)
The international review panels are asked to give their assesment of the planned open science activities
Are there any ethical issues involved and, if so, how are they taken into account?
What is the intended level of open access to research results?
Is the data management plan worked out in a sufficient way?
Funding decisions
The Academy pays attention to the scientific evaluation, and in addition to science policy objectives.
The science policy objectives also include the promotion of open science.
© ACADEMY OF FINLAND 2016

7. Scientific reporting after the funding period
The follow up of suggested open science activities is planned to be implemented also in the Academy’s new scientific reporting
At the moment bibliometric analyses are based on scientific publications; there is no comprehensive enough data available to analyse data citations
Publication data can be mapped with Web-of-Science- based data
BUT: The quality of reported publication data of pivotal importance
Correct and complete bibliographic information!
Allows a small-scale bibliometric analysis of the scientific impact of Academy funded research
© ACADEMY OF FINLAND 2016

8. On the path towards analysing data citations
What are we assessing?
Amount and types of activity (Quantity, publication volume)
Scientific impact (visibility, interest within the scientific community)
Impact indicators alone do not provide a reliable overall picture of the level of research.
Add a useful perspective to the analysis of scientific impact.
Collaboration (eg. international collaboration, national collaboration, no collaboration)
Quality
The role of peer review
© ACADEMY OF FINLAND 2016

9. Citation analyses: What needs to be taken into account?
Disciplinary differences
Various types of research data
Disciplinary differences in citation practices
Citation databases (citation indexing services)
Overall coverage of data outputs
Automated tracking of data outputs a necessary requirement
Unique identifiers, DOI
Credit must be assigned correctly to the researchers who produced the data output
Data cleaning especially important in organisation level analyses
© ACADEMY OF FINLAND 2016

10. Research indicators: Methodological issues
Normalisation of the number of citations
By research field, publication year
Comparison to the international level in the research field
Self citations often removed when anaysing scientific impact
Counting method
Whole counting of publications
Fractionalisation between countries, organisations, research fields
The fractional counting method leads to a more proper field normalisation of impact indicators
Fairer comparisons between organisations active in different fields
Importance of methodological research on research indicators
© ACADEMY OF FINLAND 2016

11. Conclusions
Openness improves the overall quality and impact of research
Disciplinary differences need to be acknowledged when designing and applying reserach indicators
Methodological research on research indicators is needed
Responsible use of research indicators is very important
Metrics has its limitations!
They need to be acknowledged and taken into account when interpreting the indicators and drawing conclusions.
© ACADEMY OF FINLAND 2016