Bibliometrics and Current Research Information Systems (CRIS) Gunnar Sivertsen Nordic Institute for Studies in Innovation, Research and Education, Oslo, Norway

Outline CRIS systems and performance-based funding – a European overview The evaluation of the CRIS-based publication indicator in Norway Measuring research responsibly with CRIS

Outline CRIS systems and performance-based funding – a European overview The evaluation of the CRIS-based publication indicator in Norway Measuring research responsibly with CRIS

Current research information systems (CRIS) “A current research information system (CRIS) is a database or other information system to store and manage data about research conducted at an institution.” (Wikipedia)

Current research information systems (CRIS) “A current research information system (CRIS) is a database or other information system to store and manage data about research conducted at an institution.” May also serve the purpose of information

Current research information systems (CRIS) “A current research information system (CRIS) is a database or other information system to store and manage data about research conducted at an institution.” May also serve the purpose of information and statistics

Current research information systems (CRIS) “A current research information system (CRIS) is a database or other information system to store and manage data about research conducted at an institution.” May also serve the purpose of information and statistics Data may be integrated at the national level

National integration of data - the Czech solution

National integration of data - the Danish solution Each university has a local Pure system. Annually, data from the local systems are exported to, and integrated in, a national database owned by the Government

The CRIS can itself be nationally integrated – Norway’s solution Principles behind the use of institutional data in a shared national Current Research Information System Completeness: All scholarly publications and other results from research are included Transparency: Every institution can see and check all other institutions’ data. The national database is also online and open to society at large. Multiple use of the data: CV’s, applications, evaluations, annual reports, internal administration, bibliography for Open Archives, links to full text, etc.

The VIRTA solution Previous data collection VIRTA publication information service Data sources Local CRISes of HEIs Local CRISes of HEIs, university hospitals, state research institutes Updates Annually Automatically and real time (statistics compiled once a year) Data format CSV XML Corrections, supplementations Not possible after annual data collection Possible any time Temporal coverage Since 2011 All data from previous years to present can be transferred Identification of authors Name Name, ORCID Identification of duplicates, faults etc. Identified manually Identified automatically and real time Services connected JUULI, Vipunen JUULI, Vipunen + researchers’ reporting for funding agencies, visualization services, other research services etc.

Commercial CRIS-solutions for institutional purposes have become widespread

Within or between institutions? Open data? Comparability? Transparency of methods? Agreement on indicators? Collaboration or competition?

The EUROCRIS conference 2016, St. Andrews, Scotland The first keynote on metrics The first euroCRIS track on metrics

CRIS and indicators for performance-based funding of research institutions Bibliometrics based on complete data is translated into funding. Often combined with other indicators.

Diffusion of the “Norwegian model” Bibliometrics based on complete data is translated into funding. Often combined with other indicators.

The «Norwegian model» at Swedish universities

Nordic collaboration supported by Nordforsk A standardized Nordic register of scholarly publication channels Analysis of CRIS-data

Outline CRIS systems and performance-based funding – a European overview The evaluation of the CRIS-based publication indicator in Norway Measuring research responsibly with CRIS

Evaluation published 16. January 2014 Performed in 2013 by the The Danish Centre for Studies in Research and Research Policy, Aarhus University, Denmark Commisioned by the Norwegian Association of Higher Education Institutions Financed by the Norwegian Ministry of Education and Research Based on surveys, interviews and bibliometrics Has a 10 page summary in English

The basis for four scientific publications Bloch, C., & Schneider, J. W. (2016). Performance-based funding models and researcher behavior: An analysis of the influence of the Norwegian Publication Indicator at the individual level. Research Evaluation, 25 (4): 371-382. Schneider, J.W., Aagaard, K., Bloch, C. (2016). What happens when national research funding is linked to differentiated publication points? A comparison of the Australian and Norwegian publication-based models. Research Evaluation 25, (3): 244-256. Aagaard, K. (2015). How incentives trickle down: Local use of a national bibliometric indicator system. Science and Public Policy, 42(5), 725-737. Aagaard, K., Bloch, C., & Schneider, J. W. (2015). Impacts of performance-based research funding systems: The case of the Norwegian Publication Indicator. Research Evaluation, 24(2), 106-117.

”The dual funding system” (Public sources) Government Direct grants Historical, Political Strategic Performance based Research institutions Research Councils Projects and Programmes Competition Research Evaluation

Direct grants «The Norwegian model» Historical, Political Strategic The model has three components: A complete representation in a national database of structured, verifiable and validated bibliographical records of the peer-reviewed scholarly literature in all areas of research; A publication indicator with a system of weights that makes field-specific publishing traditions comparable across fields in the measurement of “Publication points” at the level of institutions; A performance-based funding model which reallocates a small proportion of the annual direct institutional funding according the institutions’ shares in the total of Publication points. Direct grants Historical, Political Strategic Performance based

Books Journals and series Indexed journals Incomplete coverage of international journals in the SSH. Very limited coverage of books. Random or no coverage of the national level (books and journals)

Publication points Publication type Level 1 Level 2 Article in ISSN-title 1 3 Article in ISBN-title 0,7 ISBN-title 5 8 Publication Author 1 Institution A Author 2 Institution B Institution C Author 3 Publication points are fractionalized if the publication originates from more than one institution

Publication channels on two levels Level 2 represents the most prestigous channels and is defined in collaboration with the national councils in each field Level 2: Higher points to 20 % of the publications Level 1: Normal points to 80 % of the publications

Main results of the evaluation Increased publishing – beyond the increased funding during the period An increased number of researchers are publishing Publication patterns remain stable: Type of publications, language of publications “Quality” does not go up or down: Citation rates and share of publications on level 2 are stable

Main results of the evaluation Three major problems: The weighting model, the balance between major fields (effects of fractioning in case of multi-authorship) The transparency and legitimacy of the nomination processes for level 2 Local use of the model Continuation with increased economic effects

Main results of the evaluation Three major problems: The weighting model, the balance between major fields (effects of fractioning in case of multi-authorship) The transparency and legitimacy of the nomination processes for level 2 Local use of the model Continuation with increased economic effects

The lack of balance: Average publication points among researchers in five major areas of research

Research profiles of three Norwegian universities Based on Norwegian CRIS data

Calibrating for a better balance Data on scholarly publishing from CRIStin (Current Research Information System in Norway) Years 2011-12 Nearly 15,000 active researchers (contributing to a minimum of two publications in two years) Simulation of average number of publication points in five areas

The present model Publication type Level 1 Level 2 Article in ISSN-title 1 3 Article in ISBN-title 0,7 ISBN-title 5 8 Changing the weights does not create a better balance Publication Author 1 Institution A Author 2 Institution B Institution C Author 3 Publication points are fractionalized if the publication originates from more than one institution Publication points may then be multiplied by a certain factor if co-authors from other countries are involved

The former model Publication type Level 1 Level 2 Article in ISSN-title 1 3 Article in ISBN-title 0,7 ISBN-title 5 8 Publication Author 1 Institution A Author 2 Institution B Institution C Author 3 Publication points are fractionalized if the publication originates from more than one institution Introducing or increasing this factor contributes to a better balance Publication points may then be multiplied by a certain factor if co-authors from other countries are involved

The present model Publication type Level 1 Level 2 Article in ISSN-title 1 3 Article in ISBN-title 0,7 ISBN-title 5 8 Publication Author 1 Institution A Author 2 Institution B Institution C Author 3 Publication points are fractionalized if the publication originates from more than one institution Fractionalization has the largest effect on the problem/solution Publication points may then be multiplied by a certain factor if co-authors from other countries are involved

The new model (Norway) Publication type Level 1 Level 2 Article in ISSN-title 1 3 Article in ISBN-title 0,7 ISBN-title 5 8 Publication Author 1 Institution A Author 2 Institution B Institution C Author 3 The square root of the institution’s fraction is used instead of simply the fraction Publication points are multiplied by a factor of 1,3 if co-authors from other countries are involved The total number of unique author-institution relations (4 in this example), not the total number of authors (3 in this example), are counted

Balancing research productiviy across all fields Area of research Researchers Publications in two years Publication points (now) Publication points (new) Humanities 1074 3,7 3,9 4,6 Health Sciences 5724 6,3 1,5 4,5 Natural Sciences 3594 5,4 1,8 4,7 Social Sciences 1882 3,1 Engineering 2157 5,5 2,1 4,4

Main results of the evaluation Three major problems: The weighting model, the balance between major fields (effects of fractioning in case of multi-authorship) The transparency and legitimacy of the nomination processes for level 2 Local use of the model Continuation with increased economic effects

A new interactive portal

Main results of the evaluation Three major problems: The weighting model, the balance between major fields (effects of fractioning in case of multi-authorship) The transparency and legitimacy of the nomination processes for level 2 Local use of the model Continuation with increased economic effects

Poster in the toilets of a Department of Linguistics

Outline CRIS systems and performance-based funding – a European overview The evaluation of the CRIS-based publication indicator in Norway Measuring research responsibly with CRIS

Why is the indicator not applicable on the level of individuals? The indicator disregards the sequence of authors and their individual contributions The relevance of publication channels on the highest level may differ between individual projects Individual productivity depends on many factors that are not taken into consideration when publications are counted Individual contributions deserve to be understood and valued – not just to be counted

Why may the indicator be valuable on aggregated levels? Information: Overview of and insight into research activities at the organizational and national level Indicators: Statistics for monitoring developments in research Incentives: Motivating organizations to support their research activities in the best possible way

The “Leiden Manifesto” The 10 principles can be summarized as follows: Quantitative indicators cannot replace the judgment of expert assessors, but they can be used to help support them. Evaluation of research activity has to adapt to the mission and objectives of the institution, individual or group being evaluated. Indicators need to be developed that reflect the impact of research activities locally and regionally, and those that are developed in languages other than English. The data collection and analysis processes have to be open, transparent and simple. Those evaluated have to be able to verify the analysis of the indicators being used for the evaluation and, if they disagree, request re-evaluation. The differences existing in terms of impact in different fields of research have to be taken into account when producing indicators. Individual evaluation of researchers has to be based on qualitative assessment of their portfolio. Indicators cannot be used without taking into account the researcher’s context. False precision and misplaced concreteness must be avoided. The effects of certain indicators as incentives for certain activities and disincentives for others must be taken into account. The indicators have to be reviewed and updated regularly.

The “Leiden Manifesto” Principle 7 Individual evaluation of researchers “The older you are, the higher your h-index, even in the absence of new papers. The h-index varies by field: life scientists top out at 200; physicists at 100 and social scientists at 20–30. It is database dependent: there are researchers in computer science who have an h-index of around 10 in the Web of Science but of 20–30 in Google Scholar.”

Elsevier’s Snowball Metrics “The older you are, the higher your h-index, even in the absence of new papers. The h-index varies by field: life scientists top out at 200; physicists at 100 and social scientists at 20–30. It is database dependent: there are researchers in computer science who have an h-index of around 10 in the Web of Science but of 20–30 in Google Scholar.”

Journals and conferences Journals (selected) Scientometrics Journal of Informetrics Journal of the American Association of Information Science and Technology Research Evaluation Research Policy Science and Public Policy Conferences (selected) ISSI (International Society for Scientometrics and Informetrics) STI ENID (Science & Technology Indicators) Nordic Bibliometric Workshop

Courses in bibliometrics The European Summer School for Scientometrics Next year in Berlin The bibliometric courses at CWTS, Leiden University

Conclusions? No, let us discuss after the coffee break CRIS systems and performance-based funding – a European overview The evaluation of the CRIS-based publication indicator in Norway Measuring research responsibly with CRIS