1. Bibliometrics and Current Research Information Systems (CRIS)
Gunnar Sivertsen
Nordic Institute for Studies in Innovation, Research and Education, Oslo, Norway

2. Outline
CRIS systems and performance-based funding – a European overview
The evaluation of the CRIS-based publication indicator in Norway
Measuring research responsibly with CRIS

3. Outline
CRIS systems and performance-based funding – a European overview
The evaluation of the CRIS-based publication indicator in Norway
Measuring research responsibly with CRIS

4. Current research information systems (CRIS)
“A current research information system (CRIS) is a database or other information system to store and manage data about research conducted at an institution.”
(Wikipedia)

5. Current research information systems (CRIS)
“A current research information system (CRIS) is a database or other information system to store and manage data about research conducted at an institution.”
May also serve the purpose of information

6. Current research information systems (CRIS)
“A current research information system (CRIS) is a database or other information system to store and manage data about research conducted at an institution.”
May also serve the purpose of information
and statistics

7. Current research information systems (CRIS)
“A current research information system (CRIS) is a database or other information system to store and manage data about research conducted at an institution.”
May also serve the purpose of information
and statistics
Data may be integrated at the national level

8. National integration of data - the Czech solution

9. National integration of data - the Danish solution
Each university has a local Pure system.
Annually, data from the local systems are exported to, and integrated in, a national database owned by the Government

10. The CRIS can itself be nationally integrated – Norway’s solution
Principles behind the use of institutional data in a shared national Current Research Information System
Completeness: All scholarly publications and other results from research are included
Transparency: Every institution can see and check all other institutions’ data. The national database is also online and open to society at large.
Multiple use of the data: CV’s, applications, evaluations, annual reports, internal administration, bibliography for Open Archives, links to full text, etc.

11. The VIRTA solution Previous data collection
VIRTA publication information service
Data sources
Local CRISes of HEIs
Local CRISes of HEIs, university hospitals, state research institutes
Updates
Annually
Automatically and real time (statistics compiled once a year)
Data format
CSV
XML
Corrections, supplementations
Not possible after annual data collection
Possible any time
Temporal coverage
Since 2011
All data from previous years to present can be transferred
Identification of authors
Name
Name, ORCID
Identification of duplicates, faults etc.
Identified manually
Identified automatically and real time
Services connected
JUULI, Vipunen
JUULI, Vipunen + researchers’ reporting for funding agencies, visualization services, other research services etc.

12. Commercial CRIS-solutions for institutional purposes have become widespread

13. Within or between institutions?
Open data?
Comparability?
Transparency of methods?
Agreement on indicators?
Collaboration or competition?

15. The EUROCRIS conference 2016, St. Andrews, Scotland
The first keynote on metrics
The first euroCRIS track on metrics

16. CRIS and indicators for performance-based funding of research institutions
Bibliometrics based on complete data is translated into funding.
Often combined with other indicators.

17. Diffusion of the “Norwegian model”
Bibliometrics based on complete data is translated into funding.
Often combined with other indicators.

18. The «Norwegian model» at Swedish universities

19. Nordic collaboration supported by Nordforsk
A standardized Nordic register of scholarly publication channels
Analysis of CRIS-data

20. Outline
CRIS systems and performance-based funding – a European overview
The evaluation of the CRIS-based publication indicator in Norway
Measuring research responsibly with CRIS

21. Evaluation published 16. January 2014
Performed in 2013 by the The Danish Centre for Studies in Research and Research Policy, Aarhus University, Denmark
Commisioned by the Norwegian Association of Higher Education Institutions
Financed by the Norwegian Ministry of Education and Research
Based on surveys, interviews and bibliometrics
Has a 10 page summary in English

22. The basis for four scientific publications
Bloch, C., & Schneider, J. W. (2016). Performance-based funding models and researcher behavior: An analysis of the influence of the Norwegian Publication Indicator at the individual level. Research Evaluation, 25 (4):
Schneider, J.W., Aagaard, K., Bloch, C. (2016). What happens when national research funding is linked to differentiated publication points?
A comparison of the Australian and Norwegian publication-based
models. Research Evaluation 25, (3):
Aagaard, K. (2015). How incentives trickle down: Local use of a national bibliometric indicator system. Science and Public Policy, 42(5),
Aagaard, K., Bloch, C., & Schneider, J. W. (2015). Impacts of performance-based research funding systems: The case of the Norwegian Publication Indicator. Research Evaluation, 24(2),

23. ”The dual funding system” (Public sources)
Government
Direct grants
Historical, Political
Strategic
Performance based
Research institutions
Research Councils
Projects and Programmes
Competition
Research Evaluation

24. Direct grants «The Norwegian model» Historical, Political Strategic
The model has three components:
A complete representation in a national database of structured, verifiable and validated bibliographical records of the peer-reviewed scholarly literature in all areas of research;
A publication indicator with a system of weights that makes field-specific publishing traditions comparable across fields in the measurement of “Publication points” at the level of institutions;
A performance-based funding model which reallocates a small proportion of the annual direct institutional funding according the institutions’ shares in the total of Publication points.
Direct grants
Historical, Political
Strategic
Performance based

25. Books Journals and series
Indexed journals
Incomplete coverage of international journals in the SSH.
Very limited coverage of books.
Random or no coverage of the national level (books and journals)

26. Publication points Publication type Level 1 Level 2
Article in ISSN-title 1 3
Article in ISBN-title
0,7
ISBN-title 5 8
Publication
Author 1
Institution A
Author 2
Institution B
Institution C
Author 3
Publication points are fractionalized if the publication originates from more than one institution

27. Publication channels on two levels
Level 2 represents the most prestigous channels and is
defined in collaboration with the national councils in each field
Level 2: Higher points to 20 % of the publications
Level 1: Normal points to 80 % of the publications

28. Main results of the evaluation
Increased publishing – beyond the increased funding during the period
An increased number of researchers are publishing
Publication patterns remain stable:
Type of publications, language of publications
“Quality” does not go up or down: Citation rates and share of publications on level 2 are stable

29. Main results of the evaluation
Three major problems:
The weighting model, the balance between major fields (effects of fractioning in case of multi-authorship)
The transparency and legitimacy of the nomination processes for level 2
Local use of the model
Continuation with increased economic effects

30. Main results of the evaluation
Three major problems:
The weighting model, the balance between major fields (effects of fractioning in case of multi-authorship)
The transparency and legitimacy of the nomination processes for level 2
Local use of the model
Continuation with increased economic effects

31. The lack of balance: Average publication points among researchers in five major areas of research

32. Research profiles of three Norwegian universities Based on Norwegian CRIS data

33. Calibrating for a better balance
Data on scholarly publishing from CRIStin (Current Research Information System in Norway)
Years
Nearly 15,000 active researchers (contributing to a minimum of two publications in two years)
Simulation of average number of publication points in five areas

34. The present model
Publication type
Level 1
Level 2
Article in ISSN-title 1 3
Article in ISBN-title
0,7
ISBN-title 5 8
Changing the weights does not create a better balance
Publication
Author 1
Institution A
Author 2
Institution B
Institution C
Author 3
Publication points are fractionalized if the publication originates from more than one institution
Publication points may then be multiplied by a certain factor if co-authors from other countries are involved

35. The former model
Publication type
Level 1
Level 2
Article in ISSN-title 1 3
Article in ISBN-title
0,7
ISBN-title 5 8
Publication
Author 1
Institution A
Author 2
Institution B
Institution C
Author 3
Publication points are fractionalized if the publication originates from more than one institution
Introducing or increasing this factor contributes to a better balance
Publication points may then be multiplied by a certain factor if co-authors from other countries are involved

36. The present model
Publication type
Level 1
Level 2
Article in ISSN-title 1 3
Article in ISBN-title
0,7
ISBN-title 5 8
Publication
Author 1
Institution A
Author 2
Institution B
Institution C
Author 3
Publication points are fractionalized if the publication originates from more than one institution
Fractionalization has the largest effect on the problem/solution
Publication points may then be multiplied by a certain factor if co-authors from other countries are involved

37. The new model (Norway) Publication type Level 1 Level 2
Article in ISSN-title 1 3
Article in ISBN-title
0,7
ISBN-title 5 8
Publication
Author 1
Institution A
Author 2
Institution B
Institution C
Author 3
The square root of the institution’s fraction is used instead of simply the fraction
Publication points are multiplied by a factor of 1,3 if co-authors from other countries are involved
The total number of unique author-institution relations (4 in this example), not the total number of authors (3 in this example), are counted

38. Balancing research productiviy across all fields
Area of research
Researchers
Publications in two years
Publication points (now)
Publication points (new)
Humanities
1074
3,7
3,9
4,6
Health Sciences
5724
6,3
1,5
4,5
Natural Sciences
3594
5,4
1,8
4,7
Social Sciences
1882
3,1
Engineering
2157
5,5
2,1
4,4

40. Main results of the evaluation
Three major problems:
The weighting model, the balance between major fields (effects of fractioning in case of multi-authorship)
The transparency and legitimacy of the nomination processes for level 2
Local use of the model
Continuation with increased economic effects

41. A new interactive portal

42. Main results of the evaluation
Three major problems:
The weighting model, the balance between major fields (effects of fractioning in case of multi-authorship)
The transparency and legitimacy of the nomination processes for level 2
Local use of the model
Continuation with increased economic effects

43. Poster in the toilets of a Department of Linguistics

44. Outline
CRIS systems and performance-based funding – a European overview
The evaluation of the CRIS-based publication indicator in Norway
Measuring research responsibly with CRIS

45. Why is the indicator not applicable on the level of individuals?
The indicator disregards the sequence of authors and their individual contributions
The relevance of publication channels on the highest level may differ between individual projects
Individual productivity depends on many factors that are not taken into consideration when publications are counted
Individual contributions deserve to be understood and valued – not just to be counted

46. Why may the indicator be valuable on aggregated levels?
Information: Overview of and insight into research activities at the organizational and national level
Indicators: Statistics for monitoring developments in research
Incentives: Motivating organizations to support their research activities in the best possible way

49. The “Leiden Manifesto” The 10 principles can be summarized as follows:
Quantitative indicators cannot replace the judgment of expert assessors, but they can be used to help support them.
Evaluation of research activity has to adapt to the mission and objectives of the institution, individual or group being evaluated.
Indicators need to be developed that reflect the impact of research activities locally and regionally, and those that are developed in languages other than English.
The data collection and analysis processes have to be open, transparent and simple.
Those evaluated have to be able to verify the analysis of the indicators being used for the evaluation and, if they disagree, request re-evaluation.
The differences existing in terms of impact in different fields of research have to be taken into account when producing indicators.
Individual evaluation of researchers has to be based on qualitative assessment of their portfolio. Indicators cannot be used without taking into account the researcher’s context.
False precision and misplaced concreteness must be avoided.
The effects of certain indicators as incentives for certain activities and disincentives for others must be taken into account.
The indicators have to be reviewed and updated regularly.

50. The “Leiden Manifesto” Principle 7 Individual evaluation of researchers
“The older you are, the higher your h-index, even in the absence of new papers.
The h-index varies by field: life scientists top out at 200; physicists at 100 and social scientists at 20–30.
It is database dependent: there are researchers in computer science who have an h-index of around 10 in the Web of Science but of 20–30 in Google Scholar.”

51. Elsevier’s Snowball Metrics
“The older you are, the higher your h-index, even in the absence of new papers.
The h-index varies by field: life scientists top out at 200; physicists at 100 and social scientists at 20–30.
It is database dependent: there are researchers in computer science who have an h-index of around 10 in the Web of Science but of 20–30 in Google Scholar.”

52. Journals and conferences
Journals (selected)
Scientometrics
Journal of Informetrics
Journal of the American Association of Information Science and Technology
Research Evaluation
Research Policy
Science and Public Policy
Conferences (selected)
ISSI (International Society for Scientometrics and Informetrics)
STI ENID (Science & Technology Indicators)
Nordic Bibliometric Workshop

53. Courses in bibliometrics
The European Summer School for Scientometrics
Next year in Berlin
The bibliometric courses at CWTS, Leiden University

54. Conclusions? No, let us discuss after the coffee break
CRIS systems and performance-based funding – a European overview
The evaluation of the CRIS-based publication indicator in Norway
Measuring research responsibly with CRIS