NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education 7 th euroCRIS strategic seminar, Brussels Recording Research Use in research management and evaluations Dag W. Aksnes

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Background NIFU STEP is a Norwegian research institute doing contract research, analyses and consultancy for various organisations such as ministries and research councils. The main focus is on studies of education, research and innovation. One mission to provide knowledge for policy decisions within these areas and analyze the results and effects of policy through e.g., evaluations and studies on productivity. This presentation will focus on how CRIS data have been used in Norway in relation to research management and evaluations.

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education CRIS-data Norway Data on all types of scientific/scholarly publications in all fields of research in the Higher Education Sector are collected trough two documentation systems (Frida, ForskDOK). In the database the publication output is reported by the institutions as ordinary bibliographic references. A dynamic authority record of so far 19,000 controlled scientific and scholarly publication channels ensures that references to non- scientific publications are not entered into the system. Publication data from professional bibliographic data sources (e.g. Web of Science) are imported to the documentation system in order to facilitate the registration of publications by the employees.

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Relevance of bibliometics in science policy/evaluations Number of publications used as an indirect measure of knowledge production The extent to which the publications have been cited in the subsequent scientific literature used as an indicator scientific impact and international visibility. Co-authorship used as an indicator of collaboration, e.g. the extent of international collaboration The journal pattern used for obtaining information on the subfield distribution of the research activities and the extent of presence in leading journals In turn, such indicators can yield much interesting information on various issues relevant for evaluation purposes.

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Bibliometric analyses of research performance Traditionally, bibliometric analyses of research performance have usually been based on the ISI-database (Web of Science, Thomson Reuters) – Multidisciplinary – Structured data well adapted for bibliometric analyses – Citation counts – Covers mainly publications in international journals In Norway the CRIS-database has now been used as an additional source for bibliometric analyses The CRIS database is well suited for this purpose: – The bibliographic references in the documentation system are standardized and analyzable by publication channel and type of publication. – Complete, verifiable and structured data – Covers all scientific/scholarly publications not only journal articles. – Non-scientific/non-scholarly publications can be excluded.

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Using CRIS data for funding purposes and analyses In 2004, Norway implemented a bibliometric model for performance based budgeting of research institutions. – The funding of these institutions is now partially based on the measurement of their scientific and scholarly publishing. – The CRIS databases are used for this purpose. Other applications: analyses of the research output and evaluations

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Bibliometric analyses of research performance Particularly the humanities and the social sciences the ISI-base covers a small proportion of the research output – Due to poor coverage and a publication pattern where international journals are less important. Example: publication types and ISI-coverage in major scientific and scholarly fields:

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Figure 1. Publication types* in five major areas (N=24,154) *) Weighted: 1 book = 5 articles

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Figure 2. Publication types* in social science disciplines (N=5,616) *) Weighted: 1 book = 5 articles

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Research evaluations In Norway, the Research Council of Norway regularly carries out evaluations of the national research activities in the various disciplines (peer evaluation by international panels). NIFU STEP provides bibliometric analyses as background information for the evaluations. Till quite recently all researchers encompassed by the evaluations have submitted their publication lists. – Bibliometric analyses have been carried out using this as a data source in combination with ISI-data – Time consuming – No structured data Now the CRIS-database has instead been used as a data source – The researchers do not have to submit their publication lists anymore

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education ”Level 1 and level 2” In addition to using CRIS as the main data source, information on the “level” of the journals and book publishers are used in the analyses In the Norwegian system journals and publishers are divided into two levels. The highest level (level 2) is giving extra weight and includes only the leading and most selective international journals and publishers (account for about 20 % of the world’s publications), The national councils in each discipline participate annually in determining and revising the highest level under the guidance of the Norwegian Association of Higher Education Institutions.

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Examples The following slides show some examples of indicators and analyses provided. Levels of analyses – Country – Institutions – Departments – Divisions/research groups – (Individuals – anonymous presentation)

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Figure 3. Number and proportion of articles in leading Chemistry journals – “level 2”, Norway *) Weighted: 1 book = 5 articles

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Figure 4. Number of publications per year, , fractionalised counts. Department X. *) Weighted: 1 book = 5 articles

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Figure 5. Department x. Number of publications per person , fractionalised counts.

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Table 1. Number of publications , Department X Unit Number of persons incl. Publications - whole counts Publications - fractional counts Number of publications - fractional counts per person TOTAL Environment Organic Chemistry Physical Chemistry

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Table 2. Journal profile, publications (whole counts). Department X Unit Numb. of persons incl. Numb. of articles - level 1 Numb. of articles - level 2 Prop. of articles - level 2 Avg. numb. of articles - level 2 per person Avg. journal citation rate (impact factor) TOTAL % Environment % Organic Chemistry % Physical Chemistry %5.45.9

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Table 3 The most frequently used journals, number of publications, , by groups/sections, Department X Group/sectionJournal Numb. of articles Journal citation rate (impact factor)* Lev el Electrochemistry CORROSION SCIENCE JOURNAL OF THE ELECTROCHEMICAL SOCIETY JOURNAL OF APPLIED ELECTROCHEMISTRY Inorganic Chemistry JOURNAL OF THE AMERICAN CERAMIC SOCIETY JOURNAL OF LIGHT METALS13 –1 SOLID STATE IONICS94.5 1

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Other examples Analyses of the publication output at the University of Bergen The University Library has initiated a project providing analyses of the publication output at the university based on CRIS data Writing of reports being presented for the management at the university and departments Examples of indicators provided

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Figure 6. Number of publication points per researcher man year per department

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Proportion of publication points in ”level 2” channels Enhet Average Matematisk institutt 30 %31 %28 %21 %27 % Institutt for informatikk 12 %20 %27 %19 %20 % Institutt for fysikk og teknologi 38 %28 %39 %33 %34 % Kjemisk institutt 46 %68 %49 %45 %52 % Molekylærbiol ogisk institutt 44 % 1 %7 %24 % Institutt for geovitenskap 26 %35 %28 %26 %29 % Geofysisk institutt 11 %25 %14 %20 %17 % Institutt for biologi 19 %18 %22 %20 % Total:24 %29 %30 %24 %27 %

NIFU STEP Norwegian Institute for Studies in Innovation, Research and Education Conclusions Data quality of CRIS crucial issue for using the data for these purposes – Accurate and complete – Delimitation of scientific/scholarly publications – Use of authority registers, person and publication IDs (incl. ISI-ID) – Importing of data from professional bibliographic data sources, e.g. Web of science – In a format adapted for bibliometric analyses The imported records have highest quality – although these data have imperfections as well – Mistakes more often occur for the non-imported records registered by the researchers, e.g. Duplications Editorials, corrections etc. registered as full articles Mistakes in the title and journal title Lacking information on co-authors