1. www.nifustep.no NIFU STEP Studies in Innovation, Research and Education The Norwegian model and the indicators chosen Gunnar Sivertsen Norwegian Institute for Studies in Innovation, Research, and Education

2. Overview of this presentation 1.Performance based funding 2.Bibliometric datasources 3.An overview of models 4.Definition and delimitation of data 5.Comparable measurement (the publication indicator) 6.The effects so far 7.Conclusions

3. Overview of this presentation 1.Performance based funding 2.Bibliometric datasources 3.An overview of models 4.Definition and delimitation of data 5.Comparable measurement (the publication indicator) 6.The effects so far 7.Conclusions

4. ”The dual funding system” (A simplified model excluding international sources, etc.) Government Direct grants Historical, Political Strategic Performance based Higher Education Institutions Research Councils Projects and Programmes Competition Research Evaluation

5. Same aims, methods, and effects? Government Direct grants Historical, Political Strategic Performance based Higher Education Institutions Research Councils Projects and Programmes Competition Research Evaluation

6. Motivation in Norway: The need for more emphasis on research in the overall funding model for the Higher Education Sector Budget for HES Basic: 60%Education: 25%Research: 15% StrategicPerformance based Scholarly publications Dissertations External funding Since 2006: “Publication points” “Study points” since 2002

7. The model in Norway Panel evaluation, rankings External funding  Peer reviewed  Contract research, etc. Dissertations Metric indicators  Publication output  Citations  Patents Level of analysis:  Individuals  Groups  Programmes  Departments  Institutions  Countries

8. The publication indicator was developed in 2003-2004 for the Government by the Norwegian Association of Higher Education Institutions The Norwegian Higher Education Sector: The Norwegian Government required: 17,000 researchers… 10,000 scientific and scholarly publications annually… Collect complete, standardized, verifiable and analyzable data for all publications! From theology to geology, from internal medicine to international relations, from macroeconomics to nano-technology… Weight the publications in a comparable way that at the same time stimulates research of good quality! Seven universities and forty-seven other HE institutions with different missions and subject areas… Make the sector agree on the model!

9. Overview of this presentation 1.Performance based funding 2.Bibliometric datasources 3.An overview of models 4.Definition and delimitation of data 5.Comparable measurement (the publication indicator) 6.The effects so far 7.Conclusions

10. The standard (so far) database for bibliometric studies

11. Another starting point: Institutional information systems

12. Stockholms universitets publikationer But publishing is a shared activity in shared publication channels

13. Göteborgs universitets publikationer This publication comes from Lund, Stockholm and Gothenburg

14. Another challenge: Sociologisk Forskning is indexed by ISI, but Statsvetenskaplig tidsskrift is not

15. Publication types in five major fields Based on 33.000 (fractionalized) Norwegian publications from 2005-2008

16. Coverage of all scientific publications in Norway’s HE sector Based on 33.000 (fractionalized) publications from 2005-2008

17. From ISI From Norart One shared national database for all universities. Imported references from bibliographic data sources are validated by researchers or their assistants. Missing references are added. Books

18. Principles behind the use of institutional data on a national level Completeness: All scholarly publications should be included Simplicity: All scholars can understand the measurement and its effects Transparency: Every institution can see and check all other institutions’ data. The national database is also online and open to society at large. Participation: The indicator is developed and maintained in collaboration between the institutions and the authorities Multiple use of the data: CV’s, applications, evaluations, annual reports, internal administration, bibliography for Open Archives, links to full text, etc.

19. The next step: Norwegian Science Index from 2011 An open nation-wide database of publications from  The Higher Education Sector  The Institute Sector  The Hospital Sector Co-authored publications will not be duplicated, but shown as a common publication from two or more institutions References in the database may lead on to the full texts of publications Will be used by the Research Council instead of individual publication lists in  Applications  Evaluations

20. The next step: Norwegian Science Index from 2011 An open nation-wide database of publications from  The Higher Education Sector  The Institute Sector  The Hospital Sector Co-authored publications will not be duplicated, but shown as a common publication from two or more institutions References in the database may lead on to the full texts of publications Will be used by the Research Council instead of individual publication lists in  Applications  Evaluations Will be part of a new national Current Research Information System in Norway (CRISTIN)

21. Overview of this presentation 1.Performance based funding 2.Bibliometric datasources 3.An overview of models 4.Definition and delimitation of data 5.Comparable measurement (the publication indicator) 6.The effects so far 7.Conclusions

22. Models for performance-based funding of research institutions Panel evaluation, ranking Bibliometric models All disciplines; all publications Selected disciplines; Citation databases N GB B DK S SF ? AUS

23. Overview of this presentation 1.Performance based funding 2.Bibliometric datasources 3.An overview of models 4.Definition and delimitation of data 5.Comparable measurement (the publication indicator) 6.The effects so far 7.Conclusions

24. The three components of the model 1:Delimitation and documentation 2: Comparable measurement 3: Incentives and funding A system of ”weigths” takes into consideration different publishing traditions and make them comparable in the same measurement of ”Publication points” at the level of institutions A definition defines, and the publication database records and validates, structured and standardized bibliographic references to different types of scholarly publications in all fields of research. Author names and addresses are at the same time connected to persons and institutions. In the overall budget each year, a certain percentage of the basic annual funding from the Government is redistributed between the institutions according to their shares in the total Publication points.

25. Delimitation of publication data by definition Definition A scientific or scholarly publication must: 1.present new insight 2.in a form that allows the research findings to be verified and/or used in new research activity 3.in a language and with a distribution that makes the publication accessible for a relevant audience 4.in a publication channel with peer review*) *) In addition: Publication channels with authors coming mainly from only one institution are not included.

26. Three main publication types Channel Type E.g. scientific article in Nature or scholarly ISBN- title on Oxford University Press. 1) Article in ISSN-title 2) Article in ISBN-title 3) ISBN-title

27. Dynamic records of scientific and scholarly ISSN-titles og ISBN- publishers 0001-9887Africa Today 0001-9909African Affairs 1062-4783African American Review 0263-0338African Archaeological Review 0001-9933African Arts 1017-6772African Development Review 0145-2258African Economic History 1021-3589African Entomology 1472-5843African Identities 1684-5315African Journal of Biotechnology 1021-9730African Journal of Crop Science 0141-6707African Journal of Ecology 1438-7890 African Journal of Environmental Assessment and Management 1684-5378 African Journal of Food, Agriculture, Nutrition 1015-8618African Journal of Neurological Sciences 0065-4000African Literature Today CABI Publishing Cambridge University Press Cappelen Akademisk Forlag Carfax Publishing Carl Heymanns Verlag Carlsson bokförlag Catholic University of America Press Central European University Press Channel View Publications Chinese University Press Christian Ejlers´s Forlag 18,000 ISSN-titles and 1,000 book publishers so far

28. All channels can be searched, and new channels can be suggested

29. Overview of this presentation 1.Performance based funding 2.Bibliometric datasources 3.An overview of models 4.Definition and delimitation of data 5.Comparable measurement (the publication indicator) 6.The effects so far 7.Conclusions

30. Dynamic records of scientific and scholarly ISSN-titles og ISBN- publishers 0001-9887Africa Today 0001-9909African Affairs 1062-4783African American Review 0263-0338African Archaeological Review 0001-9933African Arts 1017-6772African Development Review 0145-2258African Economic History 1021-3589African Entomology 1472-5843African Identities 1684-5315African Journal of Biotechnology 1021-9730African Journal of Crop Science 0141-6707African Journal of Ecology 1438-7890 African Journal of Environmental Assessment and Management 1684-5378 African Journal of Food, Agriculture, Nutrition 1015-8618African Journal of Neurological Sciences 0065-4000African Literature Today CABI Publishing Cambridge University Press Cappelen Akademisk Forlag Carfax Publishing Carl Heymanns Verlag Carlsson bokförlag Catholic University of America Press Central European University Press Channel View Publications Chinese University Press Christian Ejlers´s Forlag 18,000 ISSN-titles and 1,000 book publishers so far

31. Publication channels on two levels Level 2: Higher points to 20 % of the publications Level 1: Normal points to 80 % of the publications Level 2 represents the most prestigous channels and is defined in collaboration with the national councils in each field

32. Differentiated criteria for nomination to level 2 Physics Economics Literature Only most prestigous high-impact journals Leading journals, series and book publishers with authors from several countries

33. Publication points Publication typeLevel 1Level 2 Article in ISSN-title13 Article in ISBN-title0,71 ISBN-title58 Publication Author 1 Institution A Author 2 Institution B Institution C Author 3 Institution C Publication points are fractionalized if the publication originates from more than one institution

34. A comparison of the faculties for the Humanities and for Medicine at the University of Oslo (2005)

35. Institutional research profiles in three different universities (based on publication points 2005-2008)

36. Overview of this presentation 1.Performance based funding 2.Bibliometric datasources 3.An overview of models 4.Definition and delimitation of data 5.Comparable measurement (the publication indicator) 6.The effects so far 7.Conclusions

37. Publication points in Norway’s Higher Education Sector 2004-2009

38. Shares in the world’s scientific output (ISI 2000-2009) Source: National Science Indicators (NSI), Thomson Reuters

39. Effects on the institutional level  The institutions have stronger incentives to facilitate research for their researchers  Research is now perceived as a common and institutional responsibility, not only as an individual task  New publications receive attention, not only from external peers, but also internally from the institution  Research management improves with the aid of complete bibliometric information about the research activities

40. An example of effect at department level: Poster in the toilets of The Centre of Excellence in linguistics at the University of Tromsø

41. Overview of this presentation 1.Performance based funding 2.Bibliometric datasources 3.An overview of models 4.Definition and delimitation of data 5.Comparable measurement (the publication indicator) 6.The effects so far 7.Conclusions

42. Electronic information is here to stay

43. Differences in publication patterns can be recognized

44. Incomplete representations create tensions within universities

45. Complete representations support transparence, visibility, and comparability