BIBLIOMETRICS – USE AND LIMITATIONS Wolfgang Glänzel KU Leuven, Belgium ISPR, HAS, Hungary

Wolfgang GlänzelBibliometrics. Use and limitations2/37

Wolfgang GlänzelBibliometrics. Use and limitations3/37 1. INTRODUCTION

Wolfgang GlänzelBibliometrics. Use and limitations4/37 1.Introduction

Wolfgang GlänzelBibliometrics. Use and limitations5/37 1.Introduction

Wolfgang GlänzelBibliometrics. Use and limitations6/37 1.Introduction

Wolfgang GlänzelBibliometrics. Use and limitations7/37 1.Introduction

Wolfgang GlänzelBibliometrics. Use and limitations8/37 1.Introduction

Wolfgang GlänzelBibliometrics. Use and limitations9/37 2.SCOPE AND STRUCTURE OF BIBLIOMETRICS

Wolfgang GlänzelBibliometrics. Use and limitations10/37 Structure of Bibliometrics 2. Structure of Bibliometrics

Wolfgang GlänzelBibliometrics. Use and limitations11/37 Structure of Bibliometrics 2. Structure of Bibliometrics Webometrics Informetrics Mathematics/Physics Library and Information Science Sociology of science History of science Economics Scientific information Services for Research in Librarianship Science policy Research management Scientometrics applied basic Life sciences Links of bibliometrics with related research fields and application services

Wolfgang GlänzelBibliometrics. Use and limitations12/37 3.DATA SOURCES OF BIBLIOMETRIC RESEARCH

Wolfgang GlänzelBibliometrics. Use and limitations13/37 Data sources of bibliometrics are bibliographies and bibliographic databases. Large scale analyses can only be based on bibliographic databases. Despite objections to journal coverage and data processing policy of the ISI, the databases of the Institute for Scientific Information (Thomson - ISI), above all, the Science Citation Index (Expanded) have become the most generally accepted data source. Recently several countries have developed their own national citation indexes which are also used (as supplementary) data sources. Sources of Bibliometrics 3. Sources of Bibliometrics

Wolfgang GlänzelBibliometrics. Use and limitations14/37 Sources of Bibliometrics 3. Sources of Bibliometrics

Wolfgang GlänzelBibliometrics. Use and limitations15/37 4.“PERSPECTIVE SHIFT” IN USE OF BIBLIOMETRICS

Wolfgang GlänzelBibliometrics. Use and limitations16/37 Perspective Shift 4. Perspective Shift Bibliometrics deals with the quantitative analysis of the (social) network of scientific communication. From the historical perspective, information scientists and bibliometricians proceeded from the same approach. First citation analyses were designed to aid the decision which periodicals should best be purchased by small libraries. Many tools and indicators developed by the ISI were designed for use in scientific information, information retrieval and libraries.  The application to science policy has brought a new perspective, and resulted in re-interpretation of bibliometric conceptions.

Wolfgang GlänzelBibliometrics. Use and limitations17/37 Perspective Shift 4. Perspective Shift Example: The notion of citations in information science, bibliometrics and science policy Citations became a widely used measure of the impact of scientific publications, but the notion of what citations actually express sheds light on different aspects. These different views might results in conflicts caused by re-interpretation and “perspective shift”.

Wolfgang GlänzelBibliometrics. Use and limitations18/37 Some examples: According to Merton, citations are intimately connected with the reward system of science. Cozzens: “Citation is only secondarily a reward system. Primarily, it is rhetorical-part of persuasively arguing for the knowledge claims of the citing document.” Cronin: Citations are "frozen footprints in the landscape of scholarly achievement … which bear witness to the passage of ideas“. Glänzel and Schoepflin: Citations give “a formalised account of the information use and can be taken as a strong indicator of reception at this level.” Perspective Shift 4. Perspective Shift

Wolfgang GlänzelBibliometrics. Use and limitations19/37 The process of re-interpreting the notion of citation and its consequences Information use Reward system (quality measure) Research evaluation/ Science policy Bibliometrics/ Information science citation uncitedness: unused information frequent cite: good reception self-cite: part of scient. communication interpretation re-interpretation repercussion (possible distortion of citation behaviour) uncitedness: low quality frequent cite: high quality self-cite: distortion of impact Perspective Shift 4. Perspective Shift

Wolfgang GlänzelBibliometrics. Use and limitations20/37 5.BIBLIOMETRICS AS A TOOL IN RESEARCH EVALUATION

Wolfgang GlänzelBibliometrics. Use and limitations21/37 Research Evaluation 5. Research Evaluation

Wolfgang GlänzelBibliometrics. Use and limitations22/37 What can bibliometrics not be responsible for? Bibliometrics can be used to develop and provide tools to be applied to research evaluation, but is not designed to evaluate research results. Bibliometrics does not aim at replacing qualitative methods by quantitative approaches. Consequently, bibliometrics is not designed to correct or even substitute peer reviews or evaluation by experts but qualitative and quantitative methods in science studies should complement each other. Research Evaluation 5. Research Evaluation

Wolfgang GlänzelBibliometrics. Use and limitations23/37 Distorted behaviour based on policy use and misuse of bibliometric data One issue concerns the changes in the publication, citation and collaboration behaviour of scientists (both positive and negative) that the consistent policy use of bibliometric indicators might potentially induce. Uninformed use and misuse are not always beyond the responsibility of bibliometricians. Unfortunately, bibliometricians do not always resist the temptation to follow popular, even populist, trends in order to meet the expectations of the customers. Any kind of uninformed use or misuse of bibliometric results involves the danger of bringing bibliometric research itself into disrepute. Research Evaluation 5. Research Evaluation

Wolfgang GlänzelBibliometrics. Use and limitations24/37 The problem of inappropriate use ranges from uninformed use, over selecting and collecting ‘most advantageous’ indicators to the obvious and deliberate misuse of data. Uninformed use incorrect presentation, interpretation of bibliometric indicators or their use in an inappropriate context caused by insufficient knowledge of methodology, background and data sources Misuse intentionally incorrect presentation, interpretation of bibliometric indicators or their deliberate use in inappropriate context tendentious application of biases or tendentious choice of (incompatible) indicators Research Evaluation 5. Research Evaluation

Wolfgang GlänzelBibliometrics. Use and limitations25/37 Research Evaluation 5. Research Evaluation

Wolfgang GlänzelBibliometrics. Use and limitations26/37 Schematic visualisation of the feedback of policy use of bibliometrics on the scientific community Research Evaluation 5. Research Evaluation

Wolfgang GlänzelBibliometrics. Use and limitations27/37 Possible positive effects Scientists might recognise that scientific collaboration and publishing in high-impact or even top journals pays off. Also their publication activity might be stimulated. Possible negative effects Exaggerated collaboration, even trends towards hyper- authorship, inflating publication output by splitting up publications to sequences, inflating citation impact by self- citations and forming citation cliques, etc. Trend towards replacing quality and recognition by visibility at any price or towards preferring journals as publication channels in social sciences and humanities might be among these effects. A certain “champions league” mentality is spreading among scientists (“Shanghai ranking”, H-index, “IF Filter”). Research Evaluation 5. Research Evaluation

Wolfgang GlänzelBibliometrics. Use and limitations28/37 6.BIBLIOMETRIC RESEARCH AND SERVICES AT THE STEUNPUNT O&O STATISTIEKEN

Wolfgang GlänzelBibliometrics. Use and limitations29/37 The Steunpunt O&O Statistieken (Research Centre for R&D Statistics – SOOS) was created in January 2002 after open calls for proposals. SOOS is a unique and dedicated research center to support its science and innovation policy in Flanders. This center has to develop an appropriate system of quantitative indicators to quantify R&D efforts at Flemish universities, research institutes and industry that can be used by policy makers to support the development of appropriate regional science and innovation policy for Flanders. The Steunpunt O&O Statistieken 6. The Steunpunt O&O Statistieken

Wolfgang GlänzelBibliometrics. Use and limitations30/37 ACTIVITY OVERVIEW The activities of SOOS can be classified into four categories: 1. Developing an appropriate IT-infrastructure to support S&T indicator development; 2.Providing the Flemish government with appropriate S&T indicators on which to base its science and technology policy resource allocation decisions; 3.Executing specific studies on science and technology domains or science policy relevant topics for the Flemish government; 4.Doing original research in the field of S&T studies. The Steunpunt O&O Statistieken 6. The Steunpunt O&O Statistieken

Wolfgang GlänzelBibliometrics. Use and limitations31/37 LONG- AND MEDIUM-TERM SERVICES FOR THE FLEMISH GOVERNMENT I. The BOF-key One of the major funding mechanisms for basic science in Flanders is the BOF-mechanism. The Bijzonder Onderzoeksfonds is a research fund of the Flemish government that is distributed to the Flemish universities on the basis of a specific allocation model. This allocation key comprises both input and output measures on the research performance of the universities involved. Part of the key is based on publication and citation data as derived from WoS by SOOS. The Steunpunt O&O Statistieken 6. The Steunpunt O&O Statistieken

Wolfgang GlänzelBibliometrics. Use and limitations32/37 BOF-funding for Flanders amounts to 100 million EUR annually. This amount is distributed in basic science funding amongst the six Flemish universities (LUC/UHasselt, KUB, KUL, VUB, UA and UGent). II. The IOF-key The IOF-key is a distribution key to determine the annual allocation of 12 million EUR (to be augmented in the year 2006) amongst the six Flemish universities in order to support applied research. The IOF-key is partially based on patent statistics derived from the USPTO, EPO and PCT databases. SOOS has been instrumental both in the development of and the data delivery for those distribution instruments. The Steunpunt O&O Statistieken 6. The Steunpunt O&O Statistieken

Wolfgang GlänzelBibliometrics. Use and limitations33/37 III. The Flemish Indicatorbook The Flemish Indicatorbook on Science, Technology and Innovation appears biannually, and does not only support government questions on R&D- proficiency in Flanders, but also aims to supply adequate and up- to-date information on the European R&D-potential of the Flanders region. Two editions of the Indicatorbook have been published by SOOS so far: 2003 and The Steunpunt O&O Statistieken 6. The Steunpunt O&O Statistieken

Wolfgang GlänzelBibliometrics. Use and limitations34/37 IV. Domain Studies (here bibliometrics only) Aim of these studies is to conduct and to provide analyses of Flemish research and technology activity in the international context in highly important areas. Expected outcomes are relevant information about performance and competitiveness in Flanders. Three domain studies have been prepared and published by SooS: Nanoscience and –technology, Biotechnology and molecular biology (both in 2003) and Stem-cell research (2004). The studies are based on both publication and patent data extracted from the WoS, the USPTO and EPO databases. The Steunpunt O&O Statistieken 6. The Steunpunt O&O Statistieken

Wolfgang GlänzelBibliometrics. Use and limitations35/37 Methodological and applied research is closely linked to the service tasks, for instance, in the context of methodological validation, of improvement of the efficiency of indicators or of the subject classification. ‘Basic research’ at the Steunpunt is structured in a systematic way to support policy-relevant research and services and to contribute to the advancement of the field of bibliometrics. Similarly to the services, research is done at the national/regional, institutional and research-group level. The Steunpunt O&O Statistieken 6. The Steunpunt O&O Statistieken BIBLIOMETRIC RESEARCH PROJECTS AT SOOS

Wolfgang GlänzelBibliometrics. Use and limitations36/37 1.Development of a hierarchic subject classification scheme for evaluation purposes 2.The role of author self-citations in scientific communication 3.The myth of delayed recognition 4.Analysis of possible extension of bibliometric methods to social science and humanities 5.Studying the network of science communication 6.“Indicator research” (improving journal and subject impact measures and science indicators) The Steunpunt O&O Statistieken 5. The Steunpunt O&O Statistieken

Wolfgang GlänzelBibliometrics. Use and limitations37/37 7.Bibliometrics beyond the WoS (e.g., proceedings literature) 8.Bibliometric approach to social sciences and humanities 9.Combining text-mining and bibliometrics 10.Mapping academic mobility and scientific collaboration in Europe 11.Research group dynamics and their impact on performance and visibility 12.Bisociation analysis of scientific communication The Steunpunt O&O Statistieken 6. The Steunpunt O&O Statistieken