1. MEASURING COMMUNICATION IN SCIENCE MEASURING COMMUNICATION IN SCIENCE OPPORTUNITIES AND LIMITATIONS OF BIBLIOMETRIC METHODS Wolfgang Glänzel and Koenraad Debackere SooS, Leuven, Belgium

2. W. Glänzel and K. DebackereMeasuring communication in science2

3. W. Glänzel and K. DebackereMeasuring communication in science3 INTRODUCTION

4. W. Glänzel and K. DebackereMeasuring communication in science4 Introduction What is bibliometrics? The terms bibliometrics and scientometrics were almost simultaneously introduced by Pritchard and by Nalimov & Mulchenko in 1969. According to Pritchard bibliometrics is “the application of mathematical and statistical methods to books and other media of communication”.

5. W. Glänzel and K. DebackereMeasuring communication in science5 Introduction Nalimov and Mulchenko defined scientometrics as “the application of those quantitative methods which are dealing with the analysis of science viewed as an information process”. The two terms have become almost synonyms; nowadays, the field informetrics (Gorkova, 1988) stands for a more general subfield of information science dealing with mathematical-statistical analysis of communication processes in science.

6. W. Glänzel and K. DebackereMeasuring communication in science6 Introduction

7. W. Glänzel and K. DebackereMeasuring communication in science7 Introduction

8. W. Glänzel and K. DebackereMeasuring communication in science8 Introduction

9. W. Glänzel and K. DebackereMeasuring communication in science9 Introduction What is bibliometrics dealing with and what can bibliometrics not be responsible for? Bibliometrics can be used to develop and provide tools to be applied to research evaluation, but is not designed to evaluate research results. Bibliometrics does not aim at replacing qualitative methods by quantitative approaches. Consequently, bibliometrics is not designed to correct or even substitute peer reviews or evaluation by experts but qualitative and quantitative methods in science studies should complement each other.

10. W. Glänzel and K. DebackereMeasuring communication in science10 1.STRUCTURE OF BIBLIOMETRICS

11. W. Glänzel and K. DebackereMeasuring communication in science11 Structure of Bibliometrics 1. Structure of Bibliometrics

12. W. Glänzel and K. DebackereMeasuring communication in science12 Structure of Bibliometrics 1. Structure of Bibliometrics Webometrics Informetrics Mathematics/Physics Library and Information Science Sociology of science History of science Economics Scientific information Services for Research in Librarianship Science policy Research management Scientometrics applied basic Life sciences Links of bibliometrics with related research fields and application services

13. W. Glänzel and K. DebackereMeasuring communication in science13 2.DATA SOURCES OF BIBLIOMETRIC RESEARCH AND TECHNOLOGY

14. W. Glänzel and K. DebackereMeasuring communication in science14 Data sources of bibliometric research and technology Data sources of bibliometrics are bibliographies and bibliographic databases. Large scale analyses can only be based on bibliographic databases. Prominent specialised databases are, e.g., Medline, Chemical Abstracts, INSPEC and Mathematical Reviews in the sciences and, e.g., Econlit, Sociological Abstracts and Humanities Abstracts in the social sciences and humanities.  Disadvantage: Lack of reference literature, incomplete address recording The databases of the Institute for Scientific Information (Thomson - ISI), above all, the Science Citation Index (Expanded) have become the most generally accepted source of bibliometrics. Sources of Bibliometrics 2. Sources of Bibliometrics

15. W. Glänzel and K. DebackereMeasuring communication in science15 Although, there are several objections against the journal coverage and the data processing policy of the ISI in preparing the SCI, its unique features are basic requirements of bibliometric technology. Among these features we have Multidisciplinarity Selectiveness Completeness of addresses Full coverage Bibliographical references  Disadvantage: no individual subject classification for papers available. Sources of Bibliometrics 2. Sources of Bibliometrics

16. W. Glänzel and K. DebackereMeasuring communication in science16 Sources of Bibliometrics 2. Sources of Bibliometrics

17. W. Glänzel and K. DebackereMeasuring communication in science17 3.ELEMENTS, UNITS AND MEASURES OF BIBLIOMETRIC RESEARCH

18. W. Glänzel and K. DebackereMeasuring communication in science18 Elements of Bibliometrics 3. Elements of Bibliometrics Elements, units and measures of bibliometric research Basic units in bibliometrics are usually not further subdivided. These form the elements of bibliometric analyses. Elements are, e.g., publications, (co ‑ )authors, references and citations. Publications can be assigned to the journals in which they appeared, through the corporate addresses of their authors to institutions or countries, references and citations to subject categories, and so on. Units are specific sets of elements, e.g., journals, subject categories, institutions, regions and countries to which elements can – not necessarily uniquely – be assigned. The clear definition of the assignment – or in mathematical parlance – of mappings between elements and units allows the application of mathematical models.

19. W. Glänzel and K. DebackereMeasuring communication in science19 Elements of Bibliometrics 3. Elements of Bibliometrics Publication activity and authorship Publication activity is influenced by several factors. At the micro level, we can distinguish the following four factors. 1.the subject matter 2.the author’s age 3.the author’s social status 4.the observation period The publication activity in theoretical fields (e.g., mathematics) and in engineering is lower than in experimental fields or in the life sciences. Cross-field comparison – without appropriate normalisation – would not be valid. This applies above all to comparative analyses at the meso level (universities and departments).

20. W. Glänzel and K. DebackereMeasuring communication in science20 Elements of Bibliometrics 3. Elements of Bibliometrics Can scientific collaboration be measured through co-authorship? Laudel (2001) (micro study): A large share of persons involved in the preparation of a scientific paper does thus not appear either as co-author or as a sub-author. Katz & Martin (1997) argue that co-authorship is no more than a partial indicator of collaboration. Intensifying collaboration, however, goes with growing co- authorship (Patel, 1973). There is a positive correlation between collaboration and co-authorship at the level of individual actors, too. The phenomenon described by Laudel and Katz & Martin rather applies to intramural collaboration. Extramural collaboration, above all international collaboration, is usually well acknowledged.

21. W. Glänzel and K. DebackereMeasuring communication in science21 4. CITATIONS AND SELF-CITATIONS

22. W. Glänzel and K. DebackereMeasuring communication in science22 Citations and Self-citations 4. Citations and Self-citations The notion of citations in information science and bibliometrics Citations became a widely used measure of the impact of scientific publications. Cozzens: “Citation is only secondarily a reward system. Primarily, it is rhetorical-part of persuasively arguing for the knowledge claims of the citing document.” L. C. Smith: "citations are signposts left behind after information has been utilized". Cronin: Citations are "frozen footprints in the landscape of scholarly achievement … which bear witness to the passage of ideas“.

23. W. Glänzel and K. DebackereMeasuring communication in science23 Citations and Self-citations 4. Citations and Self-citations

24. W. Glänzel and K. DebackereMeasuring communication in science24 Citations and self-citations 4. Citations and self-citations The process of re-interpreting the notion of citation and its consequences Signpost of information use Rewarding system/ Quality measure Research evaluation/ Science policy Bibliometrics/ Information science citation uncitedness: unused information frequent cite: good reception self-cite: part of scient. communication interpretation re-interpretation repercussion (possible distortion of citation behaviour) uncitedness: low quality frequent cite: high quality self-cite: manipulation of impact

25. W. Glänzel and K. DebackereMeasuring communication in science25 5.“AGAINST ABSOLUTE METHODS”

26. W. Glänzel and K. DebackereMeasuring communication in science26 “Against absolute methods” 5. “Against absolute methods” Factors influencing citation impact Citation impact is mainly influenced by the following five factors that are analogously to the case of publication activity at higher levels of aggregation practically quite inseparable. 1.the subject matter and within a subject, the “level of abstraction” 2.the paper’s age 3.the paper’s “social status” (through authors and journal) 4.the document type 5.the observation period (”citation window”)

27. W. Glänzel and K. DebackereMeasuring communication in science27 Complexity of influences and biases in calculating citation impact measures Mean citation rate of two journals in time as a function of time (source year: 1980) absolute methods” 5. “Against absolute methods”

28. W. Glänzel and K. DebackereMeasuring communication in science28 ”3-year impact measure” for selected journals by document types (source: 1995/96) Impact of different document types 5. “Against absolute methods”

29. W. Glänzel and K. DebackereMeasuring communication in science29 Influence of subject characteristics Mean citation rate of subfields (source: 1996, citation window: 1996-1998) Mechanical, civil and other engineering 1.12 Mathematics 1.46 Analytical chemistry 3.00 Solid state physics 3.06 Neurosciences 4.54 Citation measures are thus – without normalisation – not appropriate for cross-field comparisons. “Against absolute methods” 5. “Against absolute methods”

30. W. Glänzel and K. DebackereMeasuring communication in science30  The only possible way to compensate for the subject-specific characteristics is an appropriate normalisation and the application of exactly the same underlying publication period and citation window to all units under study. “Against absolute methods” 5. “Against absolute methods”

31. W. Glänzel and K. DebackereMeasuring communication in science31 “Against absolute methods” 5. “Against absolute methods”

32. W. Glänzel and K. DebackereMeasuring communication in science32 6. JOURNAL IMPACT FACTOR

33. W. Glänzel and K. DebackereMeasuring communication in science33 Journal Impact Factor 6. Journal Impact Factor On the role of the Impact Factor The Garfield ISI Impact Factor (IF) represents a paradigm in bibliometric/information science research. The IF is used frequently and has obtained a very strong ‘market’ position. From the mathematical viewpoint, the IF is the mean value, i.e., an arithmetic mean of citations in a particular (citing) year to a particular set of articles published in a particular journal one or two years earlier. The Impact Factor has become perhaps the most popular bibliometric product used in bibliometrics and outside the scientific community.

34. W. Glänzel and K. DebackereMeasuring communication in science34 Journal Impact Factor 6. Journal Impact Factor Problems in using the ISI Impact Factors The strengths of the Impact Factor lies first of all in the comprehensibility, stability and seeming reproducibility, but some flaws have provoked critical and controversial discussions about its correctness and use. The above-mentioned popularity involves also dangers. The use of impact factors ranges from well-documented and methodically sound applications to rather ‘grey’ applications as background information for scientific journalism or in the context of refereeing procedures. Impact factors are sometimes used even as substitutes for missing citation data. Although it is difficult to theoretically define the concept of (journal) impact, there is a wide spread belief that the ISI Impact Factor is affected or ‘disturbed’ by factors that have nothing to do with (journal) impact.

35. W. Glänzel and K. DebackereMeasuring communication in science35 Journal Impact Factor 6. Journal Impact Factor Being a statistical mean, the IF should be size-independent. Large journals might, however, often have a higher visibility. The robustness, comprehensibility and methodological reproducibility of the ISI journal Impact Factor is contrasted by methodological shortcomings and its technical irreproducibility. It became quite tempting to apply the impact factor as a universal bibliometric measure. This is certainly one source of possible uninformed use. Methodological improvements in combination with complementary measures and an appropriate documentation may help to overcome limitations described above. The question of reproducibility can thus at least partially be solved for those who have access to the bibliographic databases and the technology to produce journal indicators.

36. W. Glänzel and K. DebackereMeasuring communication in science36 Journal Impact Factor 6. Journal Impact Factor Visibility vs. publication targeting vs. citation impact Publication in a high-IF journal might guarantee excellent visibility, but not automatically imply high citation rates, too. In several fields, targeting, i.e., reaching the desired audience is more important than publishing in high-impact journals (e.g., in clinical medicine, mathematics). The latter observation substantiates that research may have other impact than citations. In order to gain new insight in the utility of biomedical research Grant Lewison studied citations from clinical guidelines, textbooks, government policy documents, international or national regulations and newspaper articles. Moreover, publications in technical sciences and clinical medicine might find practical application that cannot be measured through citations.  Citations and, above all, the IF do not measure all aspects of impact published research results might have.

37. W. Glänzel and K. DebackereMeasuring communication in science37 Journal Impact Factor 6. Journal Impact Factor The myth of delayed recognition An often-heard argument on limitations of citation-based indicators is that important publications are often not cited in the beginning, and only become recognised in a time that is beyond the standard citation windows used in most bibliometric studies. Studies by Glänzel at al. and Glänzel & Garfield in 2004 have shown that the chance that a paper, uncited for three to five years after publication, will ever be cited is quite low, even in slowly aging fields such as mathematics. The citation impact of papers not cited initially usually remains low even 15 to 20 years later. The potential number of delayed recognition papers is extremely small. A statistically marginal share of 1.3 per 10,000 papers published in 1980 were "neglected" at first, and then, belatedly, received relatively high citational recognition.

38. W. Glänzel and K. DebackereMeasuring communication in science38 7.DISTORTED BEHAVIOUR BASED ON POLICY USE AND MISUSE OF BIBLIOMETRIC DATA

39. W. Glänzel and K. DebackereMeasuring communication in science39 7. Distorted behaviour Distorted behaviour based on policy use and misuse of bibliometric data An additional issue concerns the changes in the publication, citation and collaboration behaviour of scientists (both positive and negative) that the consistent policy use of bibliometric indicators might potentially induce. Studies on the problem choice behaviour of academic scientists have revealed that both cognitive and social influences determine the manner in which scientists go about choosing the problems they work on (Debackere and Rappa 1994). Hence the issue should be raised to what extent the policy use of bibliometrics might or could affect this behaviour.

40. W. Glänzel and K. DebackereMeasuring communication in science40 7. Distorted behaviour

41. W. Glänzel and K. DebackereMeasuring communication in science41 7. Distorted behaviour

42. W. Glänzel and K. DebackereMeasuring communication in science42 7. Distorted behaviour

43. W. Glänzel and K. DebackereMeasuring communication in science43 Schematic visualisation of the feedback of policy use of bibliometrics on the scientific community 7. Distorted behaviour

44. W. Glänzel and K. DebackereMeasuring communication in science44 Possible positive effects Scientists might recognise that scientific collaboration and publishing in high-impact or even top journals pays. Also their publication activity might be stimulated. Possible negative effects Exaggerated collaboration, even trends towards hyper- authorship, inflating publication output by splitting up publications to sequences, inflating citation impact by self- citations and forming citation cliques, etc. Trend towards replacing quality and recognition by visibility at any price or towards preferring journals as publication channels in social sciences and humanities might be among these effects. 7. Distorted behaviour

45. W. Glänzel and K. DebackereMeasuring communication in science45 CONCLUSIONS

46. W. Glänzel and K. DebackereMeasuring communication in science46 Conclusions The future will show in how far these negative effects will become reality. Empirical monitoring and examination of hypothetical biases will be worthwhile. Similar trends could already be observed far before the time of bibliometrics: Striving after visibility and reputation is part of human nature. Most negative effects will probably be hindered or prevented through the natural competition and peer review among researchers. The only negative feedback from policy use and misuse of bibliometric data might on the long run results in general ‘inflationary values’ described, e.g., by Cronin (2001) and Persson et al. (2003). Bibliometricians have the tools to normalise and standardise indicators under such conditions, and are thus able to cope with this problem, too.