1. Citation analysis in research evaluation Wouter Gerritsma, Marianne Renkema Information Specialists, Wageningen UR Library

3. Introduction Citation Indexes Citation analysis of scientific journals Excercises Citation analysis of researchers and groups of researchers

4. When research evaluation/citation analysis? Research assement of graduate schools (5 year cycle, stipulated by the VSNU) Other research groups are reviewed in a similar way Benchmarking becomes more important Grant applications Job positions Sooner of later, you will be subjected to such an analysis

5. Objective How are citation analyses performed, and what are their shortcomings or weaknesses? What can you do to enhance your results

6. References and citations what is the difference? If paper R contains a bibliographic footnote using and describing Paper C, then R contains a reference to C, or R cites C, and C has a citation from R or is cited by R. C is a (cited) reference contained in R. R is the source of the reference. Source: Moed (2005) based on Price (1970)

7. Citation Indexes Institute for Scientific Information Founder Eugene Garfield Part of the Thomson group (commercial publisher) Index articles of about 7800 journals, standard bibliographic information + reference lists Sell the same data in various databases to libraries and others (CC, WoS, JCR, ESI) They have monopoly position for citation data; they are the Standard

8. CC, WoS, JCR, ESI CC: Current Contents, weekly updated bibliographic database covering the top of Sciences (no citation data) WoS: Web of Science, weekly updated database, that includes references and citation data as well (slower than CC) JCR: Journal Citation Report, yearly analysis of WoS data, presenting perfomance measures for journals. ESI, Essential Science Indicators, bimonthly updated analytical database giving performance measures for Countries, Institutes, Scientists and Journals next to research fronts, hot papers and baselines.

9. Database Coverage WoS/JCR/CC ExcellentGoodModerate Molecular biology & Biochemistry Applied physics & ChemistryOther Social Sciences Biological Sciences related to humans Biological Sciences related to plants & animals Humanities Clinical medicinePsychology & psychiatry Physics & AstronomyOther social sciences related to medicine & health Mathematics / Engineering / Economics Source: Moed (2005)

10. Are there competitors ? Scopus (citation tools are being tested) Google Scholar (http://scholar.google.com)http://scholar.google.com PsychInfo/SciFinder (article indexes in Desktop library) ArXiv (Physics) Spires (high energy physics) Citeseer (ICT) OA Initiatives

11. Citation analysis of journals

12. Warning ! Often it has been attempted to use Journal Impact factors in some way to judge the performance of a group or individual scientist. This is proven very bad practice!

13. Journal citation analysis What are impact factors Are there other journal performance measures? What are the pitfalls of the IF How can I make use of this knowledge?

14. Journal Citation Reports Report 3 measures Impact factor Immediacy Index Cited half life Cited half-life 50% citations 3 1 Immediacy index Window Impact Factor Window

15. Definition of Impact factor The impact factor of a journal J in year T is defined as follows: “The number of citations received in year T by all documents published in J in the years T-1 and T-2” ÷ “The number of citable documents published in J in the years T-1 and T-2”

16. Citable items Cited items (nominator) include: Research articles, review arrrticles, notes, letters, editorials, news items, corrections and meeting abstracts Citable items (by definition) Articles, notes and reviews

17. Lancet (2002) Type of document No. DocsCitesCites/Doc Articles/reviews (citable items) 1,544 (a)13,1068.5 Other types4,899 2,5640.5 Total6,44315,670(b)2.4 JCR Like IF = (b)/(a) = 10.2

18. Impact factor Performance measure for journals “… it is also used for assessment of the quality of individual papers, scientists and departments. For the latter a scientific basis is lacking, as we will demonstrate in this contribution” (Opthof, 1997) Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular Research 33(1): 1-7. http://dx.doi.org/10.1016/S0008-6363(96)00215-5 http://dx.doi.org/10.1016/S0008-6363(96)00215-5

19. 50 % of articles generate 90% of all cites Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ 314(7079): 497-502. http://bmj.bmjjournals.com/cgi/content/full/314/7079/497

20. Some other points IF measures citation impact of articles in 2 nd & 3 rd year after publication. Therefore biased towards journals revealing rapid maturing. Reference practices, particularly the number of references per article and their age distribution, vary considerably among subfields. IF between subfields can therefore not be compared.

21. Life: JCR & ESI

22. When do you make use of IF Answering the questions where do you want to publish your manuscript. Other questions, to be asked Reach colleagues ? In which article indexes included? Open access?

23. Citation analysis of paper, scientist or group

24. Science Citation Indexes Web of Science is comprised of three Citation Indexes Science Citation Index Exanded Social Science Citation Index (SSCI) Arts & Humanities Citation Index (AHCI)

25. Searching WoS General search: Searches bibliographic data: Title, Abstract, keywords, Authors, Addresses Not all citations are found due to errors in citing, indexing etc. Cited reference search: searches in the references of source data Not all articles are found. Non cited articles are missing

26. Author searches Use the index !

27. But what does it mean? Publication A (2003), 25 citations (Mol. Bio.) Publication B (1996), 17 citations (Math)

28. Essential Science Indicators (ESI) Analytical database of SCI, covering 10 years + current year building Comparison between Countries, Institutes, Scientists and Journals Hot papers / Highly cited papers Research fronts Baselines

29. Publication age : world averages citations (1)

30. Publication age : world averages citations (2)

31. In practice

32. In practice (WIAS, plant & animal sciences)

33. Relative impact of a research institute SubfieldAll groupsGroup 1Group 2Group 3 Agricultural Sciences 3,823,863,873,60 Biology & biochemistry 0,911,550,441,09 Chemistry 1,76 Clinical medicine 1,731,811,11 Microbiology 1,700.571,73 All subfields 2,062,082,261,84

34. Wageningen in Dutch perspective (2004 data) Institute No. highly cited papersPapersCitations Cites/ paper World Rank Univ Utrecht34323.938286.38511,9668 Univ Amsterdam33820.730262.18312,6575 Leiden Univ28018.565262.18314,1177 Vrije Univ Amsterdam24815.353200.38513,05111 Wageningen UR23014.707151.22710,28171 Erasmus Univ Rotterdam21713.450208.29015,49103 Univ Groningen21715.800158.83011,76128 Univ Nijmegen15214.700172.60411,74140 Delft Univ Technol14510.10570.2046,95362 Univ Maastricht1128.63695.77511,09276 Eindhoven Univ Technol926.19943.5677,03522 Univ Twente755.67939.1446,89562 Tilburg Univ111.8187.1753,951714

35. Strength & Weaknesses of Wageningen UR Subfield Highly cited papers PapersCites Cites/ Paper World rank (citations) ENVIRONMENT/ECOLOGY4018782018710,753 AGRICULTURAL SCIENCES392212145026,563 PLANT & ANIMAL SCIENCE994249392809,245 MICROBIOLOGY89511523916,0224 PHARMACOLOGY & TOXICOLOGY2189294615,59170 BIOLOGY & BIOCHEMISTRY214211540810,84197 GEOSCIENCES7299305110,20222 MOLECULAR BIOLOGY & GENETICS0467823717,64263 CHEMISTRY899596169,66284 SOCIAL SCIENCES, GENERAL32036923,41352 CLINICAL MEDICINE147911556319,68404 ENGINEERING32699433,51601

36. Example of selection of candidates for a professor in 2004 Candidate # Papers 1994- 2003 #CitatiesRelatie ve Impact RI 1994- 1998 RI 1999- 2003 #paper s bij top 10% #paper s bij top 1% a8015651,641,761,5242 b654981,931,841,95171 c939721,151,390,980 d8818861,861,691,94163 e573460,750,580,8330

37. Life WoS & ESI Exercises WOS & ESI

38. What questions should you ask Database coverage Accuracy of the counts Time period of the exercise Self citations Normalization? Verify the collected data

39. Winding down What is in a name? Persons (female PhD students!, initials!) Groups (Chair groups, Business units) Wageningen UR !