Citation analysis in research evaluation Wouter Gerritsma, Marianne Renkema Information Specialists, Wageningen UR Library

Introduction Citation Indexes Citation analysis of scientific journals Excercises Citation analysis of researchers and groups of researchers

When research evaluation/citation analysis? Research assement of graduate schools (5 year cycle, stipulated by the VSNU) Other research groups are reviewed in a similar way Benchmarking becomes more important Grant applications Job positions Sooner of later, you will be subjected to such an analysis

Objective How are citation analyses performed, and what are their shortcomings or weaknesses? What can you do to enhance your results

References and citations what is the difference? If paper R contains a bibliographic footnote using and describing Paper C, then R contains a reference to C, or R cites C, and C has a citation from R or is cited by R. C is a (cited) reference contained in R. R is the source of the reference. Source: Moed (2005) based on Price (1970)

Citation Indexes Institute for Scientific Information Founder Eugene Garfield Part of the Thomson group (commercial publisher) Index articles of about 7800 journals, standard bibliographic information + reference lists Sell the same data in various databases to libraries and others (CC, WoS, JCR, ESI) They have monopoly position for citation data; they are the Standard

CC, WoS, JCR, ESI CC: Current Contents, weekly updated bibliographic database covering the top of Sciences (no citation data) WoS: Web of Science, weekly updated database, that includes references and citation data as well (slower than CC) JCR: Journal Citation Report, yearly analysis of WoS data, presenting perfomance measures for journals. ESI, Essential Science Indicators, bimonthly updated analytical database giving performance measures for Countries, Institutes, Scientists and Journals next to research fronts, hot papers and baselines.

Database Coverage WoS/JCR/CC ExcellentGoodModerate Molecular biology & Biochemistry Applied physics & ChemistryOther Social Sciences Biological Sciences related to humans Biological Sciences related to plants & animals Humanities Clinical medicinePsychology & psychiatry Physics & AstronomyOther social sciences related to medicine & health Mathematics / Engineering / Economics Source: Moed (2005)

Are there competitors ? Scopus (citation tools are being tested) Google Scholar ( PsychInfo/SciFinder (article indexes in Desktop library) ArXiv (Physics) Spires (high energy physics) Citeseer (ICT) OA Initiatives

Citation analysis of journals

Warning ! Often it has been attempted to use Journal Impact factors in some way to judge the performance of a group or individual scientist. This is proven very bad practice!

Journal citation analysis What are impact factors Are there other journal performance measures? What are the pitfalls of the IF How can I make use of this knowledge?

Journal Citation Reports Report 3 measures Impact factor Immediacy Index Cited half life Cited half-life 50% citations 3 1 Immediacy index Window Impact Factor Window

Definition of Impact factor The impact factor of a journal J in year T is defined as follows: “The number of citations received in year T by all documents published in J in the years T-1 and T-2” ÷ “The number of citable documents published in J in the years T-1 and T-2”

Citable items Cited items (nominator) include: Research articles, review arrrticles, notes, letters, editorials, news items, corrections and meeting abstracts Citable items (by definition) Articles, notes and reviews

Lancet (2002) Type of document No. DocsCitesCites/Doc Articles/reviews (citable items) 1,544 (a)13, Other types4,899 2, Total6,44315,670(b)2.4 JCR Like IF = (b)/(a) = 10.2

Impact factor Performance measure for journals “… it is also used for assessment of the quality of individual papers, scientists and departments. For the latter a scientific basis is lacking, as we will demonstrate in this contribution” (Opthof, 1997) Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular Research 33(1):

50 % of articles generate 90% of all cites Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ 314(7079):

Some other points IF measures citation impact of articles in 2 nd & 3 rd year after publication. Therefore biased towards journals revealing rapid maturing. Reference practices, particularly the number of references per article and their age distribution, vary considerably among subfields. IF between subfields can therefore not be compared.

Life: JCR & ESI

When do you make use of IF Answering the questions where do you want to publish your manuscript. Other questions, to be asked Reach colleagues ? In which article indexes included? Open access?

Citation analysis of paper, scientist or group

Science Citation Indexes Web of Science is comprised of three Citation Indexes Science Citation Index Exanded Social Science Citation Index (SSCI) Arts & Humanities Citation Index (AHCI)

Searching WoS General search: Searches bibliographic data: Title, Abstract, keywords, Authors, Addresses Not all citations are found due to errors in citing, indexing etc. Cited reference search: searches in the references of source data Not all articles are found. Non cited articles are missing

Author searches Use the index !

But what does it mean? Publication A (2003), 25 citations (Mol. Bio.) Publication B (1996), 17 citations (Math)

Essential Science Indicators (ESI) Analytical database of SCI, covering 10 years + current year building Comparison between Countries, Institutes, Scientists and Journals Hot papers / Highly cited papers Research fronts Baselines

Publication age : world averages citations (1)

Publication age : world averages citations (2)

In practice

In practice (WIAS, plant & animal sciences)

Relative impact of a research institute SubfieldAll groupsGroup 1Group 2Group 3 Agricultural Sciences 3,823,863,873,60 Biology & biochemistry 0,911,550,441,09 Chemistry 1,76 Clinical medicine 1,731,811,11 Microbiology 1, ,73 All subfields 2,062,082,261,84

Wageningen in Dutch perspective (2004 data) Institute No. highly cited papersPapersCitations Cites/ paper World Rank Univ Utrecht ,9668 Univ Amsterdam ,6575 Leiden Univ ,1177 Vrije Univ Amsterdam ,05111 Wageningen UR ,28171 Erasmus Univ Rotterdam ,49103 Univ Groningen ,76128 Univ Nijmegen ,74140 Delft Univ Technol ,95362 Univ Maastricht ,09276 Eindhoven Univ Technol ,03522 Univ Twente ,89562 Tilburg Univ ,951714

Strength & Weaknesses of Wageningen UR Subfield Highly cited papers PapersCites Cites/ Paper World rank (citations) ENVIRONMENT/ECOLOGY ,753 AGRICULTURAL SCIENCES ,563 PLANT & ANIMAL SCIENCE ,245 MICROBIOLOGY ,0224 PHARMACOLOGY & TOXICOLOGY ,59170 BIOLOGY & BIOCHEMISTRY ,84197 GEOSCIENCES ,20222 MOLECULAR BIOLOGY & GENETICS ,64263 CHEMISTRY ,66284 SOCIAL SCIENCES, GENERAL ,41352 CLINICAL MEDICINE ,68404 ENGINEERING ,51601

Example of selection of candidates for a professor in 2004 Candidate # Papers #CitatiesRelatie ve Impact RI RI #paper s bij top 10% #paper s bij top 1% a ,641,761,5242 b654981,931,841,95171 c939721,151,390,980 d ,861,691,94163 e573460,750,580,8330

Life WoS & ESI Exercises WOS & ESI

What questions should you ask Database coverage Accuracy of the counts Time period of the exercise Self citations Normalization? Verify the collected data

Winding down What is in a name? Persons (female PhD students!, initials!) Groups (Chair groups, Business units) Wageningen UR !