1. TIPS WHEN USING BIBLIOMETRICS UNITED KINGDOM OCTOBER 2010

2. USING BIBLIOMETRICS IN CITATION ANALYSIS The Development of Publication and Citation Analysis Bibliometrics and Peer Judgment: A Two-Pronged Approach

3. ONE: CONSIDER WHETHER AVAILABLE DATA CAN ADDRESS THE QUESTION Large data sets (nations, regions, large institutions) Engineering & Applied Sciences Fundamental & Basic Sciences Small data sets (small departments, individual researchers) Low Confidence Moderate Confidence High Confidence Social Sciences Individual Medical Country

4. TWO: CHOOSE PUBLICATION TYPES, FIELD DEFINITIONS AND YEARS OF DATA Publication types –Research Articles –Notes –Review articles Field definitions –Difficulties with characterization Time frame –Citation lag

5. THREE: DECIDE ON WHOLE OR FRACTIONAL COUNTING 68 authors 17 institutions 18 citations

6. FOUR: JUDGE WHETHER DATA REQUIRE EDITING TO REMOVE ARTIFACTS Artifacts of variation –Institution names Heidelberg Univ, Heidelberg Univ Clin, Heidelberg Univ Educ, Heidelberg Univ Hosp, Heidelberg Univ I, Heidelberg Univ II, Univ Heidelberg, Univ Heidelberg BZH, Univ Heidelberg Cardiol, Univ Heidelberg Clin, Univ Heidelberg Hosp, Univ Heidelberg Hosp & Clins, Univ Heidelberg J5, Univ Heidelberg Klinikujm, Univ Heidelberg Klinikum, Univ Heidelberg Mannheim, Univ Heidelberg Miriam Roehrig, Univ Heidelberg Neuebheimer Feld, Univ Heidelberg ZMBH, Univ Heidelburg, Univ Heilderberg, Univ Hosp Heidelberg, Univ Hosp Heidelberg Mannheim, Univ Klin Heidelberg, Univ Klin Heidelburg, Univ Klinikum Heidelberg, Univ Klinikum Heidelbert, Univ Klinikum Heidelburg, Univ Klinikum Heiderlberg –Author names Is ROTHWELL N different from ROTHWELL NJ? LEE S returns 14,022 publications; SUZUKI T (21,462)

7. …ARTIFACTS contd. COMMON OBJECTIONS TO CITATION COUNTS Negative citations Over citations –Reviews –Methods Self-citation and citation circles

8. FIVE: COMPARE LIKE WITH LIKE Impact Factor

9. FIVE: COMPARE LIKE WITH LIKE Different citation behaviour between categories Even within the same category: –Age of paper –Age and stage of author’s career

10. SIX: USE RELATIVE MEASURES – NOT JUST ABSOLUTE COUNTS Absolute counts –Total publications –Papers published per year on average –Papers published in top journals (various definitions) –Total citations received Relative counts –Citations per paper compared with field average –Citations versus expected citations –Cited vs un-cited papers compared with average –Rank within field or peer group

11. EIGHT: RECOGNIZE THE SKEWED NATURE OF CITATION DATA Citation distribution is always skewed –Few highly cited papers –Majority cited little or not at all Distribution type –Always distorted –Human decision E.g. Criticality

12. TREAT CITATION ANALYSIS AS A SCIENTIFIC STUDY Are the data collected relevant to the question? Are the results reasonable? Follow scientific process for evaluating data Apply scientific skepticism