1. Bibliometrics: the black art of citation rankings
Angela Carritt
Juliet Ralph
October 2010
These slides are available on

2. Overview of Session What are bibliometrics? Why bother? Problems
Calculating impact factors using Web of Science
…for an article
…for a researcher
…for a research group / department / institution
Calculating impact factors using Scopus
Calculating journal impact factors using the Journal Citation Reports 2

3. What are bibliometrics?
…the statistical analysis of books, articles, or other publications. Oxford English Dictionary
…”ways of measuring patterns of authorship, publication and the use of literature” HEFCE, Bibliometrics and the Research Excellence Framework (REF)
Use of citation analysis to measure the impact of research
OED tells us that bibliometrics is the branch of library science concerned with the application of mathematical and statistical analysis to bibliography; the statistical analysis of books, articles, or other publications.
For our purposes what we are really talking about is the use of citation analysis and particularly the measure of times cited as a way of assessing research impact.
So for example we might be looking at an individual paper as we did in the “whos citing me” and looking at how many later articles have cited that paper. The more citations the greater the search impact in the crudest terms.
We are perhaps more likely to be looking at all the papers of an individual researcher across their career including papers that have been highly cited and those that have not been, and using these collectively to assess the research impact of that individual researcher.
Or we might be might be looking at the papers written by a research group or an academic department and looking at how often these have been cited collectively.
A number of complex formula have been developed which take the crude “times cited” figure and other elements from the citation analysis in order to produce as accurate or as fair a measure of research impact as possible. We will look at some of these and particularly the h-index later but they are all based on “times cited”. 3

4. Citations to an individual paper
2010
2010
2010
2008
2009
What is citation indexing?
Citation indexing looks at the published literature and allows you to trace the progress of research backwards, forwards and sideways.
Start with a paper – for the purposes of this presentation we are assuming that this is a paper that you wrote, but it could be something that you have read which is important to your research.
Moving backwards in time you can see which papers were cited in “your paper”. So this is in effect looking at the bibliography of the paper. Obviously if you wrote the paper you will probably know what you referred to. However, this is very useful for those reading your paper – allows them to look at how you used earlier ideas.
Moving forwards, you can see which later papers cited “your paper”. This is important in allowing you to see how research in your area has developed. If you are the author – or someone associated with the author (e.g. someone working with them) then this can help you to measure the “impact” of the research. This can be crucial in getting grants or jobs.
Finally, you can look sideways to find related papers. These are papers which are not directly related to your paper – they are neither cited by your paper and nor do they cite you paper. However, they share references in common. So for example, they may refer to papers that you also cited, or they may be cited by papers which also cited your paper.
2008 4

5. Citations to a researcher5

6. Citations to a research group/department

7. Citations to a journal

8. Continued….
Wide range of formulae developed which aim to use bibliometrics to assess research impact accurately
Some of the things that are sometimes considered:
Times cited
Number of articles published
Number of self citations
Number of authors (of cited work)
Type of article
Period over which articles were published
Prestige of citing journal
Subject/discipline… 8

9. Principles of citation analysis are not just about books and journals
Principles of citation analysis are not just about books and journals. As some of you know Google ranks search results by the number of links to each page. So a web page which is linked to from many other web pages comes up higher in your search results than one which no one has linked to.
The power behind 9

10. 10

11. Why bother? Research Excellence Framework
Benchmarking of departments and research groups
Grant applications
Recruitment of individuals
UK academic institutions became very interested in bibliometrics when the REF was proposed as a replacement to the RAE. It was proposed that the REF would use bibliometrics at least in part in some sicence subjects as an indicator of hte quality of research The REF has now been shelved and so this is of less immediate concern.
However, bibliometrics seem to be increasingly used in the benchmarking of research groups, academic department and research groups s and it is therefore important for Universities and individuals to understand how to calculate impact factors and how these may be being calculated by other external agencies who are assessing them. IN may be important for example to make sure that the papers of highly cited individuals count in your departments figures.
Scorinig highly can be important when making applications for research grants and for individual in recruitment and selection – a candidate with a high h-index may be more attractive than someone with a lower h-index.
HOwev 11

12. Lots of problems... Self-citations Negative citations
Insignificant citations
Multiple authors/research groups
Incomplete citation lists - does not include citations in books... or other publications not indexed by Web of Science/Scopus…poor coverage of conferences
Not comparable across disciplines – may disadvantage researchers in interdisciplinary fields
Review articles are more highly cited than original research
More...
Before we go on to look at Web of Science which is one of the tools that you can use to calculate impact factors, I wanted to highlight some of the problems with measuring research impact using bibliometrics.
Researcher may inflate their performance by citing themselves – although the h-index claims to compenstae for this, this is still a very real critcism of these indicators.
Not all citations may be positive Citations which disprove research count just as much as positive citations. The tools that calculate these measure simply count hte number of times cited.
The two tools which are used to calculate these impact factors do not look at all publications. Although they cover tens of thousands of journals, there are still many that are left out. Books in particular are omitted and so citations to research in books do not count Some journals are omitted – although Web of Science and Scopus would argue that they include all the most significant journals there have been complaints about journals being omitted which are important in particular disciplines. English language journals are far more likely to be included than non English language.
The impact factors which are calculated are not comparable between disciplines. This is partly because some disciplines cite more than others – so papers in some disciplines typically include more references than in others. This does not necessarily matter. However, some interdisciplinary subject may fall foul of this. 12

13. http://www. slideshare
This cartoon shows some of the problems with impact factors.
Impact factor is the number of times your work has been cited minus the number of citations that trash your research – the number of times you cited yourself – minus number of times you were cited just to pad ou the introduction, minus the number of citations by other authors who were pressurised by the author to include the citation to boost their journals inpact factor, divied by original articles you’ve written pluse articles you wree incldued in out of politics or pity plus not so origianl articles you’ve copied and pasted. 13

14. © http://ceochef.files.wordpress.com/2008/07/pinch-of-salt.jpg
...many would say that for these reasons impact factors shoudl be taken witn an enormous pinch of salt. ..but unforytunately we probably can’t do this.... © 14

15. Individual articles On Web of Science - use Cited Reference Search
Better than General Search at retrieving Variants (incorrect citations),
Example:
Effectiveness of PowerPoint presentations in lectures. Author(s): Bartsch RA, Cobern KM
Source: COMPUTERS & EDUCATION    
Volume: 41    Issue: 1    Pages: 77-86    
Published: AUG 2003.
Search example ? Could use
Effectiveness of PowerPoint presentations in lectures. Author(s): Bartsch RA, Cobern KM Source: COMPUTERS & EDUCATION    Volume: 41    Issue: 1    Pages: 77-86    Published: AUG 2003. Or
Title: Glycobiology: Toward understanding the function of sugars Author(s): Dwek, RA Source: CHEMICAL REVIEWS   Volume: 96   Issue: 2   Pages:   Published: MAR-APR 1996

17. Cited reference variant

18. Analyzing a highly cited article

19. Analyzing the Citing articles

20. …by Highest citing authors

21. …by Highest citing journals

22. Impact factors for a researcher

24. Citation report for an author

27. Citation map for an article

29. Citation map

30. Other services offering citation searching - SCOPUS
Sciences and Social Sciences
Results include journal articles and web pages
Each reference to a paper shows the number of times an article has been cited

31. Citation analysis in Scopus

34. Potentially lower h-index in scopus cos it only covers articles published from 1996 onwards. As Dwek published before then his h-index is higher in WoS.

35. …for a research group Not easy!
But can search by department or institution
Web of Science - Search the Address field
Strict rules regarding syntax
Search by postcode or name of department/college, e.g. OX1 2JD
univ oxford same pathol
oxford same trinity
oxford same Ludwig Inst Canc Res
Scopus
Affiliation Search
Much less strict!
By name not postcode

36. Searching by Address

37. Check the abbreviations lists

38. Beware of other depts with similar name eg clinical pathology
Beware of other depts with similar name eg clinical pathology. To weed them out could try univ oxford same dunn

39. Trial and error

40. Scopus Affiliation Search

41. Syntax less strict than WoS: University of oxford – or oxford university – retrieves same results. Also oxford univ* or oxford univ*

42. Scopus Affiliation ID

44. Compare with strict instructions at Imperial Coll where ranking was maximised.
Sample Addresses from WoS : Univ London Imperial Coll Sci Technol & Med, Blackett Lab, London SW7 2BZ, England Univ Oxford, John Radcliffe Hosp, Weatherall Inst Mol Med, Human Immunol Unit, Oxford OX3 9DU, England
Univ Oxford, Nuffield Dept Clin Med, Ludwig Inst Canc Res, Oxford OX1 2JD, England
Univ Oxford, Said Business Sch, Oxford OX1 2JD, England
Trinity Coll, Inst Sci & Soc, Oxford OX1 3BH, England
Trinity Coll, Oxford OX1 3BH, England
Trinity Coll, Oxford, England
Univ Oxford Univ Coll, Oxford OX1 4BH, England

45. Impact Factors - Journal Citation Reports (JCR)
Compare relative importance of journals by measuring their impact
Uses
Help you to decide where to publish
Help librarians to make decisions about journal purchase, retention etc
Abuses
Have been used to measure research impact of individual and research groups
Based on citation data from Web of Science
Covers
> 5,900 journals in science and technology
> 1,700 journals in the social sciences 45 45

46. 46

49. Immediacy Index Measures how quickly articles are cited. Calculated: no. of citations to articles published this year ÷ no. of articles published this year.
Impact Factor - Number of times the “average” article published in the previous 2 or 5 years was cited this year.
Calculated:
no. of citations to articles published in the last 2 (or 5) years ÷ no. of articles published in same period.
Cited Half-Life - How many years you have to go back to account for 50% of citations to the journal. e.g. 50% of citations were to articles published in the last 3.5 years. The rest cited earlier articles.

50. Detailed view

51. Detailed view continued

52. Detailed view continued
Citations TO the journal by year of cited article (e.g. 333 of this year’s citations to Biological Review were to articles published in in 2005 )

53. Detailed view continued
Citations from Biological Review (to other journals and self cites) by year of cited article E.g. 334 citations from Biological Reviews journal cited articles published in 2007

54. Type of articles included

55. Google’s PageRank from
Eigenfactor Metrics
Use “Google style” algorithms
Take into account prestige of citing sources
Attempts to measure how often the average researcher would encounter the journal
Google’s PageRank from

56. Eigenfactor: Score & Article Influence
Eigenfactor – increases with the size of the journal
Article Influence – Takes into account number of articles published. More comparable to the JCR impact factor

57. 57 57

58. Journal Impact Factors: Problems
Use with caution…Results are skewed by many factors…
Size
Frequency / time of publication
Type of content - review articles are more heavily cited than original research…
Journals that are not indexed by WOS are disadvantaged
Non English Language journals disadvantaged
“Cited” only journals
Problems when journals change names
Results are not comparable across discipline (some journals in the wrong discipline)
Journal impact factors should NEVER be used to assess impact of researchers / groups etc

59. Where will it all end… Watch this space!
And how much will it all cost…
Watch this space! 59

60. Brief bibliography General
Broadus, R. N., “Towards a definition of Bibliometrics” Scientometrics, vol. 12, nos 5-6, (1987)
HEFCE papers, reports, papers and pilots on the use of bibliometrics in the
H-Index
Hirsch, J. E. (15 November 2005). "An index to quantify an individual's scientific research output". PNAS 102 (46):
Ball, P. “Index aims for fair ranking of scientists”, Nature 2005 Aug : 900

61. Brief bibliography Journal Impact Factors and the JCR Want more…
The Thomson Reuters Impact Factor (originally published in the Current Contents print editions June 20,
Garfield, E. "The agony and the ecstasy: the history and meaning of the Journal Impact Factor“ Paper at the International Congress on Peer Review And Biomedical Publication, Chicago, September 16,
Want more…
Wikipedia entries on the following topics include useful bibliographies: the h-index, journal impact factors and the Eigenfactor
A Google Scholar ( search will return many useful articles including subject studies on the use of bibliometrics