Evaluation of journals based on bibliometric measures Stefan Carlstein January 14, 2010

ISI Web of Science (WoS) ISI WoS is provided by Thomson Reuters The three main databases of WoS: Science Citation Index (coverage 1945-) Social Science Citation Index (coverage 1956-) Arts & Humanities Citation Index (coverage 1975-) Approximately 9,300 indexed high impact journals (including Open Access journals) Used by The Swedish Research Council (Vetenskapsrådet) for the allocation of grants to Swedish universities and university colleges

Subject coverage in ISI Web of Science (%) Source: Vetenskapsrådet (2009). Biblimetrisk indikator som underlag för medelsfördelningVetenskapsrådet (2009). Biblimetrisk indikator som underlag för medelsfördelning

Journal Citation Reports (JCR) Covers citation data from more than 8,000 scholarly and technical journals. Covers approximately 227 disciplines. Two editions: The science edition covers over 6,500 journals The social sciences edition covers over 1,900 journals JCR allows you to see: The most frequently cited journals in a field The highest impact journals in a field Largest journals in a field Cited and citing journal statistics from 1997 forward Includes measures like: the Journal Impact Factor (JIF), Immediacy Index, Cited Half-Life, and also Eigenfactor and Article Influence (2007 and later)

The impact factor (IF) The impact factor (IF) is used as a measure of the relative importance of a journal within its field – also called Journal Impact Factor (JIF). Journals with higher impact factors are assessed as more important than those who have received a lower impact. The calculation of the impact factor is based on citations. A citation indicates that an article has influenced an author, and if an article is cited often it would indicate that it has had a strong influence on the field and the research community. Created and originally used by Eugene Garfield to help select journals that were to be covered and indexed in Science Citation Index (the Web of Science database).

Why and when do people cite? 1.Paying homage to pioneers 2.Giving credit for related work (homage to peers) 3.Identifying methodology, eqipment, etc. 4.Providing background reading 5.Correcting ones own work 6.Correcting the work of others 7.Criticizing previous work 8.Substantiating claims 9.Alerting researchers to forthcoming work 10.Providing leads to poorly disseminated, poorly indexed, or uncited work 11.Authenticating data and classes of fact - physical constants, etc. 12.Identifying original publications in which an idea or concept was discussed 13.Identifying the original publication describing an eponymic concept or term 14.Disclaiming work or ideas of others (negative claims) 15.Disputing priority claims of others (negative homage) Source: Garfield, E. (1996). When to cite. Library Quarterly, 66(4), Garfield, E. (1996). When to cite. Library Quarterly, 66(4),

Measures of the scientific quality of journals (1) Journal Impact factor (JIF) The average number of times articles from the journal published in the past two years have been cited in the JCR year (the edition of JCR = the year of measuring). Calculated by dividing the number of citations in the JCR year by the total number of articles published in the two previous years. Also: 5-Year Journal Impact Factor, calculated the same way as above but using articles published in the past five years instead of two. Example: A = the number of times articles published in 2006 and 2007 were cited by indexed journals during 2008 (=JCR year) B = the total number of citable items" published in 2006 and impact factor = A / B

Example: 2008 impact factor for JPIM JIF = A/B (citations/number of articles)

An Impact Factor of 1.0 means that, on average, the articles published one or two years ago have been cited ONE time. An Impact Factor of 2.5 means that, on average, the articles published one or two years ago have been cited TWO AND A HALF times. Citing articles may be from the same journal - most citing articles are from different journals (JCR year)

Criticisms of the Impact Factor Validity Discipline-dependent Not a normal distribution Author self-citations (although not necessarily a sign of low quality) Manipulation Journals can adopt editorial policies that increase the impact factor Misuse Misused to evaluate individual articles or researchers

The skewed distribution of citations – example journal Source: Cross (2009). Impact factors – the basics. In The e-resources management handbook.Cross (2009). Impact factors – the basics. In The e-resources management handbook.

The skewed distribution of citations - all articles Graph based on Karolinska Institutet world data acquired from Thomson Reuters, Inc.

Measures of the scientific quality of journals (2) The journal Immediacy Index Indicates how quickly articles in a journal are cited. The immediacy index is calculated by dividing the number of citations to articles published in a given year by the number of articles published in that year. The higher the index the quicker the articles are cited. The Immediacy Index tends to discount the advantage of large journals over small ones. However, frequently issued journals may have an advantage because an article published early in the year has a better chance of being cited than one published later in the year. Many publications that publish infrequently or late in the year have low Immediacy Indexes.

Measures of the scientific quality of journals (3) Journal Cited Half-Life The median age of the articles that were cited in Journal Citation Reports each year. For example, if a journal's half-life in 2005 is 5, that means the citations from are half of all the citations from that journal in 2005, and the other half of the citations precede Aggregate measures are calculated in the same way as their journal counterparts although aim at whole categories instead of specific journals: Aggregate Impact Factor Aggregate Immediacy Index Aggregate Cited Half-Life And also: Median Impact Factor (the median value of all Journal Impact Factors in the subject category). In JCR you choose View Category Data instead of View Journal Data.

Measures of the scientific quality of journals (4) The H index A scientist has index h if h of [his/her] N p papers have at least h citations each, and the other (N p h) papers have at most h citations each. Source: Hirsch (2005). An index to quantify an individual scientific research output. PNAS, 102(46), 16569–16572.Hirsch (2005). An index to quantify an individual scientific research output. PNAS, 102(46), 16569– Ranking order - articlesNumber of citations publications that have at least 6 citations each The rest of the publications have 6 citations each at the most

Measures of the scientific quality of journals (5) Eigenfactor and Article Influence Use citation data to assess and track the influence of a journal in relation to other journals. Metrics in JCR only available for JCR years 2007 and later. Aim to identify the most influential journals – where a journal is considered to be influential if it is cited more often by other influential journals. Larger journals will have have more citations and therefore larger Eigenfactor Scores. Measure the importance of a citation by the influence of the citing journal divided by the total number of citations appearing in that journal Similar to Googles PageRank. A network structure of citations.

Measures of the scientific quality of journals (5) Eigenfactor Score Measures the number of times articles from the journal published in the past five years have been cited in the JCR year. Similar to the Impact Factor - essentially a ratio of number of citations to total number of articles. But, unlike the Impact Factor, the Eigenfactor Score: Counts citations to journals in both the sciences and social sciences. Eliminates self-citations. Every reference from one article in a journal to another article from the same journal is discounted. Weights each reference according to a stochastic measure of the amount of time researchers spend reading the journal. Rates journals according to the number of incoming citations, with citations from highly-ranked journals weighted to make a larger contribution to the Eigenfactor than those from poorly-ranked journals.

Measures of the scientific quality of journals (5) Eigenfactor Score continued The algorithm effectively calculates the trajectory of a hypothetical random researcher who behaves as follows. Our random researcher begins by going to the library and selecting a journal article at random. After reading the article, she selects at random one of the citations from the article. She then proceeds to the cited work and reads a random article there. She selects a new citation from this article, and follows that citation to her next journal volume. The researcher does this ad infinitum. Since we lack the time to carry out this experiment in practice, Eigenfactor uses mathematics to stimulate this process. Source: Bergstrom (2007). Eigenfactor: measuring the value and prestige of scholarly journals. C&RL News, 68(5).Bergstrom (2007). Eigenfactor: measuring the value and prestige of scholarly journals. C&RL News, 68(5).

Measures of the scientific quality of journals (6) Article Influence Measures the relative importance of the journal on a per-article basis. It is the journal's Eigenfactor Score divided by the fraction of articles published by the journal. That fraction is normalized so that the sum total of articles from all journals is 1. The mean Article Influence Score is A score greater than 1.00 indicates that each article in the journal has above- average influence. A score less than 1.00 indicates that each article in the journal has below-average influence. More directly comparable to the Impact Factor.

SCImago Journal & Country Rank Metrics based on data from the database Scopus Scopus: approximately indexes 15,000 journals (compared to Web of Science: approx. 9,300 journals) SCImago Journal Rank Indicator is similar to Eigenfactor 27 major thematic categories (Subject Area) 313 specific subject categories (Subject Category) Comparison of up to four journals at the time Map generator for visualization of fields

Source: Stock (2009). The inflation of impact factors of journals. ChemPhysChem, 10, p. 2193Stock (2009). The inflation of impact factors of journals. ChemPhysChem, 10, p. 2193

Sources to look for journal impact indicators Journal Citation ReportsJournal Citation Reports (data from ISI Web of Science) Network login is necessary if accessing it from outside of campus! Journal EigenfactorJournal Eigenfactor (data from ISI Web of Science) Scopus Network login is necessary if accessing it from outside of campus! SCImago Journal & Country Rank SCImago Journal & Country Rank (data from Scopus) Publish or PerishPublish or Perish (free software) Uses Google Scholar to obtain raw citations and analyses the data.

Some aspects that influence journal evaluation

The impact factor – what are citations all about? From: