Does It Matter Where We Publish? Adam Eyre-Walker University of Sussex

Reasons Prestige peers employers Impact scientific community society

Measuring impact General difficult to measure Citations Easy to measure Velvet Underground effect “only sold 10,000 copies, but everyone who bought it formed a band” (Brian Eno)

Question Do papers of similar quality garner more citations in better quality journals?

Ideal experiment Papers made anonymous authors journal structure Assessed for quality by experts Assessments confidential Citations tracked

Research Assessment Exercise (RAE) Each department submits 4 publications per academic (75%) 4 indicators of Esteem (5%) Environment PhD student numbers Grant income Research structure Facilities Superceded by REF

RAE dataset 1170 papers published Scored by AEW Scored 1 to 4 (or unclassified) Subjects : evolutionary biology, genomics, bioinformatics, ecology, animal behaviour and organismal biology Ranked without reference to journal citations

RAE scores RAE scoreNo. of papers

Journal quality Impact factor Number times papers published in the two previous years are cited in target year Two components Quality of papers Effect of the journal on citations

IF categories IF categoryIFNo of journals No. of papers 1IF < < IF < < IF < < IF < < IF < < IF < < IF < IF >

Citations No. of citations from ISI Web of Science October 2008

Data For each paper Year of publication RAE score (RS) No. of citations Impact factor of journal (IF category) Otherwise anonymous

How good is the assessor? Correlation between AEW and co- scorers > 0.7 Correlation between RS and no. of citations within each IF category and within each year

How good is the assessor? Correlation between AEW and co- scorers > 0.7 Correlation between RS and no. of citations within each IF category and within each year Average r = 0.27 (p<0.001) Average r max = 0.73

RS v citation correlation r = p = 0.005

Central question Do papers of similar quality accumulate more citations in some journals than others?

RAE score 2 in 2003

RAE score 2 p < 0.001

RAE score 3 p < 0.001

ANCOVA RS gradep-value30 v (1.7) (0.3) (0.4)

ANCOVA RS grade p-valueAll 30 v 5 IF<10 30 v (1.7)1.2 (0.4) (0.3)3.0 (0.3) (0.4)5.4 (1.4)

Alternative explanation Systematically underestimate quality of papers in high ranking journals Test Assume no effect of journal Consider correlation between RS and journal (IF) controlling for citations

Partial correlations YearCorrelation *** *** *** *** *** *** ***

ANCOVA RS gradep-value30 v (1.7) (0.3) (0.4)

Question Impact factors – relative contribution of Paper quality Journal effect

Summary Assessor scores correlated to citations strength correlated to age Assessor scores influenced by journal Papers published in higher quality journals accumulate more citations independent of quality F1000 data

Question Where will we submit?

Thanks Nina Stoletzki

Assessor and journal Assume citations measure quality no journal effect