1. Does It Matter Where We Publish? Adam Eyre-Walker University of Sussex

2. Reasons Prestige peers employers Impact scientific community society

3. Measuring impact General difficult to measure Citations Easy to measure Velvet Underground effect “only sold 10,000 copies, but everyone who bought it formed a band” (Brian Eno)

4. Question Do papers of similar quality garner more citations in better quality journals?

5. Ideal experiment Papers made anonymous authors journal structure Assessed for quality by experts Assessments confidential Citations tracked

6. Research Assessment Exercise (RAE) Each department submits 4 publications per academic (75%) 4 indicators of Esteem (5%) Environment PhD student numbers Grant income Research structure Facilities Superceded by REF

7. RAE dataset 1170 papers published 2001 - 2007 Scored by AEW Scored 1 to 4 (or unclassified) Subjects : evolutionary biology, genomics, bioinformatics, ecology, animal behaviour and organismal biology Ranked without reference to journal citations

8. RAE scores RAE scoreNo. of papers 174 2539 3401 4103

9. Journal quality Impact factor Number times papers published in the two previous years are cited in target year Two components Quality of papers Effect of the journal on citations

10. IF categories IF categoryIFNo of journals No. of papers 1IF < 24153 22 < IF < 339134 33 < IF < 433159 44 < IF < 521214 55 < IF < 616128 66 < IF <1028180 710 < IF < 2013110 8IF > 208181

11. Citations No. of citations from ISI Web of Science October 2008

12. Data For each paper Year of publication RAE score (RS) No. of citations Impact factor of journal (IF category) Otherwise anonymous

13. How good is the assessor? Correlation between AEW and co- scorers > 0.7 Correlation between RS and no. of citations within each IF category and within each year

14. How good is the assessor? Correlation between AEW and co- scorers > 0.7 Correlation between RS and no. of citations within each IF category and within each year Average r = 0.27 (p<0.001) Average r max = 0.73

15. RS v citation correlation r = -0.37 p = 0.005

16. Central question Do papers of similar quality accumulate more citations in some journals than others?

17. RAE score 2 in 2003

18. RAE score 2 p < 0.001

19. RAE score 3 p < 0.001

20. ANCOVA RS gradep-value30 v 5 10.0052.1 (1.7) 20.0002.5 (0.3) 30.0002.1 (0.4) 40.60-

21. ANCOVA RS grade p-valueAll 30 v 5 IF<10 30 v 5 10.0052.1 (1.7)1.2 (0.4) 20.0002.5 (0.3)3.0 (0.3) 30.0002.1 (0.4)5.4 (1.4) 40.60-

22. Alternative explanation Systematically underestimate quality of papers in high ranking journals Test Assume no effect of journal Consider correlation between RS and journal (IF) controlling for citations

23. Partial correlations YearCorrelation 20010.54*** 20020.49*** 20030.54*** 20040.41*** 20050.31*** 20060.37*** 20070.35***

24. ANCOVA RS gradep-value30 v 5 10.0052.1 (1.7) 20.0002.5 (0.3) 30.0002.1 (0.4) 40.60-

25. Question Impact factors – relative contribution of Paper quality Journal effect

26. Summary Assessor scores correlated to citations strength correlated to age Assessor scores influenced by journal Papers published in higher quality journals accumulate more citations independent of quality F1000 data

27. Question Where will we submit?

28. Thanks Nina Stoletzki

29. Assessor and journal Assume citations measure quality no journal effect