February 2016 Peer review: A global view A white paper from Taylor & Francis Research 2 Reader Conference Will Frass Senior Research Executive authorservices.taylorandfrancis.com/peer-review-in-2015

Assessment of the make-up of House of Lords: Peer Review

February 2016 Peer review: A global view A white paper from Taylor & Francis Research 2 Reader Conference authorservices.taylorandfrancis.com/peer-review-in-2015 Will Frass Senior Research Executive 1 Methodology 2 Ideal world & real world objectives 5 Different models of peer review 3 Ethical concerns 4 Timing discrepancies

43,000 Science Technology Medicine 43,000 Humanities & Social Sciences 1 Methodology 2 Ideal world & real world objectives Peer review in 2015 | A global view A white paper from Taylor & Francis 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

43,000 Science Technology Medicine 43,000 Humanities & Social Sciences 1 Methodology 2 Ideal world & real world objectives Peer review in 2015 | A global view A white paper from Taylor & Francis 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

2,398 STM responses 4,750 HSS responses 1 Methodology 2 Ideal world & real world objectives Peer review in 2015 | A global view A white paper from Taylor & Francis 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

Confidence Interval: Confidence Level: 95% Confidence Interval: Confidence Level: 95% 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review 2,398 STM responses 4,750 HSS responses Peer review in 2015 | A global view A white paper from Taylor & Francis

2% confidence interval at 95% confidence level for the population of all 2013 published authors 95% Result -2%+2% 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review Peer review in 2015 | A global view A white paper from Taylor & Francis

2,398 STM responses 4,750 HSS responses All survey respondents: STM Authors Reviewers Editors HSS Authors Reviewers Editors 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review Peer review in 2015 | A global view A white paper from Taylor & Francis

1 Methodology 2 Ideal world & real world objectives Qualitative research 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review Peer review in 2015 | A global view A white paper from Taylor & Francis Numbers:6 focus groups 46 participants Locations:United Kingdom South Africa China Participants:Editors, authors and reviewers Minimum of 2 articles peer reviewed (with any other publisher) Disciplines:Science, Technology & Medicine Social Sciences & Humanities

In an ideal world… …to what extent do you agree or disagree the following objectives should be the purpose of peer review? 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review Peer review in 2015 | A global view A white paper from Taylor & Francis

In an ideal world… …to what extent do you agree or disagree the following objectives should be the purpose of peer review? Ideal World – rating out of 10 HSS 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

In an ideal world… …to what extent do you agree or disagree the following objectives should be the purpose of peer review? Ideal World – rating out of 10 HSS 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

In the real world… …to what extent do you agree or disagree that peer review is currently achieving the following objectives effectively? Ideal World – rating out of Real World – rating out of 10 HSS 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

In the real world… …to what extent do you agree or disagree that peer review is currently achieving the following objectives effectively? Ideal World – rating out of Real World – rating out of 10 HSS 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

In the real world… …to what extent do you agree or disagree that peer review is currently achieving the following objectives effectively? Ideal World – rating out of Real World – rating out of 10 HSS Relevant to Aims & Scope 1 Methodology 2 Ideal world & real world objectives Ethical concerns 34 Timing discrepancies 5 Different models of peer review

HSS STM 2 Ideal world & real world objectives 3 Ethical concerns Timing discrepancies 45 Different models of peer review 1 Methodology

HSS STM Detect plagiarism Ideal world mean scoreReal world mean score 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review 1 Methodology

HSS STM Improve quality of published article Ideal world mean scoreReal world mean score 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review 1 Methodology

HSS STM Provide polite feedback Ideal world mean scoreReal world mean score 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review 1 Methodology

HSS STM Correct spelling, punctuation & grammar Ideal world mean scoreReal world mean score 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review 1 Methodology

HSS Most important objective… Expectation exceeds reality… Expectation matches reality… Reality exceeds expectation… PolitenessDetect Fraud Correcting spelling, punctuation & grammar Relevant to the Aims & Scope Improve quality of published paper STM 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

HSS Most important objective… Expectation exceeds reality… Expectation matches reality… Reality exceeds expectation… PolitenessDetect Fraud Correcting spelling, punctuation & grammar Relevant to the Aims & Scope Improve quality of published paper STM 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review “The worst reviews are short, snitty, patronising and not remotely useful. The best are critically engaged, add something and improve the quality.” Editor, Linguistics, UK “The worst reviews are short, snitty, patronising and not remotely useful. The best are critically engaged, add something and improve the quality.” Editor, Linguistics, UK “Editors should be more pre-emptive in detecting plagiarism & other types of fraud.” Researcher, Medical Research, UK “Editors should be more pre-emptive in detecting plagiarism & other types of fraud.” Researcher, Medical Research, UK

How common are the following situations in peer review? Regional bias Seniority bias Gender bias 1 Methodology 2 Ideal world & real world objectives Peer review in 2015 | A global view A white paper from Taylor & Francis 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

How common are the following situations in peer review? HSS STM Gender bias Regional bias Seniority bias 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

How common are the following situations in peer review? HSS STM Gender bias Regional bias Seniority bias 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

How common are the following situations in peer review? HSS STM Reviewers delay assessment Reviewers take ideas Reviewers use false identities 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies Different models of peer review 5

Higher frequency of occurrences reported 1 Methodology 2 Ideal world & real world objectives Peer review in 2015 | A global view A white paper from Taylor & Francis Lower frequency of occurrences reported 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

As an editor, what do you consider a realistic amount of time to expect reviewers to deliver their initial report? HSS 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

As an editor, what do you consider a realistic amount of time to expect reviewers to deliver their initial report? 6% HSS 54%96% 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

HSS As a reviewer, how long was the duration between your acceptance to review and the delivery of your initial report? 96% 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

HSS As a reviewer, how long was the duration between your acceptance to review and the delivery of your initial report? 96% 7%23%63%93% 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

HSS As an author, how long did you wait after submission before you received the peer reviewer’s initial comments? 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

HSS As an author, how long did you wait after submission before you received the peer reviewer’s initial comments? 96% 93% 1%5%14%44% 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

HSS 96% 93% Editors 2 months is realistic Reviewers 2 months to deliver report Authors 2 months to receive report 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

HSS 96% 93% 44% Editors 2 months is realistic Reviewers 2 months to deliver report Authors 2 months to receive report 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

STM HSS 94% 95% 60% Editors 2 months is realistic Reviewers 2 months to deliver report Authors 2 months to receive report 96% 93% 44% Editors 2 months is realistic Reviewers 2 months to deliver report Authors 2 months to receive report 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review As an author, I am usually kept well informed about the progress of my article through the peer review process STM HSS 10 – strongly agree1 – strongly disagree

1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review As an author, I would like to see average peer review times, from submission to decision to publish, displayed on a journal's website STM HSS 10 – strongly agree1 – strongly disagree

1 Methodology 2 Ideal world & real world objectives STM HSS AuthorsReviewersEditors Very comfortable 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review Uncomfortable

1 Methodology 2 Ideal world & real world objectives STM HSS Double blind: Neither the author’s nor the reviewers’ names are known to each other AuthorsReviewersEditors Uncomfortable Very comfortable 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review

1 Methodology 2 Ideal world & real world objectives STM HSS Single blind: Only the author’s name is known to the reviewer, but the reviewers’ names are not known to the author AuthorsReviewersEditors Uncomfortable Very comfortable Ethical concerns 4 Timing discrepancies 5 Different models of peer review

1 Methodology 2 Ideal world & real world objectives STM HSS Open: Both the authors’ and reviewers’ names are known to each other AuthorsReviewersEditors & 5.9 Very comfortable Ethical concerns 4 Timing discrepancies 5 Different models of peer review Uncomfortable

1 Methodology 2 Ideal world & real world objectives STM HSS Open and published: Both the authors and reviewers’ names are known to each other and the reviewers’ signed reports are published 4.7 AuthorsReviewersEditors Uncomfortable Very comfortable Ethical concerns 4 Timing discrepancies 5 Different models of peer review

1 Methodology 2 Ideal world & real world objectives STM HSS Post-publication: Online readers comment on, or rate the paper following publication 4.4 AuthorsReviewersEditors Uncomfortable Very comfortable Ethical concerns 4 Timing discrepancies 5 Different models of peer review

1 Methodology 2 Ideal world & real world objectives On double blind “You have to be quite secure about your career to un-blind yourself. I don’t want to offend a future employer or someone sitting on an interview panel” Researcher, Environmental Sciences, UK On open and published “I think this is the most transparent way…it may put some pressure on the reviewer, but it also gives him/her credit…” Reviewer, Humanities, Lebanon On post-publication “This method is limited to those who can actually read the articles (are subscribed) online unless the articles are open access..” Reviewer, Agriculture & Food Science, Zimbabwe 4 Timing discrepancies 5 Different models of peer review 3 Ethical concerns

AuthorsReviewersEditors authorservices.taylorandfrancis.com/ peer-review-in-2015 White Paper | Key Survey Data | References Still to come in Available online now… Twitter… Motivations & Training| Geographic Breakdowns