Peer reviewer training part I: What do we know about peer review?

Slides:



Advertisements
Similar presentations
So you want to get published? Kristen L. Mauk PhD, DNP, RN, CRRN, GCNS-BC, GNP-BC, FAAN President Senior Care Central.
Advertisements

Cathy Jordan, PhD Associate Professor of Pediatrics Director, Children, Youth and Family Consortium University of Minnesota Member, Community Campus Partnerships.
Investigating Geography with the Editorial Collective Geographical Association annual conference, Manchester, April 2009.
1 Publishing in European Journal of Teacher Education 28th August 2010 Kay Livingston, Editor, EJTE Geri Smyth, Co-Editor, EJTE Katie Peace, Publisher,
2: Authors … how to capture and keep them Editors short course © 2012 Pippa Smart.
How to Review a Paper How to Get your Work Published
The Peer Review Process Adapted from a presentation by Richard Henderson, Elsevier Hong Kong.
What happens after submission? Sadeghi Ramin, MD Nuclear Medicine Research Center, Mashhad University of Medical Sciences.
Submission Process. Overview Preparing for submission The submission process The review process.
Insider's guide to getting published Getting your paper to review stage Insights from an editor Steven Dellaportas A/Prof in Accounting Co-editor: MAJ.
Doug Altman Centre for Statistics in Medicine, Oxford, UK
GETTING PUBLISHED Chapter 18.
University of Ottawa Medical Journal Workshop Feb 11, 2014 Diane Kelsall MD MEd Deputy Editor, CMAJ and Editor, CMAJ Open.
Doctoral Training Workshops Getting published and the reviewing process Steve Potter, Alex Borda-Rodriguez, Sue Oreszczyn and Julius Mugwagwa February.
Responsible Conduct of Research, Scholarship, and Creative Activities Peer Review Responsible Conduct of Research, Scholarship, and Creative Activities.
How does the process work? Submissions in 2007 (n=13,043) Perspectives.
Experiences from Editing a Journal: Case EJOR Jyrki Wallenius Helsinki School of Economics EJOR Editor Outgoing Editor till June 30, 2005 EJOR.
Reasons of rejection Paolo Russo Università di Napoli Federico II Dipartimento di Fisica Napoli, Italy 8th ECMP, Athens, Sep. 13th,
Radiography Peer Review - make your contribution Dr Pauline Reeves Associate Editor (Clinical Imaging)
ALEC 604: Writing for Professional Publication Week 10: Faculty/Peer Reviews.
II THE PUBLICATION PROCESS. Conduct literature review Start the paper Conduct study/analyze data Organize/summarize results succinctly Get early, frequent.
Manuscript Writing and the Peer-Review Process
Introduction to evidence based medicine
Publication bias in clinical trials Kamran Abbasi Deputy editor, BMJ.
FISH 521 Peer review. Peer review Mechanics Advantages Challenges Solutions.
Are the results valid? Was the validity of the included studies appraised?
Electronic Submission and Reviewing Methodology Hooman Momen Editor Bulletin of the World Health Organization.
Writing for Publication Nick Lipley Managing editor Claire Picton Editor Andrew Rideout Editorial adviser RCN Publishing Ltd.
Dr. Dinesh Kumar Assistant Professor Department of ENT, GMC Amritsar.
Getting published : how to enhance your chances of publishing in international journals PhD-school Faculty of Social Sciences, 1 december 2008 Wouter Vandenabeele.
So you want to publish an article? The process of publishing scientific papers Williams lab meeting 14 Sept 2015.
Thomas HeckeleiPublishing and Writing in Agricultural Economics 1 … 4 The review process  Overview  The author’s role  The referee’s role  The editor’s.
Motivating the Best Referees Moderator: Lynne Honigmann Scribe: Dennis Shasha.
Peer reviewer training part II: What do editors want from reviewers? Dr Trish Groves Deputy editor, BMJ.
How to Get Published in (better) International Journals Hui Wang, MD, PhD Editorial Director John Wiley & Sons.
November Does OA threaten journals’ quality? Some publishers think so… “public access equals government censorship” PR's 'pit bull' Eric Dezenhall,
Online Editorial Management On-line Management of Scholarly Journals Mahmoud Saghaei.
MedEdPORTAL Reviewer Tutorial Contact MedEdPORTAL
Dr Andrea Whittaker, Asia Institute, University of Melbourne Publishing in international journals: Realities, tips and tricks.
"Writing for Researchers" Monday, July :35-3:45PM. Laurence R Weatherley– Spahr Professor of Chemical Engineering, Department of Chemical and.
REVIEWING MANUSCRIPTS TIPS FOR REVIEWING MANUSCRIPTS IN PEER REVIEWED JOURNALS Bruce Lubotsky Levin, DrPH, MPH Associate Professor & Head Dept. of Community.
Http// 1 st Peer-Reviewer Retreat 2006 During USCAP Annual Meeting, Hyatt Regency Atlanta, Atlanta, Georgia February 12, 2006.
INANE Meeting –Principles of Transparency and Best Practice in Scholarly Publishing Charon Pierson Geraldine Pearson August 5, 2015.
FEMS Microbiology Ecology Getting Your Work Published Telling a Compelling Story Working with Editors and Reviewers Jim Prosser Chief Editor FEMS Microbiology.
Manuscript Review Prepared by Noni MacDonald MD FRCPc Editor-in-Chief Paediatrics and Child Health Former Editor-in -Chief CMAJ
Publishing Papers© Dr. Ayman Abdel-Hamid, CS5014, Fall CS5014 Research Methods in CS Dr. Ayman Abdel-Hamid Computer Science Department Virginia.
Editorial decision making and common reasons for rejection Shirin Heidari.
Dr. Sundar Christopher Navigating Graduate School and Beyond: Sow Well Now To Reap Big Later Writing Papers.
ACADEMIC PUBLISHING How a manuscript becomes an article.
Warwick Business School James Hayton Associate Dean & Professor of HRM & Entrepreneurship Editor in Chief Human Resource Management (Wiley) Past Editor:
PSYA4- research methods Section C. Validating new knowledge The role of peer review the assessment of scientific work by others who are experts in the.
Research methods revision The next couple of lessons will be focused on recapping and practicing exam questions on the following parts of the specification:
해외저널에 논문게재하기 고려대학교 이상민 “scholarly work is rooted in the lively exchange of ideas – conversation at its best” (Huff 1999)
Getting published Sue Symons Editorial Manager Karen Mattick
Publishing research in a peer review journal: Strategies for success
How does publication in psychological science work?
Publishing without tears.
Role of peer review in journal evaluation
What Can You Publish? What Is the Scope of Papers accepted by AAS
Dura censor sed censor: A reviewer’s ongoing audit of 11 years in the journals of the European Association for Cardio-thoracic Surgery Aristotle D. Protopapas,
Dealing with reviewer comments
Adam J. Gordon, MD MPH FACP DFASAM
Writing Scientific Papers: From Theory to Practice
Improving the Standards of Reporting of Clinical Trial Data
«Организация процесса прохождения рукописей в Springer Nature»
Manuscripts and publishing
Joyce Backus Associate Director, Library Operations
Writing an Effective Research Paper
Presentation transcript:

Peer reviewer training part I: What do we know about peer review? Dr Trish Groves Deputy editor, BMJ

What do editors want from papers? Importance Originality Relevance to readers Usefulness to readers and, ultimately, to patients Truth Excitement/ “wow” factor Clear and engaging writing We make many of these judgments ourselves, but also rely on reviewers’ opinions

Peer review As many processes as journals or grant giving bodies No operational definition--usually implies “external review” Largely unstudied till 1990s Benefits through improving what’s published rather than sorting wheat from chaff At BMJ we do a lot of internal review as well, and consider it an important part of our peer review process

What is peer review? Review by peers Includes: internal review (by editorial staff) external review (by experts in the field) Peer review is not used only by journals - also by grant applications, ethics committees, and for conference papers and abstracts. But we're talking here about peer review and critical appraisal for publication.

BMJ papers All manuscripts handled by our online editorial office at http://submit.bmj.com The website uses a system called Benchpress Reviewers recruited by invitation, through volunteering, and by authors’ suggestions Database also includes all authors We monitor reviewers’ workload for BMJ We rate reviewers’ reports using a 3 point scale

BMJ peer review process I 7000 research papers, 7% accepted approximate numbers at each stage: 1000 rejected by one editor within 48 hours further 3000 rejected with second editor within one week of submission 3000 read by senior editor; further 1500 rejected 1500 sent to two reviewers; then 500 more rejected approx 1000 screened by clinical epidemiology editor and more rejected

BMJ peer review process II 400-500 to weekly manuscript meeting attended by the Editor, an external editorial adviser (a specialist or primary care doctor) and a statistician.. …and the full team of BMJ research editors, plus the BMJ clinical epidemiology editor 350 research articles accepted, usually after revision value added by commissioned editorials and commentaries

BMJ peer review process III always willing to consider first appeals--but must revise the paper, respond to criticisms, not just say subject’s important perhaps 20% accepted on appeal no second appeals; always ends in tears; plenty of other journals Most papers end up being published somewhere. See: Lock, S. A difficult balance: editorial peer review in medicine. London: BMJ, 1991. 68% of papers rejected by the BMJ during 7 months of 1979 were eventually published elsewhere, most in specialist journals. A quarter, however, remained unpublished. Relman AS. Are journals really quality filters? In: Goffman W, Bruer JT, Warren KS, eds. Research on selective information systems. New York: Rockefeller Foundation, 1980. Random sample of 300 papers rejected by NEJM in 1975. Questionnaire showed that, among the 55% who replied, four fifths of authors said their papers had been published elsewhere. Only a fifth of these, however, had been revised according to peer review comments received before submission to final journal. Must interpret these findings cautiously. Both studies are old - things may have changed since.

What we know about peer review Research evidence

Peer review processes “Stand at the top of the stairs with a pile of papers and throw them down the stairs. Those that reach the bottom are published.” “Sort the papers into two piles: those to be published and those to be rejected. Then swap them over.” Quotes by former editors of the BMJ and Lancet – tongue in cheek, of course But how do we know that peer review is more reliable than this?

Some problems Means different things at different journals Slow Expensive Subjective Biased Open to abuse Poor at detecting errors Almost useless at detecting fraud

Is peer review reliable? (How often do two reviewers agree?) NEJM (Ingelfinger F 1974) Rates of agreement only “moderately better than chance” (Kappa = 0.26) Agreement greater for rejection than acceptance Grant review Cole et al, 1981 – real vs sham panel, agreed on 75% of decisions Hodgson C, 1997 – two real panels reviewing the same grants, 73% agreement Are two reviewers enough? Fletcher and Fletcher 1999 - need at least six reviewers, all favouring rejection or acceptance, to yield a stats significant conclusion (p<0.05) Ingelfinger FJ. Peer review in biomedical publication. Am J Med 1974;56:686-92. Cole S, Cole J, Simon G. Chance and consensus in peer review. Science 1981;214:881-6. Hodgson C. How reliable is peer review? A comparison of operating grant proposals simultaneously submitted to two similar peer review systems. J Clin Epidemiol 1997;50:1189-95. Fletcher RH, Fletcher SW. The effectiveness of editorial peer review. In: Peer review in health sciences.Godlee F, Jefferson T eds. London BMJ Publishing Group,1999:45-56

Should we mind if reviewers don’t agree? Very high reliability might mean that all reviewers think the same Reviewers may be chosen for differing positions or areas of expertise Peer review decisions are like diagnostic tests: false positives and false negatives are inevitable (Kassirer and Campion, 1994) Larger journals ask reviewers to advise on publication, not to decide Kassirer JP, Campion EW. Peer review:crude and understudied, but indispensable. JAMA 1994;272:96-7

Bias Author-related Prestige (author/institution) Gender Where they live and work Paper-related Positive results English language Author-related bias Bias towards successful researchers - Merton RK. Science 1968;159:56-6 Bias against women (applicants to Swedish MRC for postdoc fellowships) - Wenneras C, Wold A. Nature 1997;387:341-3 Bias against institution - next slide Publication bias - if the research question is important and interesting, the answer should be less important

Prestigious institution bias Peters and Ceci, 1982 Resubmitted 12 altered articles to psychology journals that had already published them Changed: title/abstract/introduction - only slightly authors’ names name of institution, from prestigious to unknown fictitious name (eg. “Tri-Valley Center for Human Potential”) Peters DP, Ceci SJ. Behavioural and Brain Sciences 1982;5:187-95 Randomly selected one paper from each of 13 influential peer reviewed psychology journals with high reject rates (>80%). Authors were from prestigious institutions. All papers published in past 18-32 months, all with above average citations Got permission from authors but not from editors or reviewers

Peters and Ceci - results Three articles recognised as resubmissions One accepted Eight rejected (all because of poor study design, inadequate statistical analysis, or poor quality: none on grounds of lack of originality) One paper had to be withdrawn because journal had changed its policy on type of paper accepted, leaving 12 in study. Study much criticised and called unethical (editors and reviewers not consented, copyright law probably violated)

How easy is it to hide authors’ identity? Not easy In RCTs of blinded peer review, reviewers correctly identified author or institution in 24-50% of cases Authors tend to cite their own work in references RCTs: McNutt RA, Evans AT, Fletcher RH, Fletcher SW.The effects of blinding on the quality of peer review: a randomized controlled trial. JAMA 1990;263:1371-6 An RCT. Godlee F, Gale CR, Martyn C. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA 1998;280:237-40 van Rooyen S, Godlee F, Evans S, Smith R, Black N.Effect of blinding and unmasking on the quality of peer review: a randomized controlled trial. JAMA 1998;280:234-7 (second paper from RCT above. done at the BMJ) Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D. Does masking author identity improve peer review quality? A randomized controlled trial. JAMA 1998;280:240-2 Soprano Elizabeth Schwarzkopf requested all her own records on the BBC radio programme Desert Island Discs: BMJ has thought of giving a Schwarzkopf award for self-citation And, anyway, reviewers tend to know who's doing what work in their own field

Reviewers identified (open review) – results of RCTs Asking reviewers to sign their reports in RCTs made no difference to the quality of reviews or recommendations made Godlee et al, 1998 van Rooyen et al, 1998 van Rooyen et al ,1999 Same references as earlier slide plus van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomized controlled trial. BMJ 1999;318:23-7

Open review on the web Various experiments and evaluations are underway… Long history of this in other disciplines eg physics research Open review of articles posted on the web can be done before publication or afterwards. It can be completely open - a free for all - completely closed, or something in between. It can also be moderated by invited reviewers. MJA online peer review trial. www.mja.com.au/public/papers/papers.html Medical Journal of Australia study - articles were electronically published and then, for a few weeks, the journal invited postpublication review on web from readers. Authors were encouraged to revise the paper accordingly before final print publication.

What makes a good reviewer? – results of RCTs Aged under 40 Good institution Methodological training (statistics & epidemiology) Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review in a general medical journal. JAMA 1998;280:231-3. Evans et al.The characteristic of peer reviewers who produce good-quality reviews J Gen Intern Med 1993;8:422-8. 226 reviewers of 131 papers submitted to the journal. 43% of reviews were good (on a 5 pt editors' scale). The characteristics on this slide had 87% chance of predicting a good review. Could editors be biased if they know reviewers??

What might improve the quality of reviews? Reward/credit/acknowledgement? Careful selection? Training? Greater accountability (open review on web)? Interaction between author and reviewer (real time open review)? BMJ pays reviewers but does not give feedback on performance