Peer reviewer training part I: What do we know about peer review? Dr Trish Groves Deputy editor, BMJ

What do editors want from papers? Importance Originality Relevance to readers Usefulness to readers and, ultimately, to patients Truth Excitement/ “wow” factor Clear and engaging writing We make many of these judgments ourselves, but also rely on reviewers’ opinions

Peer review As many processes as journals or grant giving bodies No operational definition--usually implies “external review” Largely unstudied till 1990s Benefits through improving what’s published rather than sorting wheat from chaff At BMJ we do a lot of internal review as well, and consider it an important part of our peer review process

What is peer review? Review by peers Includes: internal review (by editorial staff) external review (by experts in the field) Peer review is not used only by journals - also by grant applications, ethics committees, and for conference papers and abstracts. But we're talking here about peer review and critical appraisal for publication.

BMJ papers All manuscripts handled by our online editorial office at http://submit.bmj.com The website uses a system called Benchpress Reviewers recruited by invitation, through volunteering, and by authors’ suggestions Database also includes all authors We monitor reviewers’ workload for BMJ We rate reviewers’ reports using a 3 point scale

BMJ peer review process I 7000 research papers, 7% accepted approximate numbers at each stage: 1000 rejected by one editor within 48 hours further 3000 rejected with second editor within one week of submission 3000 read by senior editor; further 1500 rejected 1500 sent to two reviewers; then 500 more rejected approx 1000 screened by clinical epidemiology editor and more rejected

BMJ peer review process II 400-500 to weekly manuscript meeting attended by the Editor, an external editorial adviser (a specialist or primary care doctor) and a statistician.. …and the full team of BMJ research editors, plus the BMJ clinical epidemiology editor 350 research articles accepted, usually after revision value added by commissioned editorials and commentaries

BMJ peer review process III always willing to consider first appeals--but must revise the paper, respond to criticisms, not just say subject’s important perhaps 20% accepted on appeal no second appeals; always ends in tears; plenty of other journals Most papers end up being published somewhere. See: Lock, S. A difficult balance: editorial peer review in medicine. London: BMJ, 1991. 68% of papers rejected by the BMJ during 7 months of 1979 were eventually published elsewhere, most in specialist journals. A quarter, however, remained unpublished. Relman AS. Are journals really quality filters? In: Goffman W, Bruer JT, Warren KS, eds. Research on selective information systems. New York: Rockefeller Foundation, 1980. Random sample of 300 papers rejected by NEJM in 1975. Questionnaire showed that, among the 55% who replied, four fifths of authors said their papers had been published elsewhere. Only a fifth of these, however, had been revised according to peer review comments received before submission to final journal. Must interpret these findings cautiously. Both studies are old - things may have changed since.

What we know about peer review Research evidence

Peer review processes “Stand at the top of the stairs with a pile of papers and throw them down the stairs. Those that reach the bottom are published.” “Sort the papers into two piles: those to be published and those to be rejected. Then swap them over.” Quotes by former editors of the BMJ and Lancet – tongue in cheek, of course But how do we know that peer review is more reliable than this?

Some problems Means different things at different journals Slow Expensive Subjective Biased Open to abuse Poor at detecting errors Almost useless at detecting fraud

Is peer review reliable? (How often do two reviewers agree?) NEJM (Ingelfinger F 1974) Rates of agreement only “moderately better than chance” (Kappa = 0.26) Agreement greater for rejection than acceptance Grant review Cole et al, 1981 – real vs sham panel, agreed on 75% of decisions Hodgson C, 1997 – two real panels reviewing the same grants, 73% agreement Are two reviewers enough? Fletcher and Fletcher 1999 - need at least six reviewers, all favouring rejection or acceptance, to yield a stats significant conclusion (p<0.05) Ingelfinger FJ. Peer review in biomedical publication. Am J Med 1974;56:686-92. Cole S, Cole J, Simon G. Chance and consensus in peer review. Science 1981;214:881-6. Hodgson C. How reliable is peer review? A comparison of operating grant proposals simultaneously submitted to two similar peer review systems. J Clin Epidemiol 1997;50:1189-95. Fletcher RH, Fletcher SW. The effectiveness of editorial peer review. In: Peer review in health sciences.Godlee F, Jefferson T eds. London BMJ Publishing Group,1999:45-56

Should we mind if reviewers don’t agree? Very high reliability might mean that all reviewers think the same Reviewers may be chosen for differing positions or areas of expertise Peer review decisions are like diagnostic tests: false positives and false negatives are inevitable (Kassirer and Campion, 1994) Larger journals ask reviewers to advise on publication, not to decide Kassirer JP, Campion EW. Peer review:crude and understudied, but indispensable. JAMA 1994;272:96-7

Bias Author-related Prestige (author/institution) Gender Where they live and work Paper-related Positive results English language Author-related bias Bias towards successful researchers - Merton RK. Science 1968;159:56-6 Bias against women (applicants to Swedish MRC for postdoc fellowships) - Wenneras C, Wold A. Nature 1997;387:341-3 Bias against institution - next slide Publication bias - if the research question is important and interesting, the answer should be less important

Prestigious institution bias Peters and Ceci, 1982 Resubmitted 12 altered articles to psychology journals that had already published them Changed: title/abstract/introduction - only slightly authors’ names name of institution, from prestigious to unknown fictitious name (eg. “Tri-Valley Center for Human Potential”) Peters DP, Ceci SJ. Behavioural and Brain Sciences 1982;5:187-95 Randomly selected one paper from each of 13 influential peer reviewed psychology journals with high reject rates (>80%). Authors were from prestigious institutions. All papers published in past 18-32 months, all with above average citations Got permission from authors but not from editors or reviewers

Peters and Ceci - results Three articles recognised as resubmissions One accepted Eight rejected (all because of poor study design, inadequate statistical analysis, or poor quality: none on grounds of lack of originality) One paper had to be withdrawn because journal had changed its policy on type of paper accepted, leaving 12 in study. Study much criticised and called unethical (editors and reviewers not consented, copyright law probably violated)

How easy is it to hide authors’ identity? Not easy In RCTs of blinded peer review, reviewers correctly identified author or institution in 24-50% of cases Authors tend to cite their own work in references RCTs: McNutt RA, Evans AT, Fletcher RH, Fletcher SW.The effects of blinding on the quality of peer review: a randomized controlled trial. JAMA 1990;263:1371-6 An RCT. Godlee F, Gale CR, Martyn C. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA 1998;280:237-40 van Rooyen S, Godlee F, Evans S, Smith R, Black N.Effect of blinding and unmasking on the quality of peer review: a randomized controlled trial. JAMA 1998;280:234-7 (second paper from RCT above. done at the BMJ) Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D. Does masking author identity improve peer review quality? A randomized controlled trial. JAMA 1998;280:240-2 Soprano Elizabeth Schwarzkopf requested all her own records on the BBC radio programme Desert Island Discs: BMJ has thought of giving a Schwarzkopf award for self-citation And, anyway, reviewers tend to know who's doing what work in their own field

Reviewers identified (open review) – results of RCTs Asking reviewers to sign their reports in RCTs made no difference to the quality of reviews or recommendations made Godlee et al, 1998 van Rooyen et al, 1998 van Rooyen et al ,1999 Same references as earlier slide plus van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomized controlled trial. BMJ 1999;318:23-7

Open review on the web Various experiments and evaluations are underway… Long history of this in other disciplines eg physics research Open review of articles posted on the web can be done before publication or afterwards. It can be completely open - a free for all - completely closed, or something in between. It can also be moderated by invited reviewers. MJA online peer review trial. www.mja.com.au/public/papers/papers.html Medical Journal of Australia study - articles were electronically published and then, for a few weeks, the journal invited postpublication review on web from readers. Authors were encouraged to revise the paper accordingly before final print publication.

What makes a good reviewer? – results of RCTs Aged under 40 Good institution Methodological training (statistics & epidemiology) Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review in a general medical journal. JAMA 1998;280:231-3. Evans et al.The characteristic of peer reviewers who produce good-quality reviews J Gen Intern Med 1993;8:422-8. 226 reviewers of 131 papers submitted to the journal. 43% of reviews were good (on a 5 pt editors' scale). The characteristics on this slide had 87% chance of predicting a good review. Could editors be biased if they know reviewers??

What might improve the quality of reviews? Reward/credit/acknowledgement? Careful selection? Training? Greater accountability (open review on web)? Interaction between author and reviewer (real time open review)? BMJ pays reviewers but does not give feedback on performance