How does publication in psychological science work?

How do papers get published? Not by the Research Stork Peer review process

Peer Review Process Paper is submitted to journal Internal review (triage) Sent for external review Paper is accepted, revised, or rejected Editor makes decision External reviewers send recommendations Choosing the correct journal is very important. Your paper must fit in terms of content, structure, and quality. The editor summarizes the main points of the reviewers and makes a decision. Decisions include revise and resubmit, reject, or accept outright. The process will repeat itself if the paper is not accepted. Either at this journal if a chance to revise is offered, or at another journal if the author decides to submit elsewhere. The editor decides if the paper has any chance of being accepted. If it does, it goes on to the next step. If it doesn’t, it will be rejected outright. The paper is sent to others in the field who are familiar with the research topic and asked to evaluate the study. The paper is usually anonymous here. The reviewers critique the study and make comments in reference to improvements that should be made.

Peer review is a flawed process Slow Biased Vulnerable Peer review is a flawed process Unreliable Poor quality

Peer review is slow Papers take from several weeks to several months or longer from submission to publication My personal record: 3/27/12 : Initial submission 8/20/12 : Decision – revise 11/25/12 : Revision submitted 4/2/13: Decision – revise (Again) 5/21/13: Re-revision submitted 10/28/13 : Decision – revise a few things 10/28/13: Re-re-revision submitted 1/3/14: Acceptance (OMFG finally) 1 year, 9 months (647 days!)

Peer review is biased Paper-related biases Author-related biases Strong bias towards positive results Author-related biases

Author-related biases Reviews are ideally “blinded” Blinded = You don’t know who the authors (or reviewers) are This does not always work well Reviewers correctly identified author or institution in nearly 50% of all cases.

Why is it a problem if you know the author’s name? Matthew Effect “The rich get richer and the poor get poorer” Easier for more famous researchers to have papers accepted. Robert Merton Merton, R. K. (1968). The Matthew effect in science. Science, 159(3810), 56-63.

Why is it a problem if you know the author’s institution? Prestigious Institution Bias (Peters & Ceci, 1982) Resubmitted 12 papers to journals that had published them Original authors were from prestigious institutions Names and institutions were changed Fictitious institutions were meant to not sound prestigious Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5(02), 187-195.

Why is it a problem if you know the author’s institution? Three articles identified as resubmissions Of the nine sent to review: 8 were rejected “serious methodological flaws” 1 accepted Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5(02), 187-195.

Who picks reviewers? Two ways: editor picks reviewers, author picks reviewers When editor picks, presumably bias is reduced. But this takes longer, might not be experts, Google is probably used When author picks, presumably bias is increased People pick their friends or “easy reviewers”

Does it matter who picks reviewers? For quality: nope Quality difference between editor and author chosen reviews (5 point scale) = 0.05 , 0.10 (two studies) For acceptance: yes Author selected reviews are 2.66 times more likely to suggest revise (instead of reject) 1.64 times more likely to suggest accept (instead of reject or revise) But… No difference in actual editorial decisions Schroter, S., Tite, L., Hutchings, A., & Black, N. (2006). Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. JAMA, 295(3), 314-317. Wager, E., Parkin, E. C., & Tamber, P. S. (2006). Are reviewers suggested by authors as good as those chosen by editors? Results of a rater-blinded, retrospective study. BMC medicine, 4(1), 13.

Reviewers are not reliable Reliable = consistent Peters & Ceci (1982): 1 of 12 papers reaccepted Agreement between reviewers is bad Rothwell & Martyn (2000) Kappas between two reviewers Journal A: k = .28 Journal B: k = .12 Kappa = measure of consistency. Should be > .7 Rothwell, P. M., & Martyn, C. N. (2000). Reproducibility of peer review in clinical neuroscience Is agreement between reviewers any greater than would be expected by chance alone?. Brain, 123(9), 1964-1969.

Author-related biases Reviews are ideally “blinded” Blinded = You don’t know who the authors (or reviewers) are This does not always work well Reviewers correctly identified author or institution in nearly 50% of all cases.

Reviews are not always good quality Snell & Spencer (2005) Average time spent on review is 3 hours Probably demand characteristics here Range 30 – 810 minutes Only 14% of reviewers had received training Reviewers are not guaranteed to have spent a long time on a review Snell, L., & Spencer, J. (2005). Reviewers' perceptions of the peer review process for a medical education journal. Medical education, 39(1), 90-97.

Reviews are not always good quality Not guaranteed to have spent a long time on a review Not guaranteed to be experts, or even be the actual reviewers Common practice of having students do reviews Reviewer #1: The meaning in life was a resilience factor for suicide, as the result I had some suggestion 1.the title could be modified " the meaning in life as a protector for the suicide" 2.the mediator model of the meaning in life between the impulsivity and suicide ideation was need to analysis

What if we make the reviewers known to the authors? Doesn’t work  Review quality (1-5 scale) Anonymous = 3.06 (SD = 0.72) Identified = 3.09 (SD = 0.68) No significant difference (p = .68) Errors detected Anonymous = 2.0 errors Identified = 1.75 errors No significant difference Godlee, F., Gale, C. R., & Martyn, C. N. (1998). Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. Jama, 280(3), 237-240. Van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. BMJ, 318, 23-27.

What makes a good reviewer? Many qualities are unassociated with quality Actively involved in research, occupying academic positions, members of research funding bodies Under 40 Has more to do with need to do professional service than age Has methodological training Black, N., Van Rooyen, S., Godlee, F., Smith, R., & Evans, S. (1998). What makes a good reviewer and a good review for a general medical journal?. Jama, 280(3), 231-233.

Peer review is vulnerable