1. How does publication in psychological science work?

2. How do papers get published?
Not by the Research Stork
Peer review process

3. Peer Review Process Paper is submitted to journal
Internal review (triage)
Sent for external review
Paper is accepted, revised, or rejected
Editor makes decision
External reviewers send recommendations
Choosing the correct journal is very important. Your paper must fit in terms of content, structure, and quality.
The editor summarizes the main points of the reviewers and makes a decision. Decisions include revise and resubmit, reject, or accept outright.
The process will repeat itself if the paper is not accepted. Either at this journal if a chance to revise is offered, or at another journal if the author decides to submit elsewhere.
The editor decides if the paper has any chance of being accepted. If it does, it goes on to the next step. If it doesn’t, it will be rejected outright.
The paper is sent to others in the field who are familiar with the research topic and asked to evaluate the study. The paper is usually anonymous here.
The reviewers critique the study and make comments in reference to improvements that should be made.

4. Peer review is a flawed process
Slow
Biased
Vulnerable
Peer review is a flawed process
Unreliable
Poor quality

5. Peer review is slow
Papers take from several weeks to several months or longer from submission to publication
My personal record:
3/27/12 : Initial submission
8/20/12 : Decision – revise
11/25/12 : Revision submitted
4/2/13: Decision – revise (Again)
5/21/13: Re-revision submitted
10/28/13 : Decision – revise a few things
10/28/13: Re-re-revision submitted
1/3/14: Acceptance (OMFG finally)
1 year, 9 months
(647 days!)

6. Peer review is biased Paper-related biases Author-related biases
Strong bias towards positive results
Author-related biases

7. Author-related biases
Reviews are ideally “blinded”
Blinded = You don’t know who the authors (or reviewers) are
This does not always work well
Reviewers correctly identified author or institution in nearly 50% of all cases.

8. Why is it a problem if you know the author’s name?
Matthew Effect
“The rich get richer and the poor get poorer”
Easier for more famous researchers to have papers accepted.
Robert Merton
Merton, R. K. (1968). The Matthew effect in science. Science, 159(3810),

9. Why is it a problem if you know the author’s institution?
Prestigious Institution Bias (Peters & Ceci, 1982)
Resubmitted 12 papers to journals that had published them
Original authors were from prestigious institutions
Names and institutions were changed
Fictitious institutions were meant to not sound prestigious
Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5(02),

10. Why is it a problem if you know the author’s institution?
Three articles identified as resubmissions
Of the nine sent to review:
8 were rejected
“serious methodological flaws”
1 accepted
Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5(02),

11. Who picks reviewers?
Two ways: editor picks reviewers, author picks reviewers
When editor picks, presumably bias is reduced.
But this takes longer, might not be experts, Google is probably used
When author picks, presumably bias is increased
People pick their friends or “easy reviewers”

12. Does it matter who picks reviewers?
For quality: nope
Quality difference between editor and author chosen reviews (5 point scale) = 0.05 , 0.10 (two studies)
For acceptance: yes
Author selected reviews are 2.66 times more likely to suggest revise (instead of reject)
1.64 times more likely to suggest accept (instead of reject or revise)
But…
No difference in actual editorial decisions
Schroter, S., Tite, L., Hutchings, A., & Black, N. (2006). Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. JAMA, 295(3),
Wager, E., Parkin, E. C., & Tamber, P. S. (2006). Are reviewers suggested by authors as good as those chosen by editors? Results of a rater-blinded, retrospective study. BMC medicine, 4(1), 13.

13. Reviewers are not reliable
Reliable = consistent
Peters & Ceci (1982): 1 of 12 papers reaccepted
Agreement between reviewers is bad
Rothwell & Martyn (2000)
Kappas between two reviewers
Journal A: k = .28
Journal B: k = .12
Kappa = measure of consistency.
Should be > .7
Rothwell, P. M., & Martyn, C. N. (2000). Reproducibility of peer review in clinical neuroscience Is agreement between reviewers any greater than would be expected by chance alone?. Brain, 123(9),

14. Author-related biases
Reviews are ideally “blinded”
Blinded = You don’t know who the authors (or reviewers) are
This does not always work well
Reviewers correctly identified author or institution in nearly 50% of all cases.

15. Reviews are not always good quality
Snell & Spencer (2005)
Average time spent on review is 3 hours
Probably demand characteristics here
Range 30 – 810 minutes
Only 14% of reviewers had received training
Reviewers are not guaranteed to have spent a long time on a review
Snell, L., & Spencer, J. (2005). Reviewers' perceptions of the peer review process for a medical education journal. Medical education, 39(1),

16. Reviews are not always good quality
Not guaranteed to have spent a long time on a review
Not guaranteed to be experts, or even be the actual reviewers
Common practice of having students do reviews
Reviewer #1: The meaning in life was a resilience factor for suicide, as the result I had some suggestion
1.the title could be modified " the meaning in life as a protector for the suicide"
2.the mediator model of the meaning in life between the impulsivity and suicide ideation was need to analysis

17. What if we make the reviewers known to the authors?
Doesn’t work 
Review quality (1-5 scale)
Anonymous = 3.06 (SD = 0.72)
Identified = 3.09 (SD = 0.68)
No significant difference (p = .68)
Errors detected
Anonymous = 2.0 errors
Identified = 1.75 errors
No significant difference
Godlee, F., Gale, C. R., & Martyn, C. N. (1998). Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. Jama, 280(3),
Van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. BMJ, 318,

18. What makes a good reviewer?
Many qualities are unassociated with quality
Actively involved in research, occupying academic positions, members of research funding bodies
Under 40
Has more to do with need to do professional service than age
Has methodological training
Black, N., Van Rooyen, S., Godlee, F., Smith, R., & Evans, S. (1998). What makes a good reviewer and a good review for a general medical journal?. Jama, 280(3),

19. Peer review is vulnerable