1. Evaluating a paper (Part 2): Peer review.
Last week we began a three-week series on how to evaluate or judge the strength of a research article. We talked specifically last week about identifying an author’s hypothesis, the claims that they are putting forth in support of that hypothesis and the pieces of evidence that they have presented to support those claims. We looked at this primarily in the context of evaluating figures, and talked about how spending some quality time with the figures of a research article can yield large insights into the quality of the evidence for the author’s claims.
This week, we’re going to expand on our theme of being able to critically judge a research article’s strengths and weaknesses by looking at the process of peer review – how this process is used to try to guarantee a base-line of quality in all published articles, and how approaching an article as though you were a peer reviewer can help put you in the right skeptical mindset for evaluating the strengths of the paper. In class today you’re going to actually start writing your own peer review comments on a research article, which will be your homework assignment for this week.
Obviously, “peer reviewing” an article assumes that you are a “peer” to the author – in other words, that you have experience in the field and a depth of background knowledge about the study. This is very obviously not going to be the case for you for a little while, so next week, we’re going to talk about ways to access other people’s criticism of a research article, within the scientific community. Next week, we’re also going to talk about what happens when the peer review process fails, and something gets through that really shouldn’t have been published in the first place.

2. Revisions/new experiments
Community response
Publish
Write paper
More
experiments
In the first week of class, we talked about what happens to a paper after the “scientific process”. What happens after conducting experiments and testing a hypothesis? This is often presented as the end of the process, you “report your results”. In reality, once you’ve convinced yourself and your professor that your results are valid and your hypothesis is true, you still need to convince the people who run scientific journals to publish your results.
Retract
Corrections
Peer review

3. SEND TO (DIFFERENT)EDITOR
WRITE PAPER
SEND TO EDITOR
REJECT WITHOUT
REVIEW
SUBMIT FOR REVIEW
SEND TO (DIFFERENT)EDITOR
REJECT
MINOR REVISIONS
So once you’ve analyzed your results, made your figures, and written your paper, you send that paper draft off to the editor of whatever journal you want to publish in. That editor will then do one of two things: they might send it back to you, with a flat out rejection, this is called “rejection without review”, in which case, you would send it to a different editor. Rejection without review can happen if the paper doesn’t fit the theme of the journal that you’re submitting to, or if the editor thinks that the science isn’t important enough or flashy enough to go in their journal – or if the science is so bad that the editor, who generally won’t be an expert in your field, can tell that the paper has serious problems.
If the editor doesn’t reject the manuscript, they will submit it for review, meaning they forward your manuscript to a few researchers in the field, generally 2-3 people. Those reviewers then have the ability to reject the paper flat-out, usually this only happens if there are really major flaws in the paper, or to request specific changes to the manuscript, either in terms of writing style, method of presenting the data, or they may actually request that you do additional experiments, if they feel that your conclusions are not solid. After what’s usually mutliple rounds of revision, the paper can be either rejected, or accepted for publication.
MAJOR REVISIONS
ACCEPT

4. Peer review checks for:
Proper controls
Proper analysis of data
Soundness of conclusions
Writing quality
Can be affected by:
Time/effort
Level of experience
Personal biases
The job of the peer reviewer is to make sure that the science in the article is sound, that the data is properly analyzed, that all of the appropriate controls were done, and also, to a somewhat lesser extent, that the writing quality of the article is up to the standards of the journal.
Peer reviewers are not employees of the journal and they’re not paid for the work that they do (check with Marc), so the quality of the review can really depend on how much effort the reviewer is willing to put in. For example, in the sample reviewer comments that I’ve posted to the Resources folder on SmartSite, one reviewer left over 7 pages of comments, while the other only wrote half a page. Now, this could mean that the second reviewer thought the paper was practically perfect, but a more likely explanation is that he or she didn’t spend as much time critically evaluating the manuscript as the other reviewer. The quality of a review can also depend on the amount of experience that the reviewer has in the field. Generally, a journal attempts to recruit reviewers who have quite a bit of experience with the specific area of the research in the paper, but this can depend on the quality of the journal itself. And lastly, there are some instances in which a reviewer may have a personal bias, either against one of the authors of the paper or may have an interest in the manuscript not going to publication, because they have a very similar paper in the works. Likely, this doesn’t happen very often, but it is a conflict of interest to be aware of.

5. Peer review comments General: Somewhat specific: Very specific:
“While I think the analysis itself is a valuable contribution, I strongly believe the authors have not actually backed up one of the major claims they make.”
Somewhat specific:
“Page 4, line 16: The authors state "with the previously sequenced Haloferax volcanii included as a control". I did not see any data analysis performed on this control.”
Very specific:
Page 16, line 18: "However, . . Haloquadratum. . . had no predicted CRISPRs or Cas genes”. The C23 strain of Haloquadratum was shown to have CRISPRs.
Peer review comments can be at varying levels of specificity. On a more general level, they may question the

6. Questions?
In a moment we’re going to move on to applying some of these principles to an analysis of a paper claiming that they’ve developed a way to kill cancer cells, but before we go on, I want to stop a moment and see if there are any questions about the purpose of peer review or how peer comments are structured.

7. In-class practice
“Evaluating an Article” and “Unpacking a Figure” with
Ba IH, Diop MF, Sall BG (In Review) Vulpinic acid inhibits the growth of murine melanoma cells in vitro: 2–6.
Ok, before we take a stab at writing formal peer review comments, we’re first going to use some of the other tools that we’ve already used in previous weeks to develop a basic understanding of this article. You’re going to, with your group, go through this short article and fill out the “Evaluating an Article” and “Unpacking a Figure” handouts. You’re going to have about half an hour for this, so there should plenty of time to dig into this in depth. Then we’re going to have a class discussion about what you’ve learned about the article. And then, you’re going to spend some in-class time with your group writing a set of peer review comments, using a template that I will be handing out to help guide your thoughts. The peer review comments won’t be due until next week, so I want you to take some serious time to think about the article and what you would suggest to improve it, if you were a peer reviewer.

8. Class Discussion
Groups present answers to handouts. (15 mn) (start with Unpacking a figure)

9. In-class practice Peer review comments for
Ba IH, Diop MF, Sall BG (In Review) Vulpinic acid inhibits the growth of murine melanoma cells in vitro: 2–6.
Template provided.
This is also take-home homework.

10. Homework Peer review comments 3 pages double spaced
Include both general and specific comments.
Template provided on SmartSite/Resources