Evaluating a journal article

Overview: Start with a brief summary of the article. Give the listener an overview of what was done, what was found, and the author's conclusions.

Rationale: Why was the study done? What research question did it attempt to address? Is it filling a gap in our current understanding of the topic?

Methods (1): How was the study done? Are the methods appropriate for tackling the problem? Are there any major flaws in the design? e.g.: possible order effects, such as effects of practice or fatigue; tasks that were too artificial or too difficult for participants; ceiling effects; floor effects.

Methods (2): How might the study have been best designed? What would have been better, a between-subjects or within-subjects design? Between-subjects: Each condition is performed by a different group of participants Within-subjects: Each participant performs all conditions in the study Need more participantsNeed fewer participants No possibility of order effectsPossibility of order effects Less sensitive (differences between participants as well as between groups) More sensitive (as within- subject variability is reduced)

Methods (3): Participants - Were there enough of them? Does the study have a chance of finding any effects with this number of participants? Are the participants representative of the population to which the experimenter wants to generalise? In neuropsychology studies, were any normal control participants well-matched to the patients in terms of age, IQ, eyesight, etc.?

Results (1): Are the experimenter's claims supported by their results? (Sometimes the actual results do not tally perfectly with the experimenters' own description in the discusssion or abstract of what they found!) Do the experimenters make claims based on findings that are actually not statistically significant?

Results (2): Are there alternative explanations of the results that the experimenters have either not considered or not convincingly ruled out? How do these findings fit with what is already known about the topic? Do they confirm or contradict previous research? How could the study be improved or extended?

Conclusions: Are the author's conclusions warranted by their results? Are there other interpretations? Reading around the topic will help you see how your target article's findings relate to what is already known about it.

General points: Don't nitpick! Don't make trivial criticisms (e.g. of the typeface or general presentation). Focus on the content, not superficial aspects. Evaluation is not just negative criticism. Real-world research is hard to do, time consuming and expensive, so you need to be realistic about participant numbers, etc. Avoid unsupported value judgements like "this was very hard to understand" or "I thought this was an interesting paper". The prestige of either the author or their institution is no guide to the quality of their research - evaluate the research on its own merits.