1. Use of critical appraisal as a tool for peer instruction and assessment in post-graduate Epidemiology
Prof Philip Baker1, Dr Daniel Demant1,2, Daniel Francis1 & Abby Cathcart1
1- Queensland University of Technology, Brisbane
2-University of Technology Sydney

2. Presentation
Critical appraisal - what is it, why use it for assessment, pitfalls
Peer instruction
Integration of critical appraisal and peer instruction
Results
Future work

3. Aroma?

4. Trustworthiness = critical appraisal
May be defined as:
The process of systematically examining research evidence to assess its validity, results and relevance before using it to inform a decision.
Alison Hill, Critical Appraisal Skills Programme, Institute of Health Sciences, Oxford
What this means :
An assessment of the validity/claims of public health problems and interventions
An evaluation of the validity of past research studies/evaluations (not evaluating an intervention itself)

5. Why use critical appraisal for evaluation?

6. Pitfalls Essay No need for discussion Potential for contract cheating
Group work, uneven contribution

7. Critical appraisal in real-world
Need to appraise what we read
Journal clubs
Research work
Work-places where results are crucially discussed
Systematic reviews
Other?

8. Pedagogy of Peer Instruction
Engages students in constructing own understanding of concepts
Students individually respond to a question
Discuss with peers
Respond to the same question
(Mazur, 1997; Crouch & Mazur, 2001)

10. Peer instruction (Maznur 1997)
Questions posed
Students given time to think
Students record individual answers
Student’s convince their neighbour (peer instruction)
Students record revised answers
Feedback to teacher
Instructors explanation of correct answer

11. Peer instruction
Concept questions: higher order questions yields better student results (Finkelstein) rather than recall
Individual thinking? (Nichol and Boyle 2003, Smith 2009)
Are Clickers necessary?
Extension to a critical appraisal exercise?
The best type of questions for peer instruction are concept questions. Evidence shows student learn more effectively with these questions when peer instruction is applied
In the second step of peer instruction students are asked to think individually (Nichol and Boyle 2003) showed that 82% students found a preference to individual response to force them to think about and identify an answer. This leads to more active and engaged during the peer instruction. 90% lead to deeper thinking about the topic.
A group discussion after individual thinking leads to deeper thinking about the topic.’
Peer instruction often classroom; but other approaches with flashcards, also evidence of paper based approaches to the method

12. Post-graduate Epidemiology at QUT
Can the PI principles be replicated to paper exercise to enhance a critical appraisal assessment?

13. Introducing...... Quality Assessment Tool for Quantitative Studies
(Effective Public Health Practice Project)
Dictionary:

14. Selection bias Recruiting study population
Differences in the way patients are accepted or rejected for a trial, and the way in which interventions are assigned to individuals
Difficult in public health studies
The major difference between trials and observational studies has to do with the selection bias and the need to identify and account for potential confounders in observational studies.
Non-randomised studies – may produce groups that are unbalanced at the beginning of the study, thereby differences in outcomes cannot confidently be attributed to the effects of the intervention. Not possible to adjust for unknown or unmeasured confounding variables. 14

15. Component A: Selection Bias
A) Are the individuals selected to participate in the study likely to be representative of the target population?
1. Very likely
2. Somewhat likely
3.Not likely
4. Can’t tell

16. Component A: Selection Bias
B) What percentage of the selected individuals/schools, etc agreed to participate?
% agreement
2. 60 – 79% agreement
3. less than 60% agreement
4. Not applicable
5. Can’t tell

17. SELECTION BIAS (see EPHPP dictionary)
Good / Strong: The selected individuals are very likely to be representative of the target population (Q1 is 1) and there is greater than 80% participation (Q2 is 1).
Fair / Moderate : The selected individuals are at least somewhat likely to be representative of the target population (Q1 is 1 or 2); and there is % participation (Q2 is 2). ‘Moderate’ may also be assigned if Q1 is 1 or 2 and Q2 is 5 (can’t tell).
Poor / Weak : The selected individuals are not likely to be representative of the target population (Q1 is 3); or there is less than 60% participation (Q2 is 3) or selection is not described (Q1 is 4); and the level of participation is not described (Q2 is 5).

18. Classroom with PI
Assessment with PI
Teach study design and risks of bias with Peer Instruction using ‘clickers’, students learn discussion
Find discussion partner and sign-up as a pair
Undertake individual completion of assigned study with EPHPP tool
Upload substantial pre-discussion draft to Turnitin by the due date
Introduce EPHPP critical appraisal tool
Discuss with peer discussion each item of the EPHPP tool (peer instruction)
In-class RCT simulation ‘Live the Trial’
Combine discussion and results and submit as a pair (same grade)
Turnitin
Opportunity to modify work and submit individually (individual) Turnitin
Practice use of tool during RCT Peer Instruction

19. Approaches to Learning & Teaching in public health
Learning epidemiology can be “dull and boring”:
Make it engaging and fun.
Mock randomised controlled trial Ref: Baker APJPH
Move past-passive learning environments

20. Peer instruction
Independent work, structured essay. Prepare an essay using a critical appraisal tool (EPHPP)
Work is first done independently like systematic reviewers
All students must upload a copy of their draft to Turnitin
Students then swap and then discuss agreements and disagreements
Students can then submit combined or as individual

21. Peer instruction
Must understand and apply core concepts to problems, not memorise
Must argue their differences with peers, Pedagogy (Maznur) says students with misunderstanding have a tendency to undo themselves when they try to explain themselves.
Students learn through discussion
Reflection of process is required

22. Peer instruction Mimics the Systematic review process
Drafts are checked
Students who didn’t do the work can’t engage in a meaning full discussion
Stealing from other student easy to identify through Turnitin (several students caught for misconduct).
Purchased essay: Reported by partner “inability to engage in a meaningful discussion.”
? No change from draft to final submission.
Must have consistent writing style throughout the essay.

23. Analysis – What are the possible effects of discussion upon students grades?
Student who did well on midterm exam (>65%) paired with student who did poorly (<60%) (n=19 of 94 enrolled students)
21% higher grade (>5%)
47.4% lower grade (< -5%)
31.6% no change in their grade (-4% to +4%)
Grade may go down (why?)

24. Analysis – What are the possible effects of discussion upon students grades?
Student who did ‘poorly’ on midterm exam (<60%) paired with student who did ‘well’ (>65%) (n=18 of 94 enrolled students)
81.5% higher grade (>5%)
3.7% lower grade (< -5%)
14.8% no change in their grade (-4% to +4%)
Benefits previously poor performing students.

25. Further research In depth analysis of grades over 4 semesters
Student on-line survey on perceptions and experience

26. Conclusions
Critical appraisals present an authentic form of assessment for students
Potential to improve student understanding of key concepts
Pitfalls can be reduced