1. Critical Appraisal & Literature review
James M. Bolton MD
Professor of Psychiatry, Psychology, Community Health Sciences, U of M
Adjunct Scientist, Manitoba Centre for Health Policy
Medical Director, Crisis Response Centre & HSC Emergency Psychiatry, WRHA
Director of Research, department of psychiatry
Research Summer school
July 4, 2018

2. Disclosures Grants/Research Support:
Relationships with commercial interests:
Grants/Research Support:
Canadian Institutes of Health Research
Brain and Behavior Research Foundation (NARSAD)
Young Investigator Award
Speakers Bureau/Honoraria: Nil
Consulting Fees: Nil
Other: Nil

3. Objectives Learn how to identify and avoid bias in critical appraisal
Appreciate the difference between good and bad studies
Learn key questions to ask when reviewing a scientific article
Understand how to develop and convey the rationale for your study
The importance of a comprehensive and targeted literature review

4. PICO(T)
A useful approach to developing your research question, guiding your literature review, or critically appraising a study.
P Patient, problem, or population
I Intervention
C Comparison, control, comparator
O Outcome
T Time duration

5. Ten Key Questions in Critical Appraisal
Is the study relevant?
Does the study add anything new?
What type of research question is being asked?
Is the study design appropriate for the research question?
Did the study methods address sources of bias?
Was the study performed according to original protocol?
Did it test a stated hypothesis?
Were statistical analyses performed correctly?
Do the data justify the conclusions?
Are there any conflicts of interest?
Young JM & Solomon MJ. Nat Clin Pract Gastroenterol Hepatol. 2009

6. Question #1 So what? Is the study important? What to look for
The best done study is irrelevant if it does not address an important question
What to look for
The introduction
Authors need to convince a reviewer that the topic/area is important to health, the public, etc.
For your study
Focus on current diseases of interest, new procedures, novel technologies

7. Question #2 Does it add anything new?
What to look for
The introduction
Literature review – how thorough is the review, how well is it summarized?
The reviewer should have a clear idea that this is not only something new but also not simply an incremental addition to previous knowledge
You may need to do a quick literature search
For your study
How many studies on your topic exist? What are their limitations? What questions have not yet been answered?

8. Is the Study Design Appropriate?
Relates to the research question
Efficacy of an intervention: RCT
Frequency of events/risk factors: observational studies

9. Hierarchy of Evidence Systematic reviews and meta-analyses of RCTs
Cohort studies
Case-control studies
Cross sectional surveys
Case reports

10. Do the Methods Address Bias?
When the results of a study deviate from the truth
When bias is due to the methodology of a study (vs. random chance) – this is called systematic bias

11. Pannucci CJ & Wilkins EG. Plast Reconstuct Surg. 2010

12. Addressing Bias: Example 1
Study: Examining rates of mental disorders in parents bereaved by MVC death of their child
Method: Parents surveyed in a bereavement support group

13. Addressing Bias: Example 1
Examining rates of mental disorders in parents bereaved by MVC death of their child
Method: Parents surveyed in a bereavement support group
Potential Bias: Selection Bias
Attendees in bereavement groups have been shown to be healthier than non-attendees
Solution: conduct a population-based study of bereaved parents using administrative data

14. Addressing Bias: Example 2
Study: Is cancer associated with suicide?
Design: Using CancerCare registries, examining all people in Manitoba who were newly diagnosed with cancer, and examining suicide rates within 1 year
Statistical analysis: unadjusted logistic regression models that examined the association with cancer diagnosis and suicide

15. Addressing Bias: Example 2
Potential Bias: Confounding
Mental disorder are associated with both the predictor variable (cancer) and outcome variable (suicide)
Solution: multivariate regression that adjusts for mental disorders

16. Do the Data Justify the Conclusions?
Meta-analysis of antidepressant trials based on clinical study reports
Main outcomes: mortality and suicidality
Results:
Mortality: Odds ratio 1.28; 95% CI
Suicidality: Odds ratio 1.21; 95% CI

17. Do the Data Justify the Conclusions?
Meta-analysis of antidepressant trials based on clinical study reports
Main outcomes: mortality and suicidality
Results:
Mortality: Odds ratio 1.28; 95% CI
Suicidality: Odds ratio 1.21; 95% CI
Conclusions:
“We suggest minimal use of antidepressants in children, adolescents, and young adults, as the serious harms seem to be greater.”
Sharma T et al. BMJ. 2016

18. CASP Critical Appraisal Skills Program
casp-uk.net
Has downloadable checklists that you can use for any type of study

19. Ten Key Questions in Critical Appraisal
Is the study relevant?
Does the study add anything new?
What type of research question is being asked?
Is the study design appropriate for the research question?
Did the study methods address sources of bias?
Was the study performed according to original protocol?
Did it test a stated hypothesis?
Were statistical analyses performed correctly?
Do the data justify the conclusions?
Are there any conflicts of interest?
Young JM & Solomon MJ. Nat Clin Pract Gastroenterol Hepatol. 2009

20. Suggestions Go toward the fear – don’t avoid!
Participate in journal clubs
Take opportunities to review for journals
Learn what top researchers are working on
See comments by other reviewers
If an option, participate in grant review committees
Remember: You don’t need to be a stats wizard!
The core of critical appraisal is based on the importance of the study and the approach to the question
Clinical knowledge goes a long way