Critical Appraisal & Literature review James M. Bolton MD Professor of Psychiatry, Psychology, Community Health Sciences, U of M Adjunct Scientist, Manitoba Centre for Health Policy Medical Director, Crisis Response Centre & HSC Emergency Psychiatry, WRHA Director of Research, department of psychiatry Research Summer school July 4, 2018

Disclosures Grants/Research Support: Relationships with commercial interests: Grants/Research Support: Canadian Institutes of Health Research Brain and Behavior Research Foundation (NARSAD) Young Investigator Award Speakers Bureau/Honoraria: Nil Consulting Fees: Nil Other: Nil

Objectives Learn how to identify and avoid bias in critical appraisal Appreciate the difference between good and bad studies Learn key questions to ask when reviewing a scientific article Understand how to develop and convey the rationale for your study The importance of a comprehensive and targeted literature review

PICO(T) A useful approach to developing your research question, guiding your literature review, or critically appraising a study. P Patient, problem, or population I Intervention C Comparison, control, comparator O Outcome T Time duration

Ten Key Questions in Critical Appraisal Is the study relevant? Does the study add anything new? What type of research question is being asked? Is the study design appropriate for the research question? Did the study methods address sources of bias? Was the study performed according to original protocol? Did it test a stated hypothesis? Were statistical analyses performed correctly? Do the data justify the conclusions? Are there any conflicts of interest? Young JM & Solomon MJ. Nat Clin Pract Gastroenterol Hepatol. 2009

Question #1 So what? Is the study important? What to look for The best done study is irrelevant if it does not address an important question What to look for The introduction Authors need to convince a reviewer that the topic/area is important to health, the public, etc. For your study Focus on current diseases of interest, new procedures, novel technologies

Question #2 Does it add anything new? What to look for The introduction Literature review – how thorough is the review, how well is it summarized? The reviewer should have a clear idea that this is not only something new but also not simply an incremental addition to previous knowledge You may need to do a quick literature search For your study How many studies on your topic exist? What are their limitations? What questions have not yet been answered?

Is the Study Design Appropriate? Relates to the research question Efficacy of an intervention: RCT Frequency of events/risk factors: observational studies

Hierarchy of Evidence Systematic reviews and meta-analyses of RCTs Cohort studies Case-control studies Cross sectional surveys Case reports

Do the Methods Address Bias? When the results of a study deviate from the truth When bias is due to the methodology of a study (vs. random chance) – this is called systematic bias

Pannucci CJ & Wilkins EG. Plast Reconstuct Surg. 2010

Addressing Bias: Example 1 Study: Examining rates of mental disorders in parents bereaved by MVC death of their child Method: Parents surveyed in a bereavement support group

Addressing Bias: Example 1 Examining rates of mental disorders in parents bereaved by MVC death of their child Method: Parents surveyed in a bereavement support group Potential Bias: Selection Bias Attendees in bereavement groups have been shown to be healthier than non-attendees Solution: conduct a population-based study of bereaved parents using administrative data

Addressing Bias: Example 2 Study: Is cancer associated with suicide? Design: Using CancerCare registries, examining all people in Manitoba who were newly diagnosed with cancer, and examining suicide rates within 1 year Statistical analysis: unadjusted logistic regression models that examined the association with cancer diagnosis and suicide

Addressing Bias: Example 2 Potential Bias: Confounding Mental disorder are associated with both the predictor variable (cancer) and outcome variable (suicide) Solution: multivariate regression that adjusts for mental disorders

Do the Data Justify the Conclusions? Meta-analysis of antidepressant trials based on clinical study reports Main outcomes: mortality and suicidality Results: Mortality: Odds ratio 1.28; 95% CI 0.40-4.06 Suicidality: Odds ratio 1.21; 95% CI 0.84-1.74

Do the Data Justify the Conclusions? Meta-analysis of antidepressant trials based on clinical study reports Main outcomes: mortality and suicidality Results: Mortality: Odds ratio 1.28; 95% CI 0.40-4.06 Suicidality: Odds ratio 1.21; 95% CI 0.84-1.74 Conclusions: “We suggest minimal use of antidepressants in children, adolescents, and young adults, as the serious harms seem to be greater.” Sharma T et al. BMJ. 2016

CASP Critical Appraisal Skills Program casp-uk.net Has downloadable checklists that you can use for any type of study

Ten Key Questions in Critical Appraisal Is the study relevant? Does the study add anything new? What type of research question is being asked? Is the study design appropriate for the research question? Did the study methods address sources of bias? Was the study performed according to original protocol? Did it test a stated hypothesis? Were statistical analyses performed correctly? Do the data justify the conclusions? Are there any conflicts of interest? Young JM & Solomon MJ. Nat Clin Pract Gastroenterol Hepatol. 2009

Suggestions Go toward the fear – don’t avoid! Participate in journal clubs Take opportunities to review for journals Learn what top researchers are working on See comments by other reviewers If an option, participate in grant review committees Remember: You don’t need to be a stats wizard! The core of critical appraisal is based on the importance of the study and the approach to the question Clinical knowledge goes a long way