Nut and Bolts of Critical Appraisal of Medical Literature Focus on Diagnosis Nick Myles, MD Pathologist St.Paul’s Hospital

Objectives Learn why research evidence on diagnosis should be critically appraised Learn Evidence-Based Medicine tools for critical appraisal of medical literature on diagnosis

Objectives Learn why research evidence on diagnosis should be critically appraised Learn Evidence-Based Medicine tools for critical appraisal of medical literature on diagnosis

Why Critical Appraisal and not just use any published book/paper in practice? Your thoughts?

Examples from Clinical Medicine Dr Spock: recommends Prone against Supine Position Crib deaths best evidence: risk of crib death is much higher in Prone position

Examples from Clinical Medicine Some anti- arrhythmia drugs is MI: Pathophysiologic mechamism: prevent arrhythmia to prevent death RCT evidence: some anti-arrhythmics increase mortality

Examples from Clinical Medicine Pathophysiology: Vitamins A and E are beneficient antooxidants, so use widely RCT: use of vitamins A and E increase mortality

Example from Lab Medicine Some pathology experts claim 0% error and 0% tolerance to error EMB claims there are no tests which are 100% accurate

Examples from Lab Medicine Some experts claim 0% error tolerance EMB claims there are no tests which are 100% accurate

What to expect from the test? If I test breast cancer for hormone receptors and get negative result, how can I be certain that it’s indeed negative? What do you need to know? Your thoughts

What to expect from the test? If my wife gets her mammogram and it is negative, how can I be certain that it’s indeed negative? What do you need to know? Your thoughts

Formulate answerable question: PICO PIRTO Population Population Intervention Index test Comparator Reference test Outcome Time Outcome for Interventions for Diagnosis

Grades of Diagnostic Evidence Grade 1: Systematic Reviews of good cohorts with reference; absolute Sp or Sn Grade 2: SRs of cohorts with split sample validation; with reference test Grade 3: SRs of non-consequitive cohorts with inconsistent reference tests Grade 4: case controls, poor, non-independent inconsistent reference Grade 5: EXPERT OPINION

consistent level 1 studies B Grades of Recommendation A consistent level 1 studies B consistent level 2 or 3 studies or extrapolations from level 1 studies C level 4 studies or extrapolations from level 2 or 3 studies D level 5 evidence or troublingly inconsistent or inconclusive studies of any level

Critical Appraisal Steps Formulate answerable question: PICO or PIRTO Search for the best already APPRAISED evidence (not only PubMed!) Check: NICE guidance (National Institute for Clinical Excellence) http://www.nice.org.uk/ Cochrane library of systematic reviews http://www.thecochranelibrary.com If still no answer, then consider doing the following:

Critical Appraisal Steps Formulate answerable question: PICO or PIRTO Search for the best evidence (not only PubMed!) Extract and Appraise the best evidence Apply

Evidence Search Pubmed vs others or several engines WARNING If single engine used, 30-40% studies will not show up with the same key words

Select the Best Evidence Hierarchy of Evidence: WHY ? Your thoughts?

When finding evidence one has to go from top And only then down Why? Your thoughts?

Objectives Learn why research evidence on diagnosis should be critically appraised Learn Evidence-Based Medicine tools for critical appraisal of medical literature on diagnosis

Objectives Learn why research evidence on diagnosis should be critically appraised Learn Evidence-Based Medicine tools for critical appraisal of medical literature on diagnosis

EBM Validated Tools Easy and proven, yet comprehensive: YOU Can Do It

EBM Tools Checklists: STARD and QUADRAS Online Calculators: CEBM and U of T, +others Software: CAT Maker All open access, free

EBM Tools Checklists: STARD (25 items) Standards of Reporting Diagnostic Accuracy http://www.stard-statement.org/ Adapted by 200+ biomedical journals (as per 2009), with only one pathology journal listed to allow readers to assess the potential for bias in the study (internal validity) and to evaluate its generalisability (external validity).

EBM Tools Checklists: QUADRAS a tool for the quality assessment of diagnostic accuracy studies http://annals.org/article.aspx?articleid=474994 risk of bias assessment when extracting data from multiple studies for systematic review

EBM Tools Online Calculators: CEBM www.cebm.net Plenty of support, free access Critical appraisal checklists for all types of clinical medical literature Example: Diagnosis (next slide)

www.cebm.net

www.cebm.net

www.cebm.net

EBM Tools Online Calculators: U of Toronto http://ktclearinghouse.ca/cebm/toolbo Easy, excellent graphic output, concise Examples: diagnosis, RCTs, Cohorts, NNT etc Perfect for data simulation Example: diagnosis next page

EBM Tools Software: CAT Maker All-in-one solution Allows you to generate one page Critical Appraisal Topic Summary with all numeric output May crush..

ER-positive test ER negative test Prev 50%

Prev 80%

Simulation (if we are online) Task 1a What’s the diagnostic accuracy of mammography test (90% sensitive and 95% specific) in 40 y.o. women? So, how many errors (false +, false -) do you expect after 1000 mammograms? Overall, about 1% of women in this age group have breast cancer Need one minute?

Simulation (if we are online) Task 1b What’s the diagnostic accuracy of mammography test in 70 y.o. women? (assuming 90% sensitive and 95% specific) How many errors (false +, false -) do you expect after 1000 mammograms? Overall, 15% of women in this age group have breast cancer. Need one minute? Your answer: FP ? FN?

Simulation (if we are online) Task 1a prevalence: 1% (40 y.o) Predicted errors: False positive 50/59 total positive False negative 1/940

Task 1a

Simulation (if we are online) Task 1b prevalence: 1% (40 y.o.) Predicted errors: False positive 43/183 total test positive False negative 15/823 total test neg

NL inquiry into Estrogene Receptor Lab Accuracy Done in 1998-2006

Task 2 Adequate (ASCO2010) Breast cancer Estrogen test is at least 90% sensitive and 90% specific. NL re-tested 1200+ cases in Ontario and found 400+ false negative results. The case is settled for 17M in 2009. What is the expected False Neg error rate of adequate ER test?

Task 2 answer Conclusion: under condition of 80% prevalence of ER positive breast cancers, expected NPV of the adequate test is 64-69%

Objectives Learn why research evidence on diagnosis should be critically appraised Learn Evidence-Based Medicine tools for critical appraisal of medical literature on diagnosis

Summary Critical Appraisal is necessary to assess validity of the studies before they are applied to clinical practice EBM tools are easy to use and allow for an independent analysis of the published data and claimed results.

Thank you