How to Read an Article about a Systematic Review Maribeth Chitkara, MD Rachel Boykan, MD Stony Brook Long Island Children’s Hospital

Session Objectives Demonstrate understanding of general EBM principles. Demonstrate understanding of validity measures specific to articles about a systematic review. Critically appraise an article about a systematic review and describe its clinical applicability.

Clinical Scenario You are on inpatient service and have been taking care of a 4 month old with bronchiolitis. The infant was initially admitted to the ICU for respiratory distress, but was transferred to the general pediatric ward on the 4 th hospital day with resolved increased work of breathing but a persistent O2 requirement. Your attending suggests starting the child on systemic corticosteroids because she is getting inpatient with the prolonged length of stay. You are skeptical, and decide to see where the medical evidence lies. You are on inpatient service and have been taking care of a 4 month old with bronchiolitis. The infant was initially admitted to the ICU for respiratory distress, but was transferred to the general pediatric ward on the 4 th hospital day with resolved increased work of breathing but a persistent O2 requirement. Your attending suggests starting the child on systemic corticosteroids because she is getting inpatient with the prolonged length of stay. You are skeptical, and decide to see where the medical evidence lies.

Systematic Reviews A summary of the medical literature A summary of the medical literature Uses explicit methods to systematically search, critically appraise, and synthesize the world literature Uses explicit methods to systematically search, critically appraise, and synthesize the world literature

Systematic Reviews Goals: Minimize Bias Minimize Bias randomized trials randomized trials seeks multilingual published and unpublished reports seeks multilingual published and unpublished reports Minimize random error Minimize random error by amassing very large numbers by amassing very large numbers

Systematic Reviews May include some statistical method for combining the results of individual studies May include some statistical method for combining the results of individual studies“Meta-analyses”

Appraisal of Systematic Reviews Is the Systematic Review Valid? Is the Systematic Review Valid? What are the Results? What are the Results? Can I Apply the results to my patient? Can I Apply the results to my patient?

Is the Systematic Review Valid? Is this a systematic review of randomized trials? Is this a systematic review of randomized trials? Were the results consistent from study to study? Were the results consistent from study to study?

Is the Systematic Review Valid? Randomized trials reduce bias Randomized trials reduce bias Combining all relevant randomized trials further reduces bias and random error Combining all relevant randomized trials further reduces bias and random error Provides the highest level of evidence currently achievable Provides the highest level of evidence currently achievable

Is the Systematic Review Valid? Systematic Reviews of Non-Randomized Trials Can: Systematic Reviews of Non-Randomized Trials Can:  Compound the problems of individually misleading trials  Produce a lower quality of evidence

Is the Systematic Review Valid? If a systematic review includes both Randomized and Non-Randomized Trials, avoid it unless these types of trials are separated in the analyses. If a systematic review includes both Randomized and Non-Randomized Trials, avoid it unless these types of trials are separated in the analyses.

Is the Systematic Review Valid? Does the study have a Methods Section Does the study have a Methods Section Seek a description of how the investigators found all the relevant trials Seek a description of how the investigators found all the relevant trials If not present  STOP If not present  STOP

The Methods Section Seek reassurance that the search went beyond standard bibliographic databases Seek reassurance that the search went beyond standard bibliographic databases Should include hand-searching journals, conference proceedings, theses, databanks of pharmaceutical companies, and contacting authors of published articles Should include hand-searching journals, conference proceedings, theses, databanks of pharmaceutical companies, and contacting authors of published articles

The Methods Section Negative trials are less likely to be submitted and selected for publication Negative trials are less likely to be submitted and selected for publication A Systematic Review restricted to published trials can result in a false- positive conclusion A Systematic Review restricted to published trials can result in a false- positive conclusion Restriction to a single language can bias the conclusions Restriction to a single language can bias the conclusions

Is the Systematic Review Valid? How did the investigators assess the validity of the individual studies? How did the investigators assess the validity of the individual studies? Randomized? Randomized? Follow-Up sufficient? Follow-Up sufficient? Intention to treat analysis? Intention to treat analysis? Blinded? Blinded? Were the groups similar at the start? Were the groups similar at the start? Were the groups treated equally, apart from the experimental tx? Were the groups treated equally, apart from the experimental tx?

What are the Results? Forest Plots Each study is a line Each study is a line Black square is point estimate Black square is point estimate Size of square is weight Size of square is weight Line is 95% confidence interval Line is 95% confidence interval Pooled analysis is combined effect and CI Pooled analysis is combined effect and CI

What are the Results? Funnel Plots and Publication Bias Vertical Axis Measure of study precision Higher up: more precise, smaller CI Horizontal Axis Measures treatment effect Center Line Combined treatment effect No publication bias: Inverted funnel shape

Can I Apply the results to my patient? Is our patient so different from those in the study that the results cannot apply? Is our patient so different from those in the study that the results cannot apply? Is the treatment feasible in our setting? Is the treatment feasible in our setting? What are our patient’s potential benefits and harms from the therapy? What are our patient’s potential benefits and harms from the therapy? What are our patient’s values and preferences for both the outcome we desire and the potential side-effects of treatment? What are our patient’s values and preferences for both the outcome we desire and the potential side-effects of treatment?

Can I Apply the results to my patient? A Systematic Review provides an overall, average effect of therapy which may be derived from a heterogeneous population. A Systematic Review provides an overall, average effect of therapy which may be derived from a heterogeneous population. SR’s may provide precise information on sub-groups that may help individualize evidence to our own patients. SR’s may provide precise information on sub-groups that may help individualize evidence to our own patients. BUT... BUT...

Can I Apply the results to my patient? A qualitative difference in treatment efficacy among subgroups is likely only when ALL the following questions can be answered “YES”: Does it really make biological and clinical sense? Does it really make biological and clinical sense? Is the qualitative difference both clinically and statistically significant? Is the qualitative difference both clinically and statistically significant? Was it hypothesized before the study began (rather than a product of dredging the data) and has it been confirmed in other independent studies? Was it hypothesized before the study began (rather than a product of dredging the data) and has it been confirmed in other independent studies? Was it one of just a few subgroup analyses carried out in this study? Was it one of just a few subgroup analyses carried out in this study?

Practice Cases Review the case scenarios in your handout Review the case scenarios in your handout Articles for each clinical question have been provided Articles for each clinical question have been provided Use worksheet to help guide your appraisal for each article Use worksheet to help guide your appraisal for each article

Practice Case References Case 1: Case 1: Garrison MM. et al. Systemic Corticosteroids in Infant Bronchiolitis: A Meta-analysis. Pediatrics, 2000; 105:e44. Case 2: Case 2: Wooditch AC, Aronoff SC. Effect of Initial Corticosteroid Therapy on Coronary Artery Aneurysm Formation in Kawasaki Disease: A Meta-analysis of 862 Children. Pediatrics, 2005;116:989. Case 3: Case 3: Figuls MR, Giné-Garriga M, Granados Rugeles C, Perrotta C. Chest physiotherapy for acute bronchiolitis in paediatric patients between 0 and 24 months old. Cochrane Database of Systematic Reviews 2012.

EBM References/Resources Friedland DJ, Shilpak MG, Subak LL, Bent SW, Mendelson T. Evaluating Integrative Literature. Evidence-based medicine: a framework for clinical practice, Appleton and Lange, 1998: Friedland DJ, Shilpak MG, Subak LL, Bent SW, Mendelson T. Evaluating Integrative Literature. Evidence-based medicine: a framework for clinical practice, Appleton and Lange, 1998: Straus SE, Glasziou P, Richardson WS, Haynes RB. Therapy: Reports of Systematic Reviews. Evidence-based medicine: how to practice and teach it, 4 th ed, Churchill Livingston Elsevier, 2011: