HelpDesk Answers Evaluating the Evidence (Levels of Evidence) Corey Lyon, D.O. University of Colorado

Objectives Review the Evidence Pyramid and study design Introduction to CEBM’s level of evidence table Gain an understanding of how to apply a level of evidence to a research article Gain experience with assigning a level of evidence through exercises

Sifting The Evidence As you sift through the evidence articles that you have found, you will want to grade them based upon their quality so that you are using only the best studies in your HelpDesk Answer article. Levels of Evidence (LOEs) must be assigned to every article in your reference list (labeled as STEPs). This is helps both you and the editorial team, and provides added transparency to the reader. Assigning LOEs can be tricky work in some cases, even for the most experienced of authors and editors. Move through the steps that follow and discuss your assignments with your co-author and local editor when you have any uncertainty.

Begin with the Question Type Most HDAs deal with questions of diagnosis or therapy. Diagnosis: How to select and interpret diagnostic tests. Therapy: How to select treatments that do more good than harm and that are worth the efforts and costs involved. Prognosis: How to estimate the clinical course of the condition and anticipate likely related complications. Harm/Etiology: How to identify causes for disease (including iatrogenic forms.) Prevention: Differential diagnosis / symptom prevalence study. Cost: Economic and decision analyses. What is the most cost effective alternative.

Question Type Relates to Evidence Strength Type of Question Guide to the Relative Strength of Study Evidence Therapy RCT>cohort > case control > case series Diagnosis Prospective, blind comparison to a gold standard Etiology/Harm RCT > cohort > case control > case series Prognosis Cohort study > case control > case series Prevention RCT>cohort study > case control > case series Cost Economic analysis

Refer Back to the Pyramid, if in Doubt The Evidence Pyramid Meta-Analysis Systematic Reviews Randomized Control Trials Cohort Studies Case-Control Studies Case Series Expert Opinion

Evidence Pyramid Definitions Meta-analysis is a systematic review which uses quantitative methods to summarize the results. The analysis will thoroughly examine a number of valid studies on a topic and combine the results using accepted statistical methodology as if they were from one large study. Some clinicians put meta-analysis at the top of the pyramid because part of the methodology includes critical appraisal of the selected RCTs for analysis. Systematic reviews are articles in which the authors have systematically searched for, appraised, and summarized 'all of the medical literature' for a specific topic. Systematic reviews usually focus on a clinical topic and answer a specific question. An extensive literature search is conducted to identify all studies with sound methodology. The studies are reviewed, assessed, and the results summarized according to the predetermined criteria of the review question. The Cochrane Collaboration has done a lot of work in the area of systematic reviews. 7

Definitions Continued Randomized controlled clinical trials are carefully planned studies, testing the effect of a therapy on real patients. The group of patients is randomized into an experimental group and a control group. These groups are followed up for the variables/outcomes of interest. RCTs include methodologies that reduce the potential for bias (randomization and blinding) and that allow for comparison between intervention groups and control groups (no intervention). Cohort studies involve the identification of two groups (cohorts) of patients, one which received the exposure of interest, and one which did not, and following these cohorts forward over time for the outcome of interest. A cohort study involves a large population and follows these patients who have a specific condition or receive a particular treatment over time and compares them with another group that has not been affected by the condition or treatment being studied. Cohort studies are observational and not as reliable as randomized controlled studies, since the two groups may differ in ways other than in the variable under study. 8

Definitions Continued Case control studies are studies in which patients who already have a specific condition are compared with people who do not. They often rely on medical records and patient recall for data collection. These types of studies are often less reliable than randomized controlled trials and cohort studies because showing a statistical relationship does not mean than one factor necessarily caused the other. Case series report on a series of patients with an outcome of interest. No control group is involved. These studies consist of collections of reports on the treatment of individual patients or a report on a single patient. Because they are reports of cases and use no control groups with which to compare outcomes, they have no statistical validity. 9

Now Grade the Evidence Levels of Evidence (LOEs) are assigned to each reference using the CEBM tables. www.cebm.net The type of QUESTION directs you to the COLUMN. The type of STUDY guides you to the ROW and LOE. The CEBM table is online at www.cebm.net and in the handouts section of this module. CEBM TABLE - HANDOUT AND IN AUTHOR HANDBOOK NOTE THE TWO COLUMNS THAT ARE MOST FREQUENTLY USED IN HDAS.

Oxford Centre for Evidence-Based Medicine 2011 Levels of Evidence Question Step 1 (Level 1*) Step 2 (Level 2*) Step 3 (Level 3*) Step 4 (Level 4*) Step 5 (Level 5*) How common is the problem? Local and current random sample surveys (or censuses) Systematic review of surveys that allow matching to local circumstances** Local non-random sample** Case-series** n/a Is this diagnostic or monitoring test accurate? (Diagnosis) Systematic review of cross sectional studies with consistently applied reference standard and blinding Individual cross sectional studies with consistently applied reference standard and Blinding Non-consecutive studies, or studies without consistently applied reference standards** Case-control studies, or “poor” or non-independent reference standard** Mechanism-based reasoning What will happen if we do not add a therapy? (Prognosis) Systematic review of inception cohort studies Inception cohort studies Cohort study or control arm of randomized trial* Case-series or case-control studies, or poor quality prognostic cohort study** Does this intervention help? (Treatment Benefits) Systematic review of randomized trials or n-of-1 trials Randomized trial or observational study with dramatic effect Non-randomized controlled cohort/follow-up study** Case-series, case-control studies, or historically controlled studies** What are the COMMON harms? (Treatment Harms) Systematic review of randomized trials, systematic review of nested case-control studies, n-of-1 trial with the patient you are raising the question about, or observational study with dramatic effect Individual randomized trial or (exceptionally) observational study with dramatic effect Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)** Case-series, case-control, or historically controlled studies** What are the RARE harms? Systematic review of randomized trials or n-of-1 trial Randomized trial or (exceptionally) observational study with dramatic effect Is this (early detection) test worthwhile? (Screening) Systematic review of randomized trials Randomized trial Non -randomized controlled cohort/follow-up study** * Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size. ** As always, a systematic review is generally better than an individual study. How to cite the Levels of Evidence Table OCEBM Levels of Evidence Working Group*. "The Oxford 2011 Levels of Evidence". Oxford Centre for Evidence-Based Medicine. http://www.cebm.net/index.aspx?o=5653 * OCEBM Table of Evidence Working Group = Jeremy Howick, Iain Chalmers (James Lind Library), Paul Glasziou, Trish Greenhalgh, Carl Heneghan, Alessandro Liberati, Ivan Moschetti, Bob Phillips, Hazel Thornton, Olive Goddard and Mary Hodgkinson

Most Questions in HDAs deal with Therapy (Benefit or Harms) or Diagnosis Step 1 (Level 1*) Step 2 (Level 2*) Step 3 (Level 3*) Step 4 (Level 4*) Step 5 (Level 5*) Is this diagnostic or monitoring test accurate? (Diagnosis) Systematic review of cross sectional studies with consistently applied reference standard and blinding Individual cross sectional studies with consistently applied reference standard and Blinding Non-consecutive studies, or studies without consistently applied reference standards** Case-control studies, or “poor” or non-independent reference standard** Mechanism-based reasoning Does this intervention help? (Treatment Benefits) Systematic review of randomized trials or n-of-1 trials Randomized trial or observational study with dramatic effect Non-randomized controlled cohort/follow-up study** Case-series, case-control studies, or historically controlled studies** What are the RARE harms? (Treatment Harms) Systematic review of randomized trials or n-of-1 trial Randomized trial or (exceptionally) observational study with dramatic effect Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)** Case-series, case-control, or historically controlled studies** * Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size. ** As always, a systematic review is generally better than an individual study.

LOE Examples: Grading the References Read the abstracts that follow and assign each of them a level of evidence based on the CEBM tables that we have discussed. These abstracts are also available in the handout section of this module.

LOE Examples: Grading the References

Most Questions in HDAs deal with Therapy (Benefit or Harms) or Diagnosis Step 1 (Level 1*) Step 2 (Level 2*) Step 3 (Level 3*) Step 4 (Level 4*) Step 5 (Level 5*) Is this diagnostic or monitoring test accurate? (Diagnosis) Systematic review of cross sectional studies with consistently applied reference standard and blinding Individual cross sectional studies with consistently applied reference standard and Blinding Non-consecutive studies, or studies without consistently applied reference standards** Case-control studies, or “poor” or non-independent reference standard** Mechanism-based reasoning Does this intervention help? (Treatment Benefits) Systematic review of randomized trials or n-of-1 trials Randomized trial or observational study with dramatic effect Non-randomized controlled cohort/follow-up study** Case-series, case-control studies, or historically controlled studies** What are the RARE harms? (Treatment Harms) Systematic review of randomized trials or n-of-1 trial Randomized trial or (exceptionally) observational study with dramatic effect Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)** Case-series, case-control, or historically controlled studies** * Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size. ** As always, a systematic review is generally better than an individual study.

LOE Examples: Grading the References Step 1 Type of Question: Therapy Type of Study: Meta-analysis of RCT

LOE Examples: Grading the References

Most Questions in HDAs deal with Therapy (Benefit or Harms) or Diagnosis Step 1 (Level 1*) Step 2 (Level 2*) Step 3 (Level 3*) Step 4 (Level 4*) Step 5 (Level 5*) Is this diagnostic or monitoring test accurate? (Diagnosis) Systematic review of cross sectional studies with consistently applied reference standard and blinding Individual cross sectional studies with consistently applied reference standard and Blinding Non-consecutive studies, or studies without consistently applied reference standards** Case-control studies, or “poor” or non-independent reference standard** Mechanism-based reasoning Does this intervention help? (Treatment Benefits) Systematic review of randomized trials or n-of-1 trials Randomized trial or observational study with dramatic effect Non-randomized controlled cohort/follow-up study** Case-series, case-control studies, or historically controlled studies** What are the RARE harms? (Treatment Harms) Systematic review of randomized trials or n-of-1 trial Randomized trial or (exceptionally) observational study with dramatic effect Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)** Case-series, case-control, or historically controlled studies** * Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size. ** As always, a systematic review is generally better than an individual study.

LOE Examples: Grading the References Step 2 Type of Question: Therapy Type of Study: Therapy

20

Most Questions in HDAs deal with Therapy (Benefit or Harms) or Diagnosis Step 1 (Level 1*) Step 2 (Level 2*) Step 3 (Level 3*) Step 4 (Level 4*) Step 5 (Level 5*) Is this diagnostic or monitoring test accurate? (Diagnosis) Systematic review of cross sectional studies with consistently applied reference standard and blinding Individual cross sectional studies with consistently applied reference standard and Blinding Non-consecutive studies, or studies without consistently applied reference standards** Case-control studies, or “poor” or non-independent reference standard** Mechanism-based reasoning Does this intervention help? (Treatment Benefits) Systematic review of randomized trials or n-of-1 trials Randomized trial or observational study with dramatic effect Non-randomized controlled cohort/follow-up study** Case-series, case-control studies, or historically controlled studies** What are the RARE harms? (Treatment Harms) Systematic review of randomized trials or n-of-1 trial Randomized trial or (exceptionally) observational study with dramatic effect Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)** Case-series, case-control, or historically controlled studies** * Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size. ** As always, a systematic review is generally better than an individual study.

Step 3 Type of Question: Therapy Type of Study: Cohort 22

Most Questions in HDAs deal with Therapy (Benefit or Harms) or Diagnosis Step 1 (Level 1*) Step 2 (Level 2*) Step 3 (Level 3*) Step 4 (Level 4*) Step 5 (Level 5*) Is this diagnostic or monitoring test accurate? (Diagnosis) Systematic review of cross sectional studies with consistently applied reference standard and blinding Individual cross sectional studies with consistently applied reference standard and Blinding Non-consecutive studies, or studies without consistently applied reference standards** Case-control studies, or “poor” or non-independent reference standard** Mechanism-based reasoning Does this intervention help? (Treatment Benefits) Systematic review of randomized trials or n-of-1 trials Randomized trial or observational study with dramatic effect Non-randomized controlled cohort/follow-up study** Case-series, case-control studies, or historically controlled studies** What are the RARE harms? (Treatment Harms) Systematic review of randomized trials or n-of-1 trial Randomized trial or (exceptionally) observational study with dramatic effect Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)** Case-series, case-control, or historically controlled studies** * Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size. ** As always, a systematic review is generally better than an individual study.

Step 2 Type of Question: Diagnosis Type of Study: Individual cross sectional studies with consistently applied reference standard and Blinding

26

Most Questions in HDAs deal with Therapy (Benefit or Harms) or Diagnosis Step 1 (Level 1*) Step 2 (Level 2*) Step 3 (Level 3*) Step 4 (Level 4*) Step 5 (Level 5*) Is this diagnostic or monitoring test accurate? (Diagnosis) Systematic review of cross sectional studies with consistently applied reference standard and blinding Individual cross sectional studies with consistently applied reference standard and Blinding Non-consecutive studies, or studies without consistently applied reference standards** Case-control studies, or “poor” or non-independent reference standard** Mechanism-based reasoning Does this intervention help? (Treatment Benefits) Systematic review of randomized trials or n-of-1 trials Randomized trial or observational study with dramatic effect Non-randomized controlled cohort/follow-up study** Case-series, case-control studies, or historically controlled studies** What are the RARE harms? (Treatment Harms) Systematic review of randomized trials or n-of-1 trial Randomized trial or (exceptionally) observational study with dramatic effect Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)** Case-series, case-control, or historically controlled studies** * Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size. ** As always, a systematic review is generally better than an individual study.

Step 2 Type of Question: Therapy Type of Study: RCT 28

Additional Resources Remember that these ratings are not always clear. Refer to the Evidence Pyramid for guidance. Download the CEBM Evidence Tables from this module’s homepage for detailed explanations. Additional practice is available in the module quiz. When in doubt, ask your co-author and editor for assistance.