1. HelpDesk Answers Evaluating the Evidence (Levels of Evidence)
Corey Lyon, D.O.
University of Colorado

2. Objectives Review the Evidence Pyramid and study design
Introduction to CEBM’s level of evidence table
Gain an understanding of how to apply a level of evidence to a research article
Gain experience with assigning a level of evidence through exercises

3. Sifting The Evidence
As you sift through the evidence articles that you have found, you will want to grade them based upon their quality so that you are using only the best studies in your HelpDesk Answer article.
Levels of Evidence (LOEs) must be assigned to every article in your reference list (labeled as STEPs). This is helps both you and the editorial team, and provides added transparency to the reader.
Assigning LOEs can be tricky work in some cases, even for the most experienced of authors and editors. Move through the steps that follow and discuss your assignments with your co-author and local editor when you have any uncertainty.

4. Begin with the Question Type
Most HDAs deal with questions of diagnosis or therapy.
Diagnosis:
How to select and interpret diagnostic tests.
Therapy:
How to select treatments that do more good than harm and that are worth the efforts and costs involved.
Prognosis:
How to estimate the clinical course of the condition and anticipate likely related complications.
Harm/Etiology:
How to identify causes for disease (including iatrogenic forms.)
Prevention:
Differential diagnosis / symptom prevalence study.
Cost:
Economic and decision analyses. What is the most cost effective alternative.

5. Question Type Relates to Evidence Strength
Type of Question
Guide to the Relative
Strength of Study Evidence
Therapy
RCT>cohort > case control > case series
Diagnosis
Prospective, blind comparison to a gold standard
Etiology/Harm
RCT > cohort > case control > case series
Prognosis
Cohort study > case control > case series
Prevention
RCT>cohort study > case control > case series
Cost
Economic analysis

6. Refer Back to the Pyramid, if in Doubt
The Evidence Pyramid
Meta-Analysis
Systematic Reviews
Randomized Control Trials
Cohort Studies
Case-Control Studies
Case Series
Expert Opinion

7. Evidence Pyramid Definitions
Meta-analysis is a systematic review which uses quantitative methods to summarize the results. The analysis will thoroughly examine a number of valid studies on a topic and combine the results using accepted statistical methodology as if they were from one large study. Some clinicians put meta-analysis at the top of the pyramid because part of the methodology includes critical appraisal of the selected RCTs for analysis.
Systematic reviews are articles in which the authors have systematically searched for, appraised, and summarized 'all of the medical literature' for a specific topic. Systematic reviews usually focus on a clinical topic and answer a specific question. An extensive literature search is conducted to identify all studies with sound methodology. The studies are reviewed, assessed, and the results summarized according to the predetermined criteria of the review question. The Cochrane Collaboration has done a lot of work in the area of systematic reviews. 7

8. Definitions Continued
Randomized controlled clinical trials are carefully planned studies, testing the effect of a therapy on real patients. The group of patients is randomized into an experimental group and a control group. These groups are followed up for the variables/outcomes of interest. RCTs include methodologies that reduce the potential for bias (randomization and blinding) and that allow for comparison between intervention groups and control groups (no intervention).
Cohort studies involve the identification of two groups (cohorts) of patients, one which received the exposure of interest, and one which did not, and following these cohorts forward over time for the outcome of interest. A cohort study involves a large population and follows these patients who have a specific condition or receive a particular treatment over time and compares them with another group that has not been affected by the condition or treatment being studied. Cohort studies are observational and not as reliable as randomized controlled studies, since the two groups may differ in ways other than in the variable under study. 8

9. Definitions Continued
Case control studies are studies in which patients who already have a specific condition are compared with people who do not. They often rely on medical records and patient recall for data collection. These types of studies are often less reliable than randomized controlled trials and cohort studies because showing a statistical relationship does not mean than one factor necessarily caused the other.
Case series report on a series of patients with an outcome of interest. No control group is involved. These studies consist of collections of reports on the treatment of individual patients or a report on a single patient. Because they are reports of cases and use no control groups with which to compare outcomes, they have no statistical validity. 9

10. Now Grade the Evidence
Levels of Evidence (LOEs) are assigned to each reference using the CEBM tables.
The type of QUESTION directs you to the COLUMN.
The type of STUDY guides you to the ROW and LOE.
The CEBM table is online at and in the handouts section of this module.
CEBM TABLE - HANDOUT AND IN AUTHOR HANDBOOK
NOTE THE TWO COLUMNS THAT ARE MOST FREQUENTLY USED IN HDAS.

11. Oxford Centre for Evidence-Based Medicine 2011 Levels of Evidence
Question
Step 1 (Level 1*)
Step 2 (Level 2*)
Step 3 (Level 3*)
Step 4 (Level 4*)
Step 5 (Level 5*)
How common is the problem?
Local and current random sample surveys (or censuses)
Systematic review of surveys that allow matching to local circumstances**
Local non-random sample**
Case-series**
n/a
Is this diagnostic or monitoring test accurate?
(Diagnosis)
Systematic review of cross sectional studies with consistently applied reference standard and blinding
Individual cross sectional studies with consistently applied reference standard and Blinding
Non-consecutive studies, or studies without consistently applied reference standards**
Case-control studies, or “poor” or non-independent reference standard**
Mechanism-based reasoning
What will happen if we do not add a therapy?
(Prognosis)
Systematic review of inception cohort studies
Inception cohort studies
Cohort study or control arm of randomized trial*
Case-series or case-control studies, or poor quality prognostic cohort study**
Does this intervention help?
(Treatment Benefits)
Systematic review of randomized trials or n-of-1 trials
Randomized trial or observational study with dramatic effect
Non-randomized controlled cohort/follow-up study**
Case-series, case-control studies, or historically controlled studies**
What are the COMMON harms?
(Treatment Harms)
Systematic review of randomized trials, systematic review of nested case-control studies, n-of-1 trial with the patient you are raising the question about, or observational study with dramatic effect
Individual randomized trial or (exceptionally) observational study with dramatic effect
Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)**
Case-series, case-control, or historically controlled studies**
What are the RARE harms?
Systematic review of randomized trials or n-of-1 trial
Randomized trial or (exceptionally) observational study with dramatic effect
Is this (early detection) test worthwhile?
(Screening)
Systematic review of randomized trials
Randomized trial
Non -randomized controlled cohort/follow-up study**
* Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between
studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size.
** As always, a systematic review is generally better than an individual study.
How to cite the Levels of Evidence Table
OCEBM Levels of Evidence Working Group*. "The Oxford 2011 Levels of Evidence".
Oxford Centre for Evidence-Based Medicine.
* OCEBM Table of Evidence Working Group = Jeremy Howick, Iain Chalmers (James Lind Library), Paul Glasziou, Trish Greenhalgh, Carl Heneghan, Alessandro Liberati, Ivan Moschetti,
Bob Phillips, Hazel Thornton, Olive Goddard and Mary Hodgkinson

12. Most Questions in HDAs deal with Therapy (Benefit or Harms) or Diagnosis
Step 1 (Level 1*)
Step 2 (Level 2*)
Step 3 (Level 3*)
Step 4 (Level 4*)
Step 5 (Level 5*)
Is this diagnostic or monitoring test accurate?
(Diagnosis)
Systematic review of cross sectional studies with consistently applied reference standard and blinding
Individual cross sectional studies with consistently applied reference standard and Blinding
Non-consecutive studies, or studies without consistently applied reference standards**
Case-control studies, or “poor” or non-independent reference standard**
Mechanism-based reasoning
Does this intervention help?
(Treatment Benefits)
Systematic review of randomized trials or n-of-1 trials
Randomized trial or observational study with dramatic effect
Non-randomized controlled cohort/follow-up study**
Case-series, case-control studies, or historically controlled studies**
What are the RARE harms?
(Treatment Harms)
Systematic review of randomized trials or n-of-1 trial
Randomized trial or (exceptionally) observational study with dramatic effect
Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)**
Case-series, case-control, or historically controlled studies**
* Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between
studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size.
** As always, a systematic review is generally better than an individual study.

13. LOE Examples: Grading the References
Read the abstracts that follow and assign each of them a level of evidence based on the CEBM tables that we have discussed.
These abstracts are also available in the handout section of this module.

14. LOE Examples: Grading the References

15. Most Questions in HDAs deal with Therapy (Benefit or Harms) or Diagnosis
Step 1 (Level 1*)
Step 2 (Level 2*)
Step 3 (Level 3*)
Step 4 (Level 4*)
Step 5 (Level 5*)
Is this diagnostic or monitoring test accurate?
(Diagnosis)
Systematic review of cross sectional studies with consistently applied reference standard and blinding
Individual cross sectional studies with consistently applied reference standard and Blinding
Non-consecutive studies, or studies without consistently applied reference standards**
Case-control studies, or “poor” or non-independent reference standard**
Mechanism-based reasoning
Does this intervention help?
(Treatment Benefits)
Systematic review of randomized trials or n-of-1 trials
Randomized trial or observational study with dramatic effect
Non-randomized controlled cohort/follow-up study**
Case-series, case-control studies, or historically controlled studies**
What are the RARE harms?
(Treatment Harms)
Systematic review of randomized trials or n-of-1 trial
Randomized trial or (exceptionally) observational study with dramatic effect
Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)**
Case-series, case-control, or historically controlled studies**
* Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between
studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size.
** As always, a systematic review is generally better than an individual study.

16. LOE Examples: Grading the References
Step 1
Type of Question: Therapy
Type of Study: Meta-analysis of RCT

17. LOE Examples: Grading the References

18. Most Questions in HDAs deal with Therapy (Benefit or Harms) or Diagnosis
Step 1 (Level 1*)
Step 2 (Level 2*)
Step 3 (Level 3*)
Step 4 (Level 4*)
Step 5 (Level 5*)
Is this diagnostic or monitoring test accurate?
(Diagnosis)
Systematic review of cross sectional studies with consistently applied reference standard and blinding
Individual cross sectional studies with consistently applied reference standard and Blinding
Non-consecutive studies, or studies without consistently applied reference standards**
Case-control studies, or “poor” or non-independent reference standard**
Mechanism-based reasoning
Does this intervention help?
(Treatment Benefits)
Systematic review of randomized trials or n-of-1 trials
Randomized trial or observational study with dramatic effect
Non-randomized controlled cohort/follow-up study**
Case-series, case-control studies, or historically controlled studies**
What are the RARE harms?
(Treatment Harms)
Systematic review of randomized trials or n-of-1 trial
Randomized trial or (exceptionally) observational study with dramatic effect
Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)**
Case-series, case-control, or historically controlled studies**
* Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between
studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size.
** As always, a systematic review is generally better than an individual study.

19. LOE Examples: Grading the References
Step 2
Type of Question: Therapy
Type of Study: Therapy

20. 20

21. Most Questions in HDAs deal with Therapy (Benefit or Harms) or Diagnosis
Step 1 (Level 1*)
Step 2 (Level 2*)
Step 3 (Level 3*)
Step 4 (Level 4*)
Step 5 (Level 5*)
Is this diagnostic or monitoring test accurate?
(Diagnosis)
Systematic review of cross sectional studies with consistently applied reference standard and blinding
Individual cross sectional studies with consistently applied reference standard and Blinding
Non-consecutive studies, or studies without consistently applied reference standards**
Case-control studies, or “poor” or non-independent reference standard**
Mechanism-based reasoning
Does this intervention help?
(Treatment Benefits)
Systematic review of randomized trials or n-of-1 trials
Randomized trial or observational study with dramatic effect
Non-randomized controlled cohort/follow-up study**
Case-series, case-control studies, or historically controlled studies**
What are the RARE harms?
(Treatment Harms)
Systematic review of randomized trials or n-of-1 trial
Randomized trial or (exceptionally) observational study with dramatic effect
Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)**
Case-series, case-control, or historically controlled studies**
* Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between
studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size.
** As always, a systematic review is generally better than an individual study.

22. Step 3
Type of Question: Therapy
Type of Study: Cohort 22

24. Most Questions in HDAs deal with Therapy (Benefit or Harms) or Diagnosis
Step 1 (Level 1*)
Step 2 (Level 2*)
Step 3 (Level 3*)
Step 4 (Level 4*)
Step 5 (Level 5*)
Is this diagnostic or monitoring test accurate?
(Diagnosis)
Systematic review of cross sectional studies with consistently applied reference standard and blinding
Individual cross sectional studies with consistently applied reference standard and Blinding
Non-consecutive studies, or studies without consistently applied reference standards**
Case-control studies, or “poor” or non-independent reference standard**
Mechanism-based reasoning
Does this intervention help?
(Treatment Benefits)
Systematic review of randomized trials or n-of-1 trials
Randomized trial or observational study with dramatic effect
Non-randomized controlled cohort/follow-up study**
Case-series, case-control studies, or historically controlled studies**
What are the RARE harms?
(Treatment Harms)
Systematic review of randomized trials or n-of-1 trial
Randomized trial or (exceptionally) observational study with dramatic effect
Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)**
Case-series, case-control, or historically controlled studies**
* Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between
studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size.
** As always, a systematic review is generally better than an individual study.

25. Step 2 Type of Question: Diagnosis
Type of Study: Individual cross sectional studies with consistently applied reference standard and Blinding

26. 26

27. Most Questions in HDAs deal with Therapy (Benefit or Harms) or Diagnosis
Step 1 (Level 1*)
Step 2 (Level 2*)
Step 3 (Level 3*)
Step 4 (Level 4*)
Step 5 (Level 5*)
Is this diagnostic or monitoring test accurate?
(Diagnosis)
Systematic review of cross sectional studies with consistently applied reference standard and blinding
Individual cross sectional studies with consistently applied reference standard and Blinding
Non-consecutive studies, or studies without consistently applied reference standards**
Case-control studies, or “poor” or non-independent reference standard**
Mechanism-based reasoning
Does this intervention help?
(Treatment Benefits)
Systematic review of randomized trials or n-of-1 trials
Randomized trial or observational study with dramatic effect
Non-randomized controlled cohort/follow-up study**
Case-series, case-control studies, or historically controlled studies**
What are the RARE harms?
(Treatment Harms)
Systematic review of randomized trials or n-of-1 trial
Randomized trial or (exceptionally) observational study with dramatic effect
Non-randomized controlled cohort/follow-up study (post-marketing surveillance) provided there are sufficient numbers to rule out a common harm. (For long-term harms the duration of follow-up must be sufficient.)**
Case-series, case-control, or historically controlled studies**
* Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO does not match questions PICO), because of inconsistency between
studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size.
** As always, a systematic review is generally better than an individual study.

28. Step 2
Type of Question: Therapy
Type of Study: RCT 28

29. Additional Resources Remember that these ratings are not always clear.
Refer to the Evidence Pyramid for guidance.
Download the CEBM Evidence Tables from this module’s homepage for detailed explanations.
Additional practice is available in the module quiz.
When in doubt, ask your co-author and editor for assistance.