1. Systematic reviews, meta-analysis and critical reading of medical literature: Evidence-based medicine
Phyllis W. Speiser, MD
Chief, Div Ped Endo, CCMC
Professor of Pediatrics
Hofstra-NSLIJ School of Medicine

2. Learning Objectives Understand the rationale for a systematic review
Understand the advantages of adding a meta-analysis to a systematic review
Interpret the results of a meta-analysis
Identify the limitations of systematic review & meta-analysis

3. Question 1 The main reason we need systematic reviews is:
A. We can’t trust doctors’ judgment
B. There are conflicting reports in published literature
C. Peer review is faulty
D. There’s a long lag time to publication of clinical trial results

4. Question 2
Meta-analysis adds value to a systematic review by examining:
A. Only positive study results
B. Only randomized controlled trials
C. All studies with comparable design and endpoints
D. All studies including >100 subjects

5. Question 3 The results of meta-analysis provide a:
A. Final decision on how to treat a patient
B. Guidance for the clinician in counseling the patient about treatment choices
C. Justification for prescribing off-label drugs
D. Way of justifying the cost of treatment.

6. The need to be evidence-based
Wide variations in practice
Continued use of ineffective treatments
Excess use of inappropriate treatments
Poor uptake of effective practice
Increasing consumerism
Unvetted Internet information
Direct-to-consumer pharma advertising
Increasing demand on $ resources
Need to demonstrate efficacy
Exponential growth in research
Need to compare & evaluate many studies’ variable quality & conflicting results.

7. Information overload MEDLINE 2012 >5,000 journals surveyed
22,000,000 citations

8. Reproducibility and transparency
Non-reproducible results
Lack of transparency in research methods and findings
This is a problem in all areas of research
This issue has been observed in both clinical and preclinical research
CHALLENGES IN IRREPRODUCIBLE RESEARCH, NATURE, 11/14

9. Suggestions for research transparency
Store & save primary data in a HIPAA secure repository (eg,
Give detailed information about subject demographics and/or lab methods
List assumptions used and types of statistical analysis
Identify computer software versions used

10. Hierarchy of evidence for treatment decisions
Meta-analysis of RCTs
Systematic review of RCTs
Individual RCT
Observational studies
patient-important outcomes
Evidence from different sources can be categorized in a hierarchy:
Meta-analysis or systematic reviews are at the top of the hierarchy.
When summaries of the evidence are not available, individual randomized controlled trials provide the next best evidence.
Next are observational studies. We should try to find studies that focus on outcomes important to the patient.
If there are no clinical studies available we may look at basic scientific research, although caution must be used in extrapolating the results to the clinical setting.
Clinical experience is at the bottom of the hierarchy, either your own or that of colleagues or experts.
Basic research
test tube, animal, human physiology
Clinical experience

11. Types of EBM studies Diagnosis Therapy Prognosis Associations/ Causes
Selecting appropriate diagnostic tests
Therapy
Selecting most effective/safest treatments
Cost-benefit
Prognosis
Outcomes & complications
Associations/ Causes
Identify etiologies: e.g., infectious, environmental, iatrogenic

12. What makes a review “systematic”?
Based on a clearly formulated question
Identifies relevant studies with pre-set criteria
Appraises quality of studies
Summarizes evidence by use of explicit methodology
Recommendations are based on evidence gathered

13. Assessing quality in the medical literature
Study design
Type of intervention
Endpoints examined
Data analysis

14. Quality / validity of studies: Design
Study design
Prospective v. retrospective
Cross-sectional v. longitudinal
Clinic population only v. case-control
Patient selection
Consecutive v. nonconsecutive v. random
Age, racial, ethnic & gender balance
Power analysis to determine subject number?
Number of drop-outs
“Intention to treat”
Lijmer et al. Empiric evidence of design-related bias in studies of diagnostic studies. JAMA 1999;282:1061

15. Quality / validity of studies: Intervention
Nature of intervention
Placebo-controlled v. best current treatment v. uncontrolled
Randomized or not
Blinded or not
Dose-ranging v. single dose
Verification of methods
Same or different assays, inter- & intra- assay variability
Same or different endpoints
Empiric or historical normal reference data
Appropriateness of controls

16. Quality / validity of studies: Data
Data collection
Prospective or retrospective
Intention to treat
Exclusion criteria for outliers
Compliance assurance (eg, weekly phone calls, patient diaries, pill counts, etc)
Statistical analysis
Appropriateness of statistical methods

17. Focused study design Formulate a question Choose the intervention
Control group
Outcome of interest
Example
In adolescents with diabetes and pre-hypertension…
Will 30 min of daily yoga…
Compared with 30 min of daily aerobic exercise
Improve BP to <90% for age after 12 weeks…

18. What is a meta-analysis?
Optional part of a systematic review
Systematic reviews
Meta analysis may be part of a systematic review. May be worth asking participants for egs of when it’s not appropriate to combine studies in meta-analysis.
Systematic reviews may included meta-analyses but meta-analysis may be done with out systematically reviewing the studies – there are egs of this in journals – these therefore may be biased
In the US the terms are used interchangably, but not the case in the UK
Meta-analyses

19. Meta-analysis: Are the studies consistent?
Are variations in results between studies consistent with chance?
If NO, then WHY?
Variation in population
Variation in study methods (biases)
Variation in intervention
Variation in outcome measure (e.g., timing)

20. Pitfalls of meta-analysis
Potential bias in inclusion / exclusion criteria for study selection
Publication bias toward positive results
Keyword search
Size
Number of studies
Sample size, total & individual
Attrition
Length of follow-up

21. Pitfalls of meta-analysis, cont
Methods of meta-analysis
Sensitivity analysis for robustness
Fixed vs random effects
Outlier exclusions
Stratification of subject populations
Conclusions of meta-analysis
Weak if studies on opposite sides of Forest plot: “Heterogeneity”

22. GRADE system: Knowledge translation
Transparent process of moving from evidence to recommendations
Developed by representative group of international guideline developers
Separates quality of evidence & strength of recommendations
Stresses importance of outcomes of alternative management strategies
Explicit acknowledgment of patients’ & providers’ values and preferences
Clear, pragmatic interpretation of strong versus weak recommendations for clinicians, patients, and policy makers

23. Grading evidence
High quality— Further research is very unlikely to change our confidence in the estimate of effect
Moderate quality— Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
Low quality— Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
Very low quality— Any estimate of effect is very uncertain

24. Strength of Recommendations
Factor
High Rank
Low Rank
Quality evidence
RCT
Case series
Balance of risk & benefit
Low toxicity & High efficacy
High toxicity &
High efficacy
Values & preferences
Life-saving or QOL-enhancing
No major advance
Cost
Inexpensive
Costly

25. Evidence-based clinical decisions: Are antibiotics indicated in pediatric otitis media?
Typical case: A 3 year old child with
Ear pain, low grade fever, irritability
Examination shows bilateral otitis media
Should antibiotics be prescribed?
Benefits?
Risks?

26. Glasziou, Cochrane systematic review, 2003
Systematic review of RCTs
Question
Patients: children with otitis media
Intervention: antibiotics
Outcome: resolution of symptoms—WHEN?
Calculate odds ratios & confidence interval for each study & combine comparable data
Comprehensive search
Only 8 high quality studies (N= 2,287 children). These studies had:
Concealed randomization
Double blinding of treatments
Complete follow-up

27. Odds ratios allow comparisons of different studies in meta-analysis
Test/Tx +
Test/Tx -
Disease
A TRUE POS
B FALSE NEG
Unaffected
C FALSE POS
D TRUE NEG
Odds ratio for treatment efficacy =
AD / BC,
or TP x TN / FN x FP

28. Confidence intervals: Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times.
For a 95% confidence interval, if the experiment were repeated many times, 95% of the intervals would contain the true treatment effect.

29. Endpoint #2: Pain at 2 – 7 days improved w/tx
Timing is important!

30. Balanced decisions
Are the differences are both clinically & statistically significant?
Is the benefit greater than the risk?
Failure to resolve pain at 2 to 7 days
20% control
13% antibiotics
Absolute difference 7% in beneficial effect of antibiotics
Number Needed to Treat Effectively = 15
Rates of antibiotics’ side effects
Increase in vomiting, rash, diarrhea 5%
Number Needed to Harm = 20

31. Subclinical hypothyroidism and ischemic heart disease: Winnowing publications for meta-analysis

32. Forest plot of IHD prevalence in SCH and euthyroid controls: Age matters!
<65 yo
>65 yo
Razvi, S. et al. J Clin Endocrinol Metab 2008;93:

33. Evidence-based clinical decisions: Subclinical hypothyroidism
“Only well-powered prospective randomized studies with age-stratified groups, and vascular events as the primary endpoint rather than surrogate markers, will give clear answers to this complex question” of whether & when to treat subclinical hypothyroidism.

34. EBM: Beta blockers post MI-# needed to achieve desired outcome
Prospective studies suggest that Mr. Jones' risk of death in the first year after his infarct is 8%
A meta-analysis of RCTs of beta-blockers after MI suggests a 25% risk reduction
Must treat 50 such pts to prolong a life
Given the relatively small expense & low toxicity of generic beta-blockers, a trial of beta-blockers for Mr. Jones is clearly warranted

35. Thrombolytic therapy in MI: Power in N! Importance of current data!
Textbook/Review
Recommendations
Cumulative
0.5
1.0
2.0
Year
RCTs
Pts
1960
Experimental
Not Mentioned
Rare/Never
Routine
Specific
1965 21 5
1970 1 10 1 2
P<.01 2 8 7
1980 8 1 12
P<.001 M 8
1985 1 4 M 7 1 3 M
P<.00001 5 2 2 M 1 M 15 8 1
1990 M 6 1
Odds Ratio (Log Scale)
Favors Treatment
Favors Control

36. 2012: >5000 Cochrane reviews
Impact factor >5

37. Knowledge Translation Research…
…study of the organization, retrieval, appraisal, refinement, dissemination, and uptake of knowledge (eg, important new knowledge from health research)

38. Knowledge Translation Research
KT Type 1
KT Type 2
Lab
Clinical research
Health care
Zerhoui EA. US biomedical research: basic, translational and clinical sciences. JAMA 2005;294:
Based on Hulley et al. Designing Clinical Research, 2007, p 23

39. Step 1. Generating Research Evidence Barrier Solutions
too little research addressing “real world” problems
large, simple randomized trials
“head to head” comparisons
There are, of course, many barriers along the KT steps from evidence to practice.
I’ll briefly review one or 2 at each step, and try to do this in a constructive way, as I believe that there are practical solutions for at least some of the barriers.
First is the fact that much of the most advanced research we have to synthesize doesn’t address important questions in real world clinical settings. Of course, real world research is generally more expensive to do. Many researchers affiliated with Cochrane have pioneered large, simple trials in usual clinical settings and we will need many more of them in the future.
We also need to consistently push for head-to-head comparisons with current best therapy, rather than placebos, when efficacious options are available.

40. Possible comparators current placebo usual care best care
regulatory
requirement for
drugs (not devices or services)
Most use for clinical decisions

41. 43 Examples Computerized decision support Evidence-based textbooks
Evidence-based journal abstracts
>57,000 EBM articles, 2009
Systematic reviews
Original journal articles 43

42. Decision tools

43. Informed decision
For the physician: Integrate MD’s expertise with best published data
For the patient: MD discusses:
Benefits
Risks
Cost
Individual values & preferences

44. Other resources:
Center for Evidence Based Medicine, Oxford Univ (CEBM.net), Various tutorials.
McMaster Inst., Institute of Medicine, Online statistics calculators.
EBM online tutorial:

45. Summary: Evidence-based medicine (EBM)?
Without EBM we are helpless in the face of
misguided experts
overenthusiastic experts
failure to report negative studies/ adverse outcomes
drug company hype
Without EBM our ability is limited
to understand difficult tradeoffs
to help our patients make difficult decisions

46. Learning Objectives Understand the rationale for a systematic review
Understand the advantages of adding a meta-analysis to a systematic review
Interpret the results of a meta-analysis
Identify the limitations of systematic review & meta-analysis

47. Question 1 The main reason we need systematic reviews is:
A. We can’t trust doctors’ judgment
B. There are conflicting reports in published literature
C. Peer review is faulty
D. There’s a long lag time to publication of clinical trial results

48. Question 2
Meta-analysis adds value to a systematic review by examining:
A. Only positive study results
B. Only randomized controlled trials
C. All studies with comparable design and endpoints
D. All studies including >100 subjects

49. Question 3 The results of meta-analysis provide a:
A. Final decision on how to treat a patient
B. Guidance for the clinician in counseling the patient about treatment choices
C. Justification for prescribing off-label drugs
D. Way of justifying the cost of treatment.