1. Society of General International Medicine 32 nd Annual Meeting, May 14 th 2009 Elie A. Akl, MD, MPH, PhD David Atkins, MD, MPH Eric Bass, MD, MPH Yngve Falck-Ytter, MD Stephanie Chang, MD, MPH 1

2. Session outline  Introductions, objectives (5 min)  Overview of the GRADE approach (25 min)  Applying the GRADE approach (45 min)  Wrap-up (10 min)  Session evaluation (5 min)

3. Disclosure  Presenters are members of the GRADE working group and have received honoraria related to this work that were deposited into research accounts  No conflict of interest related to pharmaceutical industry

4. Objectives

5. Learning objectives  To enumerate GRADE categories for quality of evidence  To list the GRADE factors that affect the quality of evidence  To apply the GRADE approach to a specific body of evidence  To discuss the strengths and limitations of the GRADE approach

6. Overview of the GRADE approach

7. G rades of R ecommendation A ssessment, D evelopment and E valuation CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005, AJRCCM 2006, Chest 2006, BMJ 2008

8. “Extent to which confidence in estimate of effect adequate to support decision” GRADE definition of Quality of Evidence

9. GRADE rating of outcomes  GRADE rates the quality of evidence for each outcome separately  The type of evidence may be different for different outcomes  Different audiences are likely to have varying perspective on the importance of outcomes  GRADE considers desirable and undesirable outcomes and rates their relative importance 9

10.  Desirable outcomes  lower mortality  reduced hospital stay  reduced duration of disease  reduced resource expenditure  Undesirable outcomes  adverse reactions  the development of resistance  costs of treatment GRADE rating of outcomes

11. 2 Critical for decision making Important, but not critical for decision making Of low importance 5 6 7 8 9 3 4 1 GRADE rating of outcomes

12.  Ranking outcomes by their relative importance can help to focus attention on those outcomes that are considered most important  Outcome choice should be based on what is important, and not what was measured 12 GRADE rating of outcomes

13. GRADE uses a comprehensive and transparent conceptual framework for rating the quality of evidence 13

14.  High:  Moderate:  Low:  Very low: GRADE levels of Evidence

15.  High: considerable confidence in estimate of effect  Moderate: further research likely to have impact on confidence in estimate, may change estimate  Low: further research is very likely to impact on confidence, likely to change the estimate  Very low: any estimate of effect is very uncertain GRADE levels of Evidence

16.  Quality starts high for evidence from RCTs  Quality starts low for evidence from observational studies  5 factors lower the quality of evidence  3 factors can increase the quality of evidence Determinants of quality

17. Factors that lower quality 1. Study limitations (in design and execution) 2. Inconsistency 3. Indirectness 4. Reporting bias 5. Imprecision 17

18. 1. Study limitations (in design and execution)  Inappropriate randomization  Lack of concealment  Intention to treat principle violated  Inadequate blinding  Loss to follow-up  Early stopping for benefit Factors that lower quality

19.  From Cates, CDSR 2008 CDSR 2008 Factors that lower quality

20. Overall judgment required Factors that lower quality

21. 2. Inconsistency  Assess for inconsistency (Heterogeneity)  variation in size of effect  overlap in confidence intervals  statistical significance of heterogeneity I2I2  If inconsistency  look for explanation  patients, intervention, outcome, methods  If unexplained inconsistency  downgrade quality Factors that lower quality

22. Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ. “Anticoagulation for the long term treatment of venous thromboembolism in patients with cancer”. Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006650. 2. Inconsistency Heparin or vitamin K antagonists for survival in patients with cancer: Factors that lower quality

23. Capurso G, Schünemann HJ, Terrenato I, Moretti A, Koch M, Muti P, Capurso L, Delle Fave G. Meta-analysis: the use of non-steroidal anti-inflammatory drugs and pancreatic cancer risk for different exposure categories. Aliment Pharmacol Ther. 2007 Oct 15;26(8):1089-99. 2. Inconsistency Non-steroidal drug use and risk of pancreatic cancer: Factors that lower quality

24. 3. Indirectness of Evidence  Differences in populations/patients  mild versus severe COPD  Differences in interventions  all inhaled steroids, new vs. old  Differences in outcomes  important vs. surrogate; Factors that lower quality

25. Alendronate Risedronate Placebo 3. Indirectness of Evidence  indirect comparisons  interested in A versus B  have A versus C and B versus C Factors that lower quality

26. 4. Publication bias  Number of small studies  Faster and multiple publication of “positive” trials  Fewer and slower publication of “negative” trials Factors that lower quality

27. Egger M, Smith DS. BMJ 1995;310:752-54 27 I.V. Mg in acute myocardial infarction Publication bias Meta-analysis Yusuf S.Circulation 1993 ISIS-4 Lancet 1995

28. Egger M, Cochrane Colloquium Lyon 2001 28 Funnel plot Standard Error Odds ratio 0.10.313 3 2 1 0 100.6 Symmetrical: No publication bias

29. Egger M, Cochrane Colloquium Lyon 2001 29 Funnel plot Standard Error Odds ratio 0.10.313 3 2 1 0 100.6 Asymmetrical: Publication bias?

30. Egger M, Smith DS. BMJ 1995;310:752-54 30 I.V. Mg in acute myocardial infarction Publication bias Meta-analysis Yusuf S.Circulation 1993 ISIS-4 Lancet 1995

31. Egger M, Smith DS. BMJ 1995;310:752-54 31 Meta- analysis confirme d by mega- trials

32. 5. Imprecision  small sample size  small number of events  wide confidence intervals  uncertainty about magnitude of effect  how to decide if CI too wide?  grade down one level?  grade down two levels? Factors that lower quality

33. Factors that raise quality 1. Large magnitude of effect 2. Dose response relation 3. All plausible confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed 33

34. 1. Large magnitude of effect  large (RRR 50%) can raise by one level  very large (RRR 80%) can raise by two levels  common criteria  everyone used to do badly  almost everyone does well  Examples  oral anticoagulation for mechanical heart valves  insulin for diabetic ketoacidosis  hip replacement for severe osteoarthritis Factors that raise quality

35. 2. Dose response relation  higher INR – increased bleeding  childhood lymphoblastic leukemia  risk for CNS malignancies 15 years after cranial irradiation  no radiation: 1% (95% CI 0% to 2.1%)  12 Gy: 1.6% (95% CI 0% to 3.4%)  18 Gy: 3.3% (95% CI 0.9% to 5.6%) Factors that raise quality

36. 3. All plausible confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed Factors that raise quality

37.  Example 1: higher death rates in private for- profit versus private not-for-profit hospitals  patients in the not-for-profit hospitals likely sicker than those in the for-profit hospitals  for-profit hospitals are likely to admit a larger proportion of well-insured patients than not-for-profit hospitals (and thus have more resources with a spill over effect) Factors that raise quality

38.  Example 2: hypoglycaemic drug phenformin causes lactic acidosis  The related agent metformin is under suspicion for the same toxicity.  Large observational studies have failed to demonstrate an association  Clinicians would be more alert to lactic acidosis in the presence of the agent Factors that raise quality

39. Summary of GRADE framework for rating the quality of evidence 39

41. Evidence Profiles and Summary of Findings (SoF) Tables summarize the rating of the quality of evidence across selected outcomes 41

42. 42

43. 43

44. Applying the GRADE approach Exercise: parenteral anticoagulation for prolonging the survival of patients with cancer

45. Wrap-up

46. 46

47. Advantages of GRADE  Developed by a widely representative group of international guideline developers  Clear separation between quality of evidence and strength of recommendations  Explicit evaluation of the importance of outcomes  Explicit, comprehensive criteria for downgrading and upgrading quality of evidence ratings 47

48. Advantages of GRADE  Transparent process of moving from evidence to recommendations  Explicit acknowledgment of values and preferences  Clear, pragmatic interpretation of strong versus weak recommendations for clinicians, patients, and policy makers  Useful for systematic reviews and health technology assessments, as well as guidelines 48

49. Disadvantages of GRADE  Involves a number of judgments that might affect its reliability  Requires expertise/training 49

50. Session evaluation

51. Thank you!