1. August 27 - 30, 2012 GUIDELINE AND SYSTEMATIC REVIEW WORKSHOP Dr. Elie Akl Dr. Holger Schünemann Dr. Ruth Kalda Dr. Alar Irs

2. August 27, 2012 INTRODUCTION TO GUIDELINE DEVELOPMENT IN THE CONTEXT OF EVIDENCE BASED MEDICINE Dr. Holger Schünemann

3. History - 1967 – Founded by David Sackett - 6 chairs since - Instrumental in specialty of Clinical Epidemiology, origin of “Evidence-Based Medicine” People 45 full time and joint faculty ~ 120 associate & part time faculty; 19 emeritus ~ 180 staff ~ 200 PhD and Master students The Department of Clinical Epidemiology & Biostatistics at McMaster

4. Why EBM? Thrombolysis in Myocardial infarction Antman et al., JAMA, 1992; 268: 240-248

6. What is a guideline? "Guidelines are recommendations intended to assist providers and recipients of health care and other stakeholders to make informed decisions. Recommendations may relate to clinical interventions, public health activities, or government policies." WHO 2003, 2007

7. When do we need guidelines? Knowledge gap? – Is a guideline the right approach? Diagnosis? – Too many cases? Too few? Variation? Treatment? – Under? Over? Variation? Something new? Screening? Quality of care? Integration of care? Other?

8. What healthcare workers want… A guideline is not a textbook or a cookbook To KNOW that the guideline is evidence based But not necessarily all of the evidence… To have it easy to use and accessible Clear recommendations (more on that later)

10. Guideline development Process

11. Working with evidence For key recommendations: – Search for and retrieve all available evidence – Identify relevant SRs – Formally assess quality of evidence – GRADE (systematic and transparent approach)

12. Institute of Medicine Report on Trustworthy guidelines Be based on a systematic review of the existing evidence; Be developed by a knowledgeable, multidisciplinary panel of experts and representatives from key affected groups; Consider important patient subgroups and patient preferences as appropriate; Be based on an explicit and transparent process that minimizes distortions, biases, and conflicts of interest; Provide a clear explanation of the logical relationships between alternative care options and health outcomes, and provide ratings of both the quality of evidence and the strength of recommendations; and Be reconsidered and revised as appropriate when important new evidence warrants modifications of recommendations.

13. Guideline International Network

14. Institute of Medicine Report on Trustworthy guidelines 2011 Be based on a systematic review of the existing evidence; Be developed by a knowledgeable, multidisciplinary panel of experts and representatives from key affected groups; Consider important patient subgroups and patient preferences as appropriate; Be based on an explicit and transparent process that minimizes distortions, biases, and conflicts of interest; Provide a clear explanation of the logical relationships between alternative care options and health outcomes, and provide ratings of both the quality of evidence and the strength of recommendations; and Be reconsidered and revised as appropriate when important new evidence warrants modifications of recommendations.

15. The origin of evidence appraisal systems Canadian Task Force on the Periodic Health Examination, CMAJ, 1979

16. Oxford Centre for Evidence Based Medicine

17. USPSTF - Grade Definitions After May 2007: Certainty Level of CertaintyDescription HighThe available evidence usually includes consistent results from well-designed, well-conducted studies in representative primary care populations. These studies assess the effects of the preventive service on health outcomes. This conclusion is therefore unlikely to be strongly affected by the results of future studies. Moderate The available evidence is sufficient to determine the effects of the preventive service on health outcomes, but confidence in the estimate is constrained by such factors as: The number, size, or quality of individual studies. Inconsistency of findings across individual studies. Limited generalizability of findings to routine primary care practice. Lack of coherence in the chain of evidence. As more information becomes available, the magnitude or direction of the observed effect could change, and this change may be large enough to alter the conclusion. Low The available evidence is insufficient to assess effects on health outcomes. Evidence is insufficient because of: The limited number or size of studies. Important flaws in study design or methods. Inconsistency of findings across individual studies. Gaps in the chain of evidence. Findings not generalizable to routine primary care practice. Lack of information on important health outcomes. More information may allow estimation of effects on health outcomes. The USPSTF defines certainty as "likelihood that the USPSTF assessment of the net benefit of a preventive service is correct."

19. Recommendations for prognosis – Use prognostic information to determine baseline risk for healthcare decisions

20. 20

21. 21

22. Center for Disease Control and Prevention (CDC)

23. Your patient…as an internist 68 year old man with hypertension and non- valvular atrial fibrillation > 3 months

24. Atrial Fibrillation - Stroke

25. The clinically sensible question Population: Does in patients with atrial fibrillation Intervention: oral anticoagulation Comparison:compared with no therapy Outcomes:reduce the risk for embolic stroke, increase the risk for bleeding, increase burden…? PICO

26. Which approach? Evidence Recommendation B Class I A 1 IV C Organization  AHA  ACCP  SIGN Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic mitral valve disease

27. What to do?

28. Hierarchy of evidence based on quality STUDY DESIGN Randomized Controlled Trials Cohort Studies and Case Control Studies Case Reports and Case Series, Non-systematic observations Expert Opinion BIAS

29. Issues with evidence hierarchies Does one size fit all? Should RCTs be on top? What are the special strength of observational studies?

30. Healthcare problem recommendation “Healthy people” “Rare disease” “Long term perspective” “Few RCTs” “Lots of other things”

31. Explain the following? Confounding, effect modification & ext. validity Concealment of randomization Blinding (who is blinded in a double blinded study?) Intention to treat analysis and its correct application P-values and confidence intervals “Everything should be made as simple as possible but not simpler.”

32. BMJ 2003 BMJ, 2003 Relative risk reduction: ….> 99.9 % (1/100,000) U.S. Parachute Association reported 821 injuries and 18 deaths out of 2.2 million jumps in 2007

33. Simple hierarchies are (too) simplistic STUDY DESIGN Randomized Controlled Trials Cohort Studies and Case Control Studies Case Reports and Case Series, Non-systematic observations BIAS Expert Opinion Schünemann & Bone, 2003

34. GRADE Working Group G rades of R ecommendation A ssessment, D evelopment and E valuation CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005, AJRCCM 2006, Chest 2006, BMJ 2008 International group: ACCP, AHRQ, Australian NMRC, BMJ Clinical Evidence, CC, CDC, McMaster Uni., NICE, Oxford CEBM, SIGN, UpToDate, USPSTF, WHO Aim: to develop a common, transparent and sensible system for grading the quality of evidence and the strength of recommendations (over 100 systems) International group of guideline developers, methodologists & clinicians from around the world (>300 contributors) – since 2000

35. GRADE Uptake  World Health Organization  Allergic Rhinitis in Asthma Guidelines (ARIA)  American Thoracic Society  American College of Physicians  European Respiratory Society  European Society of Thoracic Surgeons  British Medical Journal  Infectious Disease Society of America  American College of Chest Physicians  UpToDate®  National Institutes of Health and Clinical Excellence (NICE)  Scottish Intercollegiate Guideline Network (SIGN)  Cochrane Collaboration  Infectious Disease Society of America  Clinical Evidence  Agency for Health Care Research and Quality (AHRQ)  Partner of GIN  Over 60 (major) organizations

36. Evidence based healthcare decisions Research evidence Population/societal values and preferences (Clinical) state and circumstances Expertise Haynes et al. 2002

37. Your patient…as an internist 68 year old man with hypertension and non- valvular atrial fibrillation > 3 months – diabetes – large left atrium (→ cardioversion unlikely to be successful) – no history of strokes or transient ischemic attacks (TIAs) Terrified of having a stroke

38. Risk factors for stroke with NVAF CHADS 2 score for assessment of stroke risk in patients with non-rheumatic AF Risk factorPoints Recent Congestive heart failure exacerbation 1 History of Hypertension1 Age 75 years or older1 Diabetes1 Prior history of Stroke or TIA2

39. Risk factors for stroke with NVAF CHADS 2 score for assessment of stroke risk in patients with non-rheumatic AF Risk factorPoints Recent Congestive heart failure exacerbation 1 History of Hypertension1 Age 75 years or older1 Diabetes1 Prior history of Stroke or TIA2 CHADS = 2

40. Evidence concerning NVAF and stroke* Risk of stroke if untreated (CHADS =2): 45/1000 per year Relative Risk Reduction for stroke – Warfarin:0.64 (95%CI 0.51-0.77) RRI for major bleeding – Warfarin: 2.58 (95%CI 1.12-5.97) * Pooled estimates of treatment effect in this evidence profile are from a meta-analysis conducted for these guidelines, including data from 6 RCTs of adjusted-dose vitamin K antagonist therapy versus no antithrombotic therapy (AFASAK I, BAATAF, CAFA, EAFT, SPAF I, SPINAF), You et al., in press. Physician accuracy in estimating risk: no better than chance…

41. Primum non nocere “Primum non net nocere”

42. Evidence based healthcare decisions Research evidence Population/societal values and preferences (Clinical) state and circumstances Expertise Haynes et al. 2002

43. Balancing desirable and undesirable consequences ↑ burden ↑ resources ↑ dietary restriction ↑ bleeding ↑ QoL ↓ stroke ↓ Morbidity ↑ survival ForAgainst Conditional Strong

44. Balancing desirable and undesirable consequences ↑ burden ↑ resources ↑ dietary restriction ↑ bleeding ↑ QoL ↓ stroke ↓ Morbidity ↑ survival ForAgainst Conditional Strong

45. Balancing desirable and undesirable consequences ↑ burden ↑ resources ↑ dietary restriction ↑ bleeding ↑ QoL ↓ stroke ↓ Morbidity ↑ survival ForAgainst Conditional Strong

46. Balancing desirable and undesirable consequences ↑ burden ↑ resources ↑ dietary restriction ↑ bleeding ↑ QoL ↓ stroke ↓ Morbidity ↑ survival ForAgainst Conditional Strong

47. Balancing desirable and undesirable consequences ↑ burden ↑ resources ↑ dietary restriction ↑ bleeding ↑ QoL ↓ stroke ↓ Morbidity ↑ survival ForAgainst Conditional Strong

48. Balancing desirable and undesirable consequences ↑ burden ↑ resources ↑ dietary restriction ↑ bleeding ↑ QoL ↓ stroke ↓ Morbidity ↑ survival ForAgainst Conditional Strong

49. Summary from the practitioner’s perspective for this patient must anticoagulate 100 people with NVAF for 1 year to prevent 3 strokes per year (30 fewer per 1000 or NNT of 33) for 100 anticoagulated patients in the community, this will cause 1 additional people to have a major bleed per year (8 more per 1000 or NNT of 125)

50. Summary from this patient’s perspective If you take anticoagulants – your risk of stroke in the coming year will decrease from 4.5% to 1.5% per year but – your risk of having a major bleed will increase from 0.5% to 1% per year

51. GRADE recommendations For patients with AF, including those with paroxysmal AF, who are at high risk of stroke (e.g., CHADS 2 score ≥ 2), we recommend oral anticoagulation rather than no therapy (strong recommendation, high quality evidence) For patients with AF, including those with paroxysmal AF, who are at low risk of stroke (e.g., CHADS 2 score = 0), we suggest no therapy rather than antithrombotic therapy (weak recommendation, moderate quality evidence).

52. Remark: Patients who place an exceptionally high value on stroke reduction and a low value on avoiding bleeding and the burden associated with antithrombotic therapy are likely to choose antithrombotic therapy rather than no antithrombotic therapy. Other factors that may influence the choices above are a consideration of patient-specific bleeding risk and the presence of additional risk factors for stroke, including age 65 to 74 years and female gender, which have been more consistently validated, and vascular disease, which has been less well validated. The presence of multiple non-CHADS 2 risk factors for stroke may favor oral anticoagulation therapy. GRADE recommendations

53. Systematic review Guideline development PICOPICO Outcome Formulate question Rate importance Critical Important Critical Not important Create evidence profile with GRADEpro Summary of findings & estimate of effect for each outcome Grade overall quality of evidence across outcomes based on lowest quality of critical outcomes Randomization increases initial quality 1.Risk of bias 2.Inconsistency 3.Indirectness 4.Imprecision 5.Publication bias Grade down Grade up 1.Large effect 2.Dose response 3.Opposing bias & Confounders Rate quality of evidence for each outcome Select outcomes Very low Low Moderate High Grade recommendations For or against (direction)  Strong or conditional/weak (strength) By considering balance of:  Quality of evidence  Balance benefits/harms  Values and preferences Revise if necessary by considering:  Resource use (cost) Formulate Recommendations (  |  …) “We recommend using…” | “Clinicians should…” “We suggest using…” | “Clinicians might…” “We suggest not using…”| “Clinicians … not…” “We recommend not using…”| “Clinicians should not…” Outcomes across studies Guideline Input?

54. Summary Evidence based decision making requires consideration of many factors Evidence can be complex and needs careful integration

55. Asking questions and choosing outcomes Elie Akl

56. Systematic review Guideline development PICOPICO Outcome Formulate question Rate importance Critical Important Critical Not important Create evidence profile with GRADEpro Summary of findings & estimate of effect for each outcome Grade overall quality of evidence across outcomes based on lowest quality of critical outcomes Randomization increases initial quality 1.Risk of bias 2.Inconsistency 3.Indirectness 4.Imprecision 5.Publication bias Grade down Grade up 1.Large effect 2.Dose response 3.Opposing bias & Confounders Rate quality of evidence for each outcome Select outcomes Very low Low Moderate High Grade recommendations For or against (direction)  Strong or conditional/weak (strength) By considering balance of:  Quality of evidence  Balance benefits/harms  Values and preferences Revise if necessary by considering:  Resource use (cost) Formulate Recommendations (  |  …) “The panel recommends that ….should...” (  |  …) “The panel suggests that ….should...” (  ? |  …) “The panel suggests to not...” (  ? |  …) “The panel recommends to not...”(  |  …) Outcomes across studies Guideline Input?   OOO  O  OO

57. Guidelines and questions Guidelines are a way of answering questions about clinical, communication, organisational or policy interventions, in the hope of improving health care or health policy. It is therefore helpful to structure a guideline in terms of answerable questions. WHO Guideline Handbook, 2008

58. Questions Should be practice NOT evidence driven

59. Types of questions Background Questions Definition: What is COPD? Mechanism: What is the mechanism of action of mucolytic therapy? Foreground Questions Efficacy : In patients with COPD, does mucolytic therapy improve survival?

60. Good questions... Questions you have when trying to decide what to prescribe/recommend to your patient Questions you have when trying to decide what to provide in your country/region/ clinic What should you do with the person in front of you?

61. Types of questions Background Questions Definition: What is COPD? Mechanism: What is the mechanism of action of beta-agonists? Foreground Questions Efficacy : In patients with COPD, do beta-agonists improve survival?

62. Should we recommend oseltamivir versus no antiviral therapy in adults with influenza-like illness? Should we recommend that soft drink machines be banned from schools to prevent childhood obesity? Should in vitro specific IgE determination be used for the diagnosis of IgE-mediated cow’s milk allergy in patients suspected of cow’s milk allergy?

63. Framing a foreground question PICOPICO

64. Population: Intervention: Comparison: Outcomes:

65. Framing a foreground question PICOPICO

66. Population: Intervention: Comparison: Outcomes:

67. Case scenario A 13 year old girl who lives in rural Indonesia presented with flu symptoms and developed severe respiratory distress over the course of the last 2 days. She required intubation. The history reveals that she shares her living quarters with her parents and her three siblings. At night the family’s chicken stock shares this room too and several chicken had died unexpectedly a few days before the girl fell sick. Potential interventions: antivirals, such as neuraminidase inhibitors oseltamivir and zanamivir

68. What are examples of: Background questions Foreground questions Population: Intervention: Comparison: Outcomes: 68

69. Framing a foreground question Population: Avian Flu/influenza A (H5N1) patients Intervention: Oseltamivir (or Zanamivir) Comparison: No pharmacological intervention Outcomes: Mortality, hospitalizations, resource use, adverse outcomes, antimicrobial resistance Schunemann, Hill et al., The Lancet ID, 2007

70. Choosing outcomes Every decision comes with desirable and undesirable consequences  Developing recommendations must include a consideration of desirable and undesirable outcomes  Outcomes should be patient important outcomes.

71. Outcomes Should be importance driven NOT evidence driven

72. desirable outcomes – lower mortality – reduced hospital stay – reduced duration of disease – reduced resource expenditure undesirable outcomes – adverse reactions – the development of resistance – costs of treatment Choosing outcomes

73.  What if what is important is not measured?  What if what is measured is not important?  How do we make sure we’ve covered all important outcomes? Choosing outcomes

74. Decision makers (and guideline authors) need to consider the relative importance of outcomes when balancing these outcomes to make a recommendation Relative importance vary across populations Relative importance may vary across patient groups within the same population When considered critical - evaluate Relative importance of outcomes

75. 2 Critical for decision making Important, but not critical for decision making Of low importance 5 6 7 8 9 3 4 1 Relative importance of outcomes

76. Nausea 2 Hierarchy of outcomes according to their importance to assess the effect of oseltamivir in patients with H5N1 influenza Importance of endpoints Critical for decision making Important, but not critical for decision making Of low importance 5 Neurological complications 6 Pneumonia 7 Hospital admission 8 Mortality 9 3 4 1

77. Good questions lead to good recommendations There is controversy around the answer There is doubt around the answer Want to confirm the present answer Has a chance of being answered, or will determine research in future Will improve care, cost, quality of life

78. Agenda