Two questions in grading recommendations Are you sure?Are you sure? –Yes: Grade 1 –No: Grade 2 What is the methodological quality of the underlying evidenceWhat is the methodological quality of the underlying evidence –High quality: Grade A –Intermediate quality: Grade B –Poor quality: Grade C

What is the methodological quality of the underlying evidence? High quality evidence: Grade AHigh quality evidence: Grade A - Randomized trials, few limitations Intermediate quality evidence: Grade BIntermediate quality evidence: Grade B –Randomized trials with important limitations varying results (heterogeneity)varying results (heterogeneity) major methodological flawsmajor methodological flaws total sample size in all studies combined under 100total sample size in all studies combined under 100 Poor quality: Grade CPoor quality: Grade C –Observational studies B: Randomized trials, inconsistent results C: Observational studies Benefits vs Risks/costs TradeoffBenefits vs Risks/costs Tradeoff I: Clear that benefits do/don’t outweight risks/cost II: Benefit vs risk/cost tradeoff not clear

Generalizing result ASA in unstable angina 50% RRRASA in unstable angina 50% RRR –trials of patients up to 80 years old –no trials in those over still Grade A, or C? Warfarin in atrial fibrillationWarfarin in atrial fibrillation –lots of trials in non-valvular atrial fibrillation –no trials in valvular a fib -- still Grade A, or C? IV heparin for pregnant women with DVTIV heparin for pregnant women with DVT –lots of trials in non-pregnant –no trials in pregnant -- still Grade A, or C

What do we mean by “are you sure? 1st: Is there uniformity in assessment of risk/benefit in your consensus group and in the community1st: Is there uniformity in assessment of risk/benefit in your consensus group and in the community –If yes, probably Grade 1 –If no, probably Grade 2 1st: Is the risk/benefit clear1st: Is the risk/benefit clear –Grade 1: Benefit clearly greater than risk or risk clearly greater than benefit –Grade 2: Risk/benefit uncertain

What do we mean by “are you sure” Consider patient valuesConsider patient values Example: Different values of stroke/bleedingExample: Different values of stroke/bleeding 3rd: Would (almost) all your patients make the same choice?3rd: Would (almost) all your patients make the same choice? –Yes: Grade 1 –No: Grade 2 4th: Would a decision aid be useful and worthwhile?4th: Would a decision aid be useful and worthwhile? –No, no need: Grade 1 –Yes, needed: Grade 2

What do we mean by “are you sure? 5th: Directive to clinicians5th: Directive to clinicians –Grade 1: just do it –Grade 2: think about it your own judgment of strength of evidenceyour own judgment of strength of evidence your own judgment of risk/benefityour own judgment of risk/benefit talk to your patients, their values may impacttalk to your patients, their values may impact

Risk/Benefit clear Aspirin with acute myocardial infarctionAspirin with acute myocardial infarction –25% reduction in relative risk, narrow confidence interval –side effects trivial, cost negligible –benefit obviously much greater than risk/cost, 1(A) Thrombolysis in MI symptoms with only ST changesThrombolysis in MI symptoms with only ST changes –no difference from placebo, narrow confidence interval –small risk of intracranial hemorrhage –risk obviously greater than possible benefit, 1(A)

Judgment: Benefits vs Risks/Costs Seriousness of outcomeSeriousness of outcome Magnitude of effectMagnitude of effect Precision of treatment effectPrecision of treatment effect Risk of target eventRisk of target event Risk of serious adverse eventsRisk of serious adverse events Cost of therapyCost of therapy ValuesValues

Cost and Magnitude of Effect Clopidigrel vs ASA for atheroembolismClopidigrel vs ASA for atheroembolism –8.7% RRR relative to ASA –5.83% to 5.32% in MI, ischemic stoke and death –NNT 200, cost $1,052 vs. $21 –some will feel benefits not worth extra costs and therefore 2(A) worth extra costs and therefore 2(A)

Cost and toxicity TPA versus streptokinaseTPA versus streptokinase –RCT shows 15% RRR with TPA TPA larger costTPA larger cost TPA increased risk of intracranial hemorrhageTPA increased risk of intracranial hemorrhage Varying practice, unclear risk/benefitVarying practice, unclear risk/benefit –Grade 2 (B)

Imprecision of treatment effect Should dipyridamole be added to aspirin after MI?Should dipyridamole be added to aspirin after MI? 1998 single RCT1998 single RCT –85 deaths in 810 ASA alone, 87 in 810 ASA and dipyridamole –RR with ASA 0.98 (95% CI 0.70 to 1.26) Recommendation: don’t use dipyridamoleRecommendation: don’t use dipyridamole Clearly Grade A; ? 1 or 2Clearly Grade A; ? 1 or 2 Consensus criterion: Grade 1(A)Consensus criterion: Grade 1(A)

Precision of estimate RR with ASA 0.98 (95% CI 0.70 to 1.26)RR with ASA 0.98 (95% CI 0.70 to 1.26) –ASA may reduce risk relative to combination by 30% –combination may reduce risk relative to ASA by 26% Are we sure dipyridamole doesn’t add - NoAre we sure dipyridamole doesn’t add - No Patient: I’ll take any low cost low toxicity medication that MIGHT helpPatient: I’ll take any low cost low toxicity medication that MIGHT help Risk/benefit or patient value criteria: 2 (A)Risk/benefit or patient value criteria: 2 (A) How to use confidence intervalHow to use confidence interval –look at boundaries, is decision same at either end?

Judgement: benefits versus risks/costs Seriousness of outcomeDeath vs post-phlebitic syndrome Magnitude of effect68% RRR warfarin in a fib, vs 9% RRR with clopidigrel in CAD 9% RRR with clopidigrel in CAD Precision of treatment effectwarfarin in a fib vs. ASA in a fib Risk of target eventwarfarin in high vs low risk a fib Risk of serious adverse eventcoumadin versus aspirin CostsASA vs. clopidigrel Values (every decision)high value on avoiding stroke: TPA; clopidigrel; warfarin

1 A recommendation Patients with atrial fibrillation and additional risk factors for arterial embolism without excessive bleeding risk should receive warfarinPatients with atrial fibrillation and additional risk factors for arterial embolism without excessive bleeding risk should receive warfarin strong recommendation, can apply to most patients in most circumstances with no reservations

1 B recommendation Clinicians should not administer magnesium sulfate to patients with acute myocardial infarctionClinicians should not administer magnesium sulfate to patients with acute myocardial infarction –meta-analysis of smaller RCTs +ve, large RCT -ve Strong recommendation, likely to apply to most patients

1 C recommendation Patients with acute peripheral arterial thrombi or emboli should be systematically heparinizedPatients with acute peripheral arterial thrombi or emboli should be systematically heparinized –No RCTs, strong biological rationale Intermediate strength recommendation, may change when stronger evidence available

2 A recommendation Men over 50 without established CAD, but with one or more additional risk factors for CAD should take daily ASAMen over 50 without established CAD, but with one or more additional risk factors for CAD should take daily ASA –RCT shows lower risk of MI but may be higher risk of cerebral bleed –both risks very low, individual values may determine decision Intermediate strength recommendation, best action may differ depending on circumstances or patients’ or societal values

2 B recommendation Intra-arterial thrombolytic therapy may be used as an alternative to surgery in patients with acute peripheral arterial thrombi or emboliIntra-arterial thrombolytic therapy may be used as an alternative to surgery in patients with acute peripheral arterial thrombi or emboli –2 RCTs, show comparable results, 1 surgery clearly better Weak recommendation, alternative approaches likely to be better for some patients or circumstances

2 C recommendation Pregnant women with previous venous thrombosis associated with a transient risk factor should receive surveillance only during pregnancy and heparin and warfarin post-partumPregnant women with previous venous thrombosis associated with a transient risk factor should receive surveillance only during pregnancy and heparin and warfarin post-partum –incidence of thrombosis and magnitude of benefit with therapy unestablished Very weak recommendation, alternatives equally reasonable

Are we producing guidelines? Yes!Yes! –Recommendations from authoritative body intended to influence clinical practice Shanneyfelt, JAMA;1999;281:1900Shanneyfelt, JAMA;1999;281:1900 –279 guidelines published 1985 to 1997 –adherence to standards for evidence summary 34% –adherence to standards for making recommendations 6% ConclusionsConclusions –we should do better or –everyone else doing equally badly, we don’t need to worry

Systematic review of evidence Formal statement of eligibility criteria for each questionFormal statement of eligibility criteria for each question –patients, interventions, outcomes, methodology Systematic search for evidenceSystematic search for evidence Explicit process of data abstractionExplicit process of data abstraction Pooling of resultsPooling of results –wherever appropriate –systematic approach

From evidence to recommendations Value judgments implicit in every recommendationValue judgments implicit in every recommendation Whose valuesWhose values –Ours? –Society? –Patients? PossibilitiesPossibilities –explicit elicitation of values –include people with different values/perspectives patient; primary care doctorpatient; primary care doctor