Why Grade the Evidence? target audience for Cochrane reviewstarget audience for Cochrane reviews –clinicians interested in the question –policy makers, consumers many in audience will have limitationsmany in audience will have limitations –time –methodological sophistication

Implications of Limitations limited time/sophisticationlimited time/sophistication –is evidence strong, inferences secure? –is evidence weak, inferences insecure? shorthand summaryshorthand summary –enhance usefulness of reviews Gray Elrodt in the BerkshiresGray Elrodt in the Berkshires –post-MI smoking cessation, aspirin, beta blocker, ACE inhibitor, statin

Grade for Specific Question too vague: alendronate in osteoporosistoo vague: alendronate in osteoporosis patients – post-menopausal womenpatients – post-menopausal women interventionintervention –daily alendronate, dose 10 to 20 mg. outcome – non-vertebral fracturesoutcome – non-vertebral fractures

What Influences Grade? study designstudy design –basic –detailed design and execution consistencyconsistency directnessdirectness reporting biasreporting bias

Summary Methodological Quality study designstudy design –randomization –observational study detailed design and executiondetailed design and execution –concealment –balance in known prognostic factors –intention to treat principle observed –blinding –completeness of follow-up

Summary Methodologic Quality consistency of resultsconsistency of results if inconsistency, look for explanationif inconsistency, look for explanation –patients, intervention, outcome, methods no clear thresholdno clear threshold –size of effect, confidence intervals, statistical significance

Risk Difference of Conversion to Sinus Rhythm Amiodarone vs Placebo or Digoxin or CCB Favours Control Favours Amiodarone ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' Bianconi (-0.06 to 0.11) Galperin (0.19 to 0.47) Hohnloser (0.06 to 0.26) Natale (0.51 to 0.83) Villani (0.06 to 0.33) Cowan (-0.19 to 0.36) Noc (0.52 to 1.00) Capucci (-0.41 to 0.20) Cochrane (-0.06 to 0.33) Donovan (-0.21to 0.27) Hou (-0.01 to 0.43) Kondili (-0.16 to 0.44) Galve (-0.11 to 0.27) Kontoyannis (0.11 to 0.50) Bellandi (0.13 to 0.41) Cotter (0.13 to 0.43) Kochiadakis (0.15 to 0.41) Joseph (-0.02 to 0.39) Peukurinen (0.31 to 0.72) Vardas (0.28 to 0.53) Cybulski (0.24 to 0.54) Pooled Risk Difference 0.26 (0.18 to 0.34) AF Duration > 48 hrs AF Duration </= 48 hrs

Relative Risk of Hospital Mortality: Adult Patients ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! Shortell , Keeler 220 4, Hartz 2,368 3,107, Manheim MH 1,252 1,537, Manheim FS 1,617 2,228, Kuhn 2,580 3,353, Pitterle 3,482 4,529, Mukamel 1,653 5,298, Bond 3,224 4,210, Yuan Medical 3,316 7,386, Yuan Surgical -- 4,396, Lanska , McClellan 2, , Sloan 2,360 7, Totals 26,399 36,402, Relative Risk and 95% CI Favours Private Not-For-Profit Favours Private For-Profit Study Number of Hospitals Number of Patients % Weight Random Effects Pooled Estimate

Directness of Evidence indirect treatment comparisonsindirect treatment comparisons –interested in A versus B –have A versus C and B versus C alendronate vs risedronatealendronate vs risedronate –both versus placebo, no head-to-head

Four Levels of “Directness” patients meet trials’ eligibility criteriapatients meet trials’ eligibility criteria not included, but no reason to questionnot included, but no reason to question –slight age difference, comorbidity, race some question, bottom line applicablesome question, bottom line applicable –valvular atrial fibrillation serious question about biologyserious question about biology –heart failure trials applicability to aortic stenosis

Levels of Directness interventionsinterventions –same drugs and doses –similar drugs and doses –same class and biology –questionable class and biology outcomesoutcomes –same outcomes –similar (duration, quality of life) –less breathlessness for role function –laboratory exercise capacity for q of life

Magnitude, Precision, Reporting Bias magnitude not generally part of qualitymagnitude not generally part of quality –but very large magnitude can upgrade precision not generally part of qualityprecision not generally part of quality –but sparse data can lower quality reporting biasreporting bias –high likelihood can lower quality

Grading System high qualitywell done RCThigh qualitywell done RCT intermediatequasi-RCTintermediatequasi-RCT lowwell done observationallowwell done observational insufficient anything elseinsufficient anything else

Moving Down study execution – –serious flaws can lower by one level – –fatal flaws can lower by two levels consistency – –important inconsistency can lower by one level directness of evidence – –some uncertainty re relevance lower by one level – –major uncertainty re relevance lower by two levels selection bias – –strong evidence lower by 1 level

Moving Up very strong association, up 2 levels – –insulin in diabetic ketoacidosis strong, consistent association with no plausible confounders, up 2 levels – –fluoride for preventing cavities strong association can move up 1 level – –? HRT ?

Conclusion challenges in gradingchallenges in grading –balancing simplicity and complexity –judgement always required consistent grading system requiredconsistent grading system required must consider study design, execution, consistency, directnessmust consider study design, execution, consistency, directness –magnitude, precision, publication bias only when extreme

Questions for Discussion should Cochrane reviews provide:should Cochrane reviews provide: –grades of evidence across studies for all important outcomes? –overall grade of evidence across outcomes? –same grading system across reviews? bonus; while Cochrane reviews shouldn’t provide recommendations should they include structured discussion includingbonus; while Cochrane reviews shouldn’t provide recommendations should they include structured discussion including –trade-offs –translation of evidence in to practice