1. Applying the GRADE approach to diagnostic technologies Holger Schünemann for the GRADE working group

2. Today’s talk 1) Focus on patient important outcomes 2) Factors influencing the quality of evidence in diagnostic recommendations

3. GRADE for diagnosis Shares the fundamental logic of assessment for treatment However, assessments present unique challenges Examples and solutions for how to deal with challenges Focus on importance to patients and consumers!

4. Summary Quality of evidence reflects our confidence that estimates of benefits and downsides from a diagnostic strategy generated from research are correct. Consideration of the directness of evidence is based on how confident we are of the relation between being classified correctly (as a true positive or negative) or incorrectly (as a false positive or negative) and patient- important consequences. Recommendation depends on the balance between desirable and undesirable effects of the diagnostic test or strategy in terms of patient-important outcomes.

5. Testing makes a variety of contributions to patient care Clinicians use tests that are usually referred to as “diagnostic” signs and symptoms, imaging, biochemistry, pathology, and psychological Some tests naturally report positive and negative results (pregnancy) Other tests report their results in categories (e.g. imaging) Today we assume a diagnostic approach that ultimately categorizes test results as positive or negative

6. Purpose of a test Triage to minimize use of an invasive or expensive test Add-on to improve diagnosis beyond what is already done Replacement to replace test that is harmful or costly Bossuyt et al. BMJ 2006

7. What is the sensitivity, specificity (accuracy), or likelihood ratios of multislice spiral computed tomography (CT) of coronary arteries compared with conventional invasive angiography?

8. Test accuracy is a surrogate for patient important outcomes When clinicians think about diagnostic tests, they focus on their accuracy Underlying assumption: obtaining a better idea of whether a target condition is present or absent will result in superior patient management and improved outcome.

9. Test and treatment threshold Diagnostic tests are used in patients suspected of disease to exclude or confirm a diagnosis.

10. Purpose of the test  Identify the limitations for which the (new or) alternative test offers a putative remedy eliminating a high proportion of false positive or negative results, enhancing availability, decreasing invasiveness, or decreasing cost  Identification of sensible clinical questions: patients, diagnostic intervention, comparison, management, and outcomes of interest Bossuyt et al. BMJ 2006

12. Sensible clinical question Population: In patients suspected of coronary artery disease Intervention: does multislice spiral computed tomography (CT) of coronary arteries Comparison: compared with conventional invasive coronary angiography Outcomes: lower complications with acceptable rates of false negatives (associated with coronary events) and false positives (leading to unnecessary treatment and complications)?

13. Study designs for diagnosis If a test fails to improve important outcomes: no reason to use it, whatever its accuracy Best way to assess diagnostic strategy: randomized controlled trial in which investigators randomize patients to different diagnostic strategies

14. Study designs Focus on: mortality, morbidity, symptoms, and quality of life GRADE approach for treatment or intervention

15. Study designs II

16. Patient benefit required Inference from accuracy data that a diagnostic test or strategy improves patient-important outcome requires availability of effective management strategies Others: Reduction in test-related adverse effects Exclusion of a disease and reduction in anxiety Confirming a diagnosis improves patient well-being from the prognostic information it imparts

17. Patient benefit required Genetic testing for Huntington’s chorea reassurance that a patient will not suffer from the condition ability to plan for future knowing that patient will sadly fall victim Ability to plan is analogous to an effective treatment Almqvist et al Clin Gen 2003

18. Balance between presumed patient outcomes, complications and cost: Less complications and downsides compared to IVP would support the new test’s usefulness, but the balance between desirable and undesirable effect not clear in view of the uncertain consequences of identifying smaller stones.

19. Today’s talk 1) Focus on patient important outcomes 2) Factors influencing the quality of evidence in diagnostic recommendations

20. Factors that decrease the quality of evidence (and how they differ from treatment approach) Study design Different quality criteria for accuracy studies Valid accuracy studies: Diagnostic uncertainty Consecutive patients Evaluators should be blinded

21. Factors that decrease the quality of evidence Indirectness Population, Test and Comparison Outcomes Similar quality criteria Usual absence of direct evidence about impact on patient-important outcomes Accuracy studies typically provide low quality evidence

22. Factors that decrease the quality of evidence Inconsistency in study results Similar quality criteria & judgments but: other measures Imprecise evidence Similar quality criteria & judgments, but: Wide confidence intervals for estimates of test accuracy, true and false positives and negatives Reporting bias Similar quality criteria & judgments

23. Balance between presumed patient outcomes, complications and cost: Avoiding the undesirable consequences of more false positives with CT is preferable to avoiding the higher rate of complications (infarction and death) and higher cost with angiography

24. Evaluating tests – when can comparative evidence of test accuracy and other intermediate outcomes be used as an alternative to randomized trials (Lord, Irwig, Bossuyt)

25. Terminology Directness = PICO Synonyms: Generalizability, Applicability Directness of outcomes: Linkage assumptions Explicit assumptions about management of TP, TN, FP, FN, complications, indeterminates Treatment effects on: Mortality, morbidity, HRQL (including psychological well being) 25

27. Summary of findings

28. 28 Outcomes Illustrative Risks (95% CI) Number of participants (studies) Quality of the Evidence 1 Comments Assumed outcome with CT – prevalence of 20% True positives (Patients correctly classified as having coronary artery disease) 192 per 1000 1570 (21)   Moderate 2 Benefit from treatment and fewer complications.* Some patients will have to undergo angiography. True negatives (Patients correctly classified as not having coronary artery disease) 592 per 1000 1570 (21)   Moderate 2 Benefit from reassurance and fewer complications False positives (Patients incorrectly classified as having coronary artery disease) 208 per 1000 1570 (21)   Moderate 2 Harm from unnecessary treatment False negatives (Patients incorrectly classified as not having coronary artery disease) 8 per 1000 1570 (21)   Low 2, 3 Detriment from delayed diagnosis or myocardial insult Complications (MI, allergic reactions, renal failure) 99 per 1000 1570 (21)   Low 2 There is a higher rate of rare complications (infarction and death) and higher cost with angiography – a full profile would be required. Resource use* (cost of CT and Angiography) See comment Cost are higher for angiography, 1- Quality rated from 1 (very low quality) to 4 (high quality), 2- Cross sectional studies. Indirectness of outcomes in a wide spectrum of patients and indirect comparison of tests, 3– there is greater uncertainty whether these patients will have negative outcomes. *Assumed efficacy of: 1) aspirin daily = 20% RRR; 2) beta-blockage = 18% RRR.

29. Summary (1) Quality of evidence reflects our confidence that estimates of benefits and downsides from a diagnostic strategy generated from research are correct.

30. Summary (2) Consideration of the directness of evidence is based on how confident we are of the relation between being classified correctly (as a true positive or negative) or incorrectly (as a false positive or negative) and patient-important consequences.

31. Summary Recommendation depends on the balance between desirable and undesirable effects of the diagnostic test or strategy in terms of patient-important outcomes.

32. Thanks to GRADE group Contributors: Andrew D Oxman, Jan Brozek, Paul Glasziou, Roman Jaeschke, Gunn E Vist, Regina Kunz, John Williams, Jonathan Craig, Victor Montori, Patrick Bossuyt, Gordon H Guyatt

33. Eskerrik asko A USTEDES!

34. GRADE and diagnostic tests Diagnostic tests are also performed in patients already known to have the disease. Purpose may be to classify (e.g. stage or grade) or monitor the disease to guide treatment or inform prognosis. Being clear and specific about the purpose of a diagnostic test or strategy is crucial to making a proper recommendation. Diagnosis alone does not improve patient important outcomes.

35. Balance between presumed patient outcomes, complications and cost: Less complications and downsides compared to IVP would support the new test’s usefulness, but the balance between desirable and undesirable effect not clear in view of the uncertain consequences of identifying smaller stones.