Applying the GRADE approach to diagnostic technologies Holger Schünemann for the GRADE working group

Today’s talk 1) Focus on patient important outcomes 2) Factors influencing the quality of evidence in diagnostic recommendations

GRADE for diagnosis Shares the fundamental logic of assessment for treatment However, assessments present unique challenges Examples and solutions for how to deal with challenges Focus on importance to patients and consumers!

Summary Quality of evidence reflects our confidence that estimates of benefits and downsides from a diagnostic strategy generated from research are correct. Consideration of the directness of evidence is based on how confident we are of the relation between being classified correctly (as a true positive or negative) or incorrectly (as a false positive or negative) and patient- important consequences. Recommendation depends on the balance between desirable and undesirable effects of the diagnostic test or strategy in terms of patient-important outcomes.

Testing makes a variety of contributions to patient care Clinicians use tests that are usually referred to as “diagnostic” signs and symptoms, imaging, biochemistry, pathology, and psychological Some tests naturally report positive and negative results (pregnancy) Other tests report their results in categories (e.g. imaging) Today we assume a diagnostic approach that ultimately categorizes test results as positive or negative

Purpose of a test Triage to minimize use of an invasive or expensive test Add-on to improve diagnosis beyond what is already done Replacement to replace test that is harmful or costly Bossuyt et al. BMJ 2006

What is the sensitivity, specificity (accuracy), or likelihood ratios of multislice spiral computed tomography (CT) of coronary arteries compared with conventional invasive angiography?

Test accuracy is a surrogate for patient important outcomes When clinicians think about diagnostic tests, they focus on their accuracy Underlying assumption: obtaining a better idea of whether a target condition is present or absent will result in superior patient management and improved outcome.

Test and treatment threshold Diagnostic tests are used in patients suspected of disease to exclude or confirm a diagnosis.

Purpose of the test  Identify the limitations for which the (new or) alternative test offers a putative remedy eliminating a high proportion of false positive or negative results, enhancing availability, decreasing invasiveness, or decreasing cost  Identification of sensible clinical questions: patients, diagnostic intervention, comparison, management, and outcomes of interest Bossuyt et al. BMJ 2006

Sensible clinical question Population: In patients suspected of coronary artery disease Intervention: does multislice spiral computed tomography (CT) of coronary arteries Comparison: compared with conventional invasive coronary angiography Outcomes: lower complications with acceptable rates of false negatives (associated with coronary events) and false positives (leading to unnecessary treatment and complications)?

Study designs for diagnosis If a test fails to improve important outcomes: no reason to use it, whatever its accuracy Best way to assess diagnostic strategy: randomized controlled trial in which investigators randomize patients to different diagnostic strategies

Study designs Focus on: mortality, morbidity, symptoms, and quality of life GRADE approach for treatment or intervention

Study designs II

Patient benefit required Inference from accuracy data that a diagnostic test or strategy improves patient-important outcome requires availability of effective management strategies Others: Reduction in test-related adverse effects Exclusion of a disease and reduction in anxiety Confirming a diagnosis improves patient well-being from the prognostic information it imparts

Patient benefit required Genetic testing for Huntington’s chorea reassurance that a patient will not suffer from the condition ability to plan for future knowing that patient will sadly fall victim Ability to plan is analogous to an effective treatment Almqvist et al Clin Gen 2003

Balance between presumed patient outcomes, complications and cost: Less complications and downsides compared to IVP would support the new test’s usefulness, but the balance between desirable and undesirable effect not clear in view of the uncertain consequences of identifying smaller stones.

Today’s talk 1) Focus on patient important outcomes 2) Factors influencing the quality of evidence in diagnostic recommendations

Factors that decrease the quality of evidence (and how they differ from treatment approach) Study design Different quality criteria for accuracy studies Valid accuracy studies: Diagnostic uncertainty Consecutive patients Evaluators should be blinded

Factors that decrease the quality of evidence Indirectness Population, Test and Comparison Outcomes Similar quality criteria Usual absence of direct evidence about impact on patient-important outcomes Accuracy studies typically provide low quality evidence

Factors that decrease the quality of evidence Inconsistency in study results Similar quality criteria & judgments but: other measures Imprecise evidence Similar quality criteria & judgments, but: Wide confidence intervals for estimates of test accuracy, true and false positives and negatives Reporting bias Similar quality criteria & judgments

Balance between presumed patient outcomes, complications and cost: Avoiding the undesirable consequences of more false positives with CT is preferable to avoiding the higher rate of complications (infarction and death) and higher cost with angiography

Evaluating tests – when can comparative evidence of test accuracy and other intermediate outcomes be used as an alternative to randomized trials (Lord, Irwig, Bossuyt)

Terminology Directness = PICO Synonyms: Generalizability, Applicability Directness of outcomes: Linkage assumptions Explicit assumptions about management of TP, TN, FP, FN, complications, indeterminates Treatment effects on: Mortality, morbidity, HRQL (including psychological well being) 25

Summary of findings

28 Outcomes Illustrative Risks (95% CI) Number of participants (studies) Quality of the Evidence 1 Comments Assumed outcome with CT – prevalence of 20% True positives (Patients correctly classified as having coronary artery disease) 192 per (21)   Moderate 2 Benefit from treatment and fewer complications.* Some patients will have to undergo angiography. True negatives (Patients correctly classified as not having coronary artery disease) 592 per (21)   Moderate 2 Benefit from reassurance and fewer complications False positives (Patients incorrectly classified as having coronary artery disease) 208 per (21)   Moderate 2 Harm from unnecessary treatment False negatives (Patients incorrectly classified as not having coronary artery disease) 8 per (21)   Low 2, 3 Detriment from delayed diagnosis or myocardial insult Complications (MI, allergic reactions, renal failure) 99 per (21)   Low 2 There is a higher rate of rare complications (infarction and death) and higher cost with angiography – a full profile would be required. Resource use* (cost of CT and Angiography) See comment Cost are higher for angiography, 1- Quality rated from 1 (very low quality) to 4 (high quality), 2- Cross sectional studies. Indirectness of outcomes in a wide spectrum of patients and indirect comparison of tests, 3– there is greater uncertainty whether these patients will have negative outcomes. *Assumed efficacy of: 1) aspirin daily = 20% RRR; 2) beta-blockage = 18% RRR.

Summary (1) Quality of evidence reflects our confidence that estimates of benefits and downsides from a diagnostic strategy generated from research are correct.

Summary (2) Consideration of the directness of evidence is based on how confident we are of the relation between being classified correctly (as a true positive or negative) or incorrectly (as a false positive or negative) and patient-important consequences.

Summary Recommendation depends on the balance between desirable and undesirable effects of the diagnostic test or strategy in terms of patient-important outcomes.

Thanks to GRADE group Contributors: Andrew D Oxman, Jan Brozek, Paul Glasziou, Roman Jaeschke, Gunn E Vist, Regina Kunz, John Williams, Jonathan Craig, Victor Montori, Patrick Bossuyt, Gordon H Guyatt

Eskerrik asko A USTEDES!

GRADE and diagnostic tests Diagnostic tests are also performed in patients already known to have the disease. Purpose may be to classify (e.g. stage or grade) or monitor the disease to guide treatment or inform prognosis. Being clear and specific about the purpose of a diagnostic test or strategy is crucial to making a proper recommendation. Diagnosis alone does not improve patient important outcomes.

Balance between presumed patient outcomes, complications and cost: Less complications and downsides compared to IVP would support the new test’s usefulness, but the balance between desirable and undesirable effect not clear in view of the uncertain consequences of identifying smaller stones.