Reference standard Diagnosis: the pathway of a diagnostic test From bench to bedside Reference standard Gennaro D’Amico UOC Gastroenterologia Ospedale V Cervello– Palermo gedamico@libero.it

Terminology In diagnosis research, the Reference Standard (RS) is the procedure (or test) that is used to define the true state of the patient (disease vs no disease) A major aim of diagnosis research is to find new diagnostic tests (Index Test) (IT) less invasive and expensive than the RS

The index test (IT) Disease Positive Test under hypothesis study Negative

The question underlying test accuracy research Are these patients truly Positive With disease Free of disease True Positive True Negative ? Test under study Negative To answer, a verification test is needed: the RS

The ideal (perfect) reference (gold) standard RS “An ideal RS, in an optimal diagnostic accuracy study, would fulfill the following criteria: The RS provides error-free classification of all subjects. (2) The same RS is used to verify all IT results. (3) The IT and RS can be performed within a short interval to avoid changes in target condition status″ Retisma JB. J Clin Epidemiol 2009; 62: 787-806

Verification of a new test accuracy Reference standard Positive Negative Disease No disease Positive Test under study true false Negative false true

Imperfect RS A perfect gold standard with 100% sensitivity and specificity is exceptional The clinical condition to which the IT is to be applied may hamper the application of the (same) RS in all the subjects The time interval from IT to RS may be too long A satisfactory RS may not be available For these reasons the term reference standard is preferred to gold standard

Problems related to imperfect RS Problem with RS Consequence / Bias Possible error / limitation sensitivity and specificity <100% Imprecise IT accuracy estimation Under- or over- estimation of accuracy RS is invasive and not performed in all negative IT Partial verification bias Over-estimation of sensitivity The RS is not independent of the IT and vice versa Diagnostic test bias Test review bias Incorporation bias Over-estimation of sensitivity, unclear effect on specificty Time interval between IT and RS is too long Verification bias Diagnosis may change along time Satisfactory RS is not available New definition needed, usually complex Reproducibility of IT accuracy Retisma JB. J Clin Epidemiol 2009; 62: 787-806 Whiting P. Ann Int Med 2004;140:189-202

Different RS diagnosis of depression IT: TRH stimulation test for depression (TSH < 7 µIU/ml post-TRH infusion of 500 p.g IV). Ten sensitivity studies used two different RSs, based on different validated questionnaires Diagnostic and Statistical Manual III criteria (DSM III), American Psychiatry Association (APA, 1980) Research Diagnostic Critera (RDC) (Spitzer, 1978) Arana GW. Biol Psych 1990;28:733-737

Different RSs Sensitivity of TRH-ST for depression 20 30 40 50 60 70 DSM RDC Sensitivity % Applying different standard procedures for different patients may yield inconsistent reference for the IT as each of the «standards» will have its own error rate

Inappropriate RS sensitivity of excercise scintigraphy for coronary disease Planar vs TC coronarography as RS Planar TC Detrano R. Arch Int Med 1988;148:1289-1295

Trade off Sensitivity/Specificity Courtesy of dr Mirella Fraquelli Trade off Sensitivity/Specificity Imperfect RS vs true disease status IT Broad histopathologic criteria to diagnose colon dysplasia Liver biopsy in diagnosing hepatic fibrosis Specificity Sensitivity Underestimates specificity Underestimates sensitivity

Verification bias RF is not performed in all subjects or different RSs are used Partial verification: RS performed on test-positives, but not on test-negatives Differential verification: RS used for test-positives is different from that used for test-negatives

Prospective investigation of Pulmonary Embolism Investigation PIOPED Diagnostic accuracy of ventilation perfusion scan was assessed by angiography Angio was more commonly done in patients with higher probability of PE based on VQ scan results Partial verification bias JAMA 1990;263:2753-59

Independent assessment of RS Diagnostic review bias Test under study Reference standard blind unblind test review bias RS + - + - FP TP Test under study FN TN

Incorporation bias The test that is being evaluated is included in the RS It can lead to overestimation of test accuracy It can occur if final diagnosis is made on the basis of all clinical data (which might include the IT) Examples: PCR for tuberculosis, Mantoux for TB among kids, screening for depression

Possible solutions for imperfect or missing RS Description limitation Composite RS Two or more tests combined by a prespecified rule Residual misclassification Expert panel All the relevant information per each patient, possibly including follow-up Disagreement Latent class analysis Statistical model providing probabilities of presence or absence of the diseas No clinical definition Validation Comparing IT with different items related to the disease or its severity Relevance of the considered items Delayed verification Short follow-up to obtain delayed cross-sectional information; RCT Disease status may change with time

Solving problems of imperfect RS Delayed-type cross-sectional study RS is invasive and may not be performed in all subjects or different standards have to be used The disease state of included subjects may be verified after a predefined follow-up Knotterus JA. J Clin Epidemiol 2003;56:118-1128

Prospective investigation of Pulmonary Embolism Investigation PIOPED Diagnostic accuracy of ventilation perfusion scan was assessed by angiography Angio was more commonly done in patients with higher probability of PE based on VQ scan results 1-year follow-up confirmed the diagnosis in subjects not undergoing angio JAMA 1990;263:2753-59

Index test in patients with suspected PE Verification Performance of the Wells score in patients with suspected pulmonary embolism during hospitalization: A delayed-type cross sectional study in a community hospital Index test in patients with suspected PE Verification Angio TC V/Q pulmonary scinti-scan Angiography Leg venous US showing DVT Wells score >4 PE likely ≤ 4 PE unlikely Posadas-Martines ML Thromb Res 2014;133:177–181

Performance of the Wells score in patients with suspected pulmonary embolism during hospitalization: A delayed-type cross sectional study in a community hospital 613 pts Suspected PE No PE 90 days followup +1 PE Posadas-Martines ML Thromb Res 2014;133:177–181

Validation of IT in the absence of satisfactory RS Interferon gamma levels for latent tuberculosis The better test would be the one more strongly correlated with exposure E= IFN-Ỿ T= tubercolin skin test Ewer K Lancet 2003;361:1168-73

Conclusions The RS is the diagnostic instrument used to verify the accuracy of the results of a new test The same RS should independently, or even blindly,verify all the new test results and should be correctly performed Since the RS is almost always imperfect it should be appropriate for the new test under study Imperfect RS or inappropriate use of satisfactory RS may leed to incorrect conclusions on the IT accuracy Composite RS, expert panel judgement, cross-sectional delayed verification and other validation methods, may help to overcome the lack of a satisfactory RS

Independent assessment of RS Diagnostic review bias Test under study Reference standard unblind blind test review bias RS + - TP FP + - Test under study FN TN