1. Reference standard Diagnosis: the pathway of a diagnostic test
From bench to bedside
Reference standard
Gennaro D’Amico
UOC Gastroenterologia
Ospedale V Cervello– Palermo

2. Terminology
In diagnosis research, the Reference Standard (RS) is the procedure (or test) that is used to define the true state of the patient (disease vs no disease)
A major aim of diagnosis research is to find new diagnostic tests (Index Test) (IT) less invasive and expensive than the RS

3. The index test (IT) Disease Positive Test under hypothesis study
Negative

4. The question underlying test accuracy research
Are these patients truly
Positive
With disease
Free of disease
True Positive
True Negative ?
Test under
study
Negative
To answer, a verification test is needed: the RS

5. The ideal (perfect) reference (gold) standard RS
“An ideal RS, in an optimal diagnostic accuracy study, would fulfill the following criteria:
The RS provides error-free classification of all subjects.
(2) The same RS is used to verify all IT results.
(3) The IT and RS can be performed within a short interval to avoid changes in target condition status″
Retisma JB. J Clin Epidemiol 2009; 62:

6. Verification of a new test accuracy
Reference standard
Positive
Negative
Disease
No disease
Positive
Test under
study
true
false
Negative
false
true

7. Imperfect RS
A perfect gold standard with 100% sensitivity and specificity is exceptional
The clinical condition to which the IT is to be applied may hamper the application of the (same) RS in all the subjects
The time interval from IT to RS may be too long
A satisfactory RS may not be available
For these reasons the term reference standard is preferred to gold standard

8. Problems related to imperfect RS
Problem with RS
Consequence / Bias
Possible error / limitation
sensitivity and specificity <100%
Imprecise IT accuracy estimation
Under- or over- estimation of accuracy
RS is invasive and not performed in all negative IT
Partial verification bias
Over-estimation of sensitivity
The RS is not independent of the IT and vice versa
Diagnostic test bias
Test review bias
Incorporation bias
Over-estimation of sensitivity, unclear effect on specificty
Time interval between IT and RS is too long
Verification bias
Diagnosis may change along time
Satisfactory RS is not available
New definition needed, usually complex
Reproducibility of IT accuracy
Retisma JB. J Clin Epidemiol 2009; 62:
Whiting P. Ann Int Med 2004;140:

9. Different RS diagnosis of depression
IT: TRH stimulation test for depression (TSH < 7 µIU/ml post-TRH infusion of 500 p.g IV).
Ten sensitivity studies used two different RSs, based on different validated questionnaires
Diagnostic and Statistical Manual III criteria (DSM III), American Psychiatry Association (APA, 1980)
Research Diagnostic Critera (RDC) (Spitzer, 1978)
Arana GW. Biol Psych 1990;28:

10. Different RSs Sensitivity of TRH-ST for depression20 30 40 50 60 70
DSM
RDC
Sensitivity %
Applying different standard
procedures for different
patients may yield
inconsistent reference
for the IT as each of
the «standards» will have
its own error rate

11. Inappropriate RS sensitivity of excercise scintigraphy for coronary disease
Planar vs TC coronarography as RS
Planar TC
Detrano R. Arch Int Med 1988;148:

12. Trade off Sensitivity/Specificity
Courtesy of dr Mirella Fraquelli
Trade off Sensitivity/Specificity
Imperfect RS
vs true disease
status IT
Broad histopathologic
criteria to diagnose colon dysplasia
Liver biopsy in diagnosing hepatic fibrosis
Specificity
Sensitivity
Underestimates specificity
Underestimates
sensitivity

13. Verification bias RF is not performed in all subjects or different RSs are used
Partial verification:
RS performed on test-positives, but not on test-negatives
Differential verification:
RS used for test-positives is different from that used for test-negatives

14. Prospective investigation of Pulmonary Embolism Investigation PIOPED
Diagnostic accuracy of ventilation perfusion scan was assessed by angiography
Angio was more commonly done in patients with higher probability of PE based on VQ scan results
Partial verification bias
JAMA 1990;263:

15. Independent assessment of RS
Diagnostic review
bias
Test
under study
Reference
standard
blind
unblind
test review
bias RS + - FP TP
Test
under study FN TN

16. Incorporation bias
The test that is being evaluated is included in the RS
It can lead to overestimation of test accuracy
It can occur if final diagnosis is made on the basis of all clinical data (which might include the IT)
Examples: PCR for tuberculosis, Mantoux for TB among kids, screening for depression

17. Possible solutions for imperfect or missing RS
Description
limitation
Composite RS
Two or more tests combined by a prespecified rule
Residual misclassification
Expert panel
All the relevant information per each patient, possibly including follow-up
Disagreement
Latent class analysis
Statistical model providing probabilities of presence or absence of the diseas
No clinical definition
Validation
Comparing IT with different items related to the disease or its severity
Relevance of the considered items
Delayed verification
Short follow-up to obtain delayed cross-sectional information; RCT
Disease status may change with time

18. Solving problems of imperfect RS Delayed-type cross-sectional study
RS is invasive and may not be performed in all subjects or different standards have to be used
The disease state of included subjects may be verified after a predefined follow-up
Knotterus JA. J Clin Epidemiol 2003;56:

19. Prospective investigation of Pulmonary Embolism Investigation PIOPED
Diagnostic accuracy of ventilation perfusion scan was assessed by angiography
Angio was more commonly done in patients with higher probability of PE based on VQ scan results
1-year follow-up confirmed the diagnosis in subjects not undergoing angio
JAMA 1990;263:

20. Index test in patients with suspected PE Verification
Performance of the Wells score in patients with suspected pulmonary embolism during hospitalization: A delayed-type cross sectional study in a community hospital
Index test in patients with suspected PE
Verification
Angio TC
V/Q pulmonary scinti-scan
Angiography
Leg venous US showing DVT
Wells score
>4 PE likely
≤ 4 PE unlikely
Posadas-Martines ML Thromb Res 2014;133:177–181

21. Performance of the Wells score in patients with suspected pulmonary embolism during hospitalization: A delayed-type cross sectional study in a community hospital
613 pts
Suspected PE
No PE
90 days followup
+1 PE
Posadas-Martines ML Thromb Res 2014;133:177–181

22. Validation of IT in the absence of satisfactory RS Interferon gamma levels for latent tuberculosis
The better test
would be the one
more strongly
correlated with
exposure
E= IFN-Ỿ
T= tubercolin
skin test
Ewer K Lancet 2003;361:

23. Conclusions
The RS is the diagnostic instrument used to verify the accuracy of the results of a new test
The same RS should independently, or even blindly,verify all the new test results and should be correctly performed
Since the RS is almost always imperfect it should be appropriate for the new test under study
Imperfect RS or inappropriate use of satisfactory RS may leed to incorrect conclusions on the IT accuracy
Composite RS, expert panel judgement, cross-sectional delayed verification and other validation methods, may help to overcome the lack of a satisfactory RS

24. Independent assessment of RS
Diagnostic review
bias
Test
under study
Reference
standard
unblind
blind
test review
bias RS TP FP + -
Test
under study FN TN