1 Lecture 2 Screening and diagnostic tests Normal and abnormal Validity: “gold” or criterion standard Sensitivity, specificity, predictive value Likelihood ratio ROC curves Bias: spectrum, verification, information

2 Clinical/public health applications screening: for asymptomatic disease (e.g., Pap test, mammography) case-finding: testing of patients for diseases unrelated to their complaint diagnostic: to help make diagnosis in symptomatic disease or to follow-up on screening test

3 Evaluation of screening and diagnostic tests Performance characteristics –test alone Effectiveness (on outcomes of disease): –test + intervention

4 Criteria for test selection Reproducibility Validity Feasibility Simplicity Cost Acceptability

5 Sources of variation: Biological or true variation between individuals within individuals (e.g., diurnal variation in BP) –“controlled” by standardizing time of measurement

6 Sources of variation: Measurement error random error vs systematic error (bias) method (measuring instrument) observer

7

8 Quality of measurements Validity (accuracy) –Does it measure what it is intended to? –Lack of bias Reproducibility (reliability, precision, consistency) of measurements

9 Examples of types of reproducibility Between and within observer (inter- and intra-observer variation) –May be random or systematic Regression toward the mean –Systematic error when subjects have extreme values (more likely to be in error than typical values)

10 Validity (accuracy) Criterion validity –concurrent –predictive Face validity, content validity: judgement of the appropriateness of content of measurement Construct validity: validity of underlying entity or theoretical construct

11 Normal vs abnormal Statistical definition –“Gaussian” or “normal” distribution Clinical definition –using criterion

12

13

14

15

16 Selection of criterion Concurrent –salivary screening test for HIV –history of cough more than 2 weeks (for TB) Predictive –APACHE (acute physiology and chronic disease evaluation) instrument for ICU patients –blood lipid level –maternal height

17

18 Sensitivity and specificity Assess correct classification of: People with the disease (sensitivity) People without the disease (specificity)

19 Predictive value More relevant to clinicians and patients Affected by prevalence

20 Choice of cut-point If higher score increases probability of disease Lower cut-point: –increases sensitivity, reduces specificity Higher cut-point: –reduces sensitivity, increases specificity

21 Considerations in selection of cut-point Implications of false positive results burden on follow-up services labelling effect Implications of false negative results Failure to intervene

22 Likelihood ratio Likelihood ratio (LR) = sensitivity 1-specificity Used to compute post-test odds of disease from pre-test odds: post-test odds = pre-test odds x LR pre-test odds derived from prevalence post-test odds can be converted to predictive value of positive test

23 Example of LR prevalence of disease in a population is 25% sensitivity is 80% specificity is 90%, pre-test odds = 0.25 = 1/ likelihood ratio = 0.80 =

24 Example of LR If prevalence of disease in a population is 25% pre-test odds = 0.25 = 1/ post-test odds = 1/3 x 8 = 8/3 predictive value of positive result = 8/3+8 = 8/11 = 73%

25 Receiver operating characteristic (ROC) curve Evaluates test over range of cut-points Plot of sensitivity against 1-specificity Area under curve (AUC) summarizes performance: –AUC of 0.5 = no better than chance

26

27 Spectrum bias Study population should be representative of population in which test will be used Is range of subjects tested adequate? –In population with low risk of outcome, sensitivity will be lower, specificity higher –In population with high risk of outcome, sensitivity will be higher, specificity lower Comorbidity may affect sensitivity and specificity

28 Verification bias results of test affect intensity of subsequent investigation increasing probability of detection of outcome in those with positive test result

29 Information bias Diagnosis is not blind to test result Improves test performance

30 Example: Screening seniors in the emergency department (ED) for risk of function decline High risk group Many not adequately evaluated or referred for appropriate services Development and validation of a brief screening tool to identify those at increased risk of functional decline and other adverse outcomes

31 Two multi-site studies in Montreal EDs Study 1: development of ISAR –Prospective observational cohort study –JAGS (1999) 47: Study 2: evaluation of 2-step intervention –randomized controlled trial –JAGS (2001) 49:

32 Common features of 2 studies 4 Montreal hospitals (2 participated in both studies) Patients aged 65+, community dwelling, English or French-speaking Exclusions: –cognitively impaired or severe illness with no proxy informant – language barrier (no English or French)

33 Differences between 2 studies: Study design Study 1 –Observational study –Follow-up at 3 and 6 months after ED visit Study 2 –Randomized controlled trial: 2-step intervention vs usual care –Randomization by day of visit –Follow-up at 1 and 4 months after ED visit

34 RESULTS: ISAR development Adverse health outcome defined as any of following during 6 months after ED visit >10% ADL decline Death Institutionalization

35 Scale development Selection of items that predicted all adverse health events Multiple logistic regression - “best subsets” analysis Review of candidate scales with clinicians to select clinically relevant scale

36 Identification of Seniors At Risk (ISAR) 1. Before the illness or injury that brought you to the Emergency, did you need someone to help you on a regular basis? (yes) 2. Since the illness or injury that brought you to the Emergency, have you needed more help than usual to take care of yourself? (yes) 3. Have you been hospitalized for one or more nights during the past 6 months (excluding a stay in the Emergency Department)? (yes) 4. In general, do you see well? (no) 5. In general, do you have serious problems with your memory? (yes) 6. Do you take more than three different medications every day? (yes) Scoring: (positive score shown in parentheses)

37

38 Other Outcomes Related to ISAR Source: Dendukuri et al, JAGS, in press Does ISAR score identify patients with current functional problems? –Self-reported premorbid function (OARS) –Function at home visit assessed by nurse 1-2 weeks after ED visit (SMAF)

39

40 Other Outcomes Related to ISAR Does ISAR predict adverse outcomes (other than functional decline) during the subsequent 5 or 6 months? –High hospital utilization (11+ days/5 months) –Frequent ED visits –Frequent community health center visits –Increase in depressive symptoms

41

42 Summary of data on performance Very good detection of patients with current functional problems and depression (AUC values ) Moderate ability to predict future adverse health events (functional decline) and health center utilization (AUC values around 0.7) Fair ability to predict future hospital and ED utilization (AUC values )

43 Comparison with other screening tools for patients admitted to hospital Source: McCusker et al, J Gerontol 2002; 57A: M Systematic literature review Predictors of functional decline (including nursing home admission) among hospitalized seniors Investigated individual risk factors and predictive indices

44 Predictive indices Inouye (1993): FD and NH at 3 mo –4 factors: decubitus ulcer, cognitive impairment, premorbid functional impairment, low social activity Mateev(1998): D/NH at 3 mo. –clinical targeting criteria

45 Predictive indices (cont) McCusker (1999): FD/NH/ D at 6 mo. –Identification of Seniors At Risk (ISAR): 6- item self-report questionnaire Narain (1988): NH at 6 mo –hand-developed algorithm based on residence, mental status, diagnosis

46 Predictive indices (cont) Rubenstein (1984): FD and NH at 12 mo –expected discharge location and diagnosis Sager (1996): FD at 3mo –Hospital Admission Risk Profile (HARP) (age, MMSE and IADL) Zureik (1997): NH at discharge –6-item index

47 Performance of 7 predictive indices for functional decline

48 Performance of predictive indices Moderate performance (AUC )