1. Screening

2. Outline
Screening basics
Evaluation of screening programs

3. Where are we going today?
Definition of screening?
Whether it is always beneficial?
Types of bias in screening?
Principles for the development of screening.
The test: Validity, LR, Multiple testing, ROC curve, Kappa
The disease;
Evaluation of a screening program

4. Where are we? Definition of screening?
Whether it is always beneficial?
Types of bias in screening?
Principles for the development of screening.
The test: Validity, LR, Multiple testing, ROC curve, Kappa
The disease;
Evaluation of a screening program

5. Screening Basics What does “screening” mean?
What do we screen for (objective)?
What makes a disease an appropriate target for screening?
What makes a test a good screening test?

6. Levels of Prevention (Mausner and Kramer 1985)
Primary Prevention - Prevention of the occurrence of disease (reduce incidence of disease)
Secondary Prevention - Early detection and prompt treatment of disease for cure, to slow progression, to prevent complications, or to limit disability (reduce prevalence of disease)
Tertiary Prevention - Limitation of disability and rehabilitation where disease has already occurred and left residual damage

7. Natural History of Disease

8. Pre-clinical Phase The Pre-Clinical Phase (PCP) is
the period between when early detection by screening is possible and when the clinical diagnosis would usually be made.
Pathology begins
Disease detectable
Normal Clinical Presentation
Pre-Clinical Phase

9. Objective and Definition
Objective: to reduce mortality and/or morbidity by early detection and treatment.
Definition: Asymptomatic individuals are classified as either unlikely or possibly having disease.

11. Trisha Greenhalgh, BMJ 1997;315:540-543 (30 August)

12. Where are we? Definition of screening?
Whether it is always beneficial?
Types of bias in screening?
Principles for the development of screening.
The test: Validity, LR, Multiple testing, ROC curve, Kappa
The disease;
Evaluation of a screening program

13. Principles for the development of screening;
The condition screened for is an important cause of morbidity, disability, or mortality.
The natural history of the disease is sufficiently well known.
The test must have high levels performance.
The test must be acceptable to the target population and their health care providers, and appropriate follow-up of positive findings must be ensured.
Sam Shapiro in Epidemiology and Health Services

14. Consequence of a screening test:
Beneficence
Non-beneficence
Do harm;
Clofibrate in US
Labeling effect; Social psychology

15. Biases in assessing efficacy of screening
Two major biases affect these data:
lead time bias
length bias

16. Where are we? Definition of screening?
Whether it is always beneficial?
Types of bias in screening?
Principles for the development of screening.
The test: Validity, LR, Multiple testing, ROC curve, Kappa
The disease;
Evaluation of a screening program

17. Lead Time Lead time = amount of time by which diagnosis is advanced
or made earlier
Pathology begins
Disease detectable
Normal Clinical Presentation
Lead Time
Screen

18. Lead time bias We think early detection has increased survival
in fact all it has done is increase the time the patient is aware of his disease!
treatment could even hasten death and it might appear survival is longer post diagnosis!!
Cannot just look at survival time post diagnosis.

19. Lead-time Bias

20. Length bias
Survival due to screening and treatment may be over rated because screening will tend to discover more slow-growing disease.

21. Length-time Bias

22. Suppose there are two subtypes of the disease:
Type 1: fast progression
Biologic onset
Usual time of diagno-sis
Severe clinical illness (eg metasta-ses)
Death from the disease
First detect-able by screen-ing test
Type 2: slow progression
Biologic onset
Usual time of diagnosis
Severe clinical illness (eg metastases)
Death from the disease
First detectable by screening test

23. Length of time in pre-clinical phase longer in Type 2 than in Type 1
Biologic onset
Usual time of diagno-sis
Severe clinical illness (eg metasta-ses)
Death from the disease
First detect-able by screen-ing test
Type 2
Biologic onset
Usual time of diagnosis
Severe clinical illness (eg metastases)
Death from the disease
First detectable by screening test

24. Periodic screening will tend to detect more of Type 2, as these have longer “exposure” in the critical interval for screening.
Type 1
Biologic onset
Usual time of diagno-sis
Severe clinical illness (eg metasta-ses)
Death from the disease
First detect-able by screen-ing test
Type 2
Biologic onset
Usual time of diagnosis
Severe clinical illness (eg metastases)
Death from the disease
First detectable by screening test

25. But look!! Type 2 individuals have a longer survival time from time of diagnosis than do Type 1.
Biologic onset
Usual time of diagno-sis
Severe clinical illness (eg metasta-ses)
Death from the disease
First detect-able by screen-ing test
Type 2
Biologic onset
Usual time of diagnosis
Severe clinical illness (eg metastases)
Death from the disease
First detectable by screening test

26. Length bias
Without screening, suppose type 1 and type 2 were equal fractions of the population
average survival time is 50:50 mixture of the short and long survival times.
With screening, the screen-detected population has a higher fraction of type 2 (slow) individuals
mix will be proportional to ratio of the two intervals
suppose it is 70:30 in favor of long interval
average survival time will be longer in screen detected individuals!

27. Length bias
Even if the treatment tended to be harmful and shorten life, because more longer interval individuals tend to be detected by screening, the screening program will appear to be effective!!

29. Where are we? Definition of screening?
Whether it is always beneficial?
Types of bias in screening?
Principles for the development of screening.
The test: Validity, LR, Multiple testing, ROC curve, Kappa
The disease;
Evaluation of a screening program

30. Principles for the development of screening;
The test must have high levels performance.
The condition screened for is an important cause of morbidity, disability, or mortality.
The natural history of the disease is sufficiently well known.
The test must be acceptable to the target population and their health care providers, and appropriate follow-up of positive findings must be ensured.
Sam Shapiro in Epidemiology and Health Services

31. Characteristics of Test
Safety
Cost
Acceptability
Validity
Reliability

32. Characteristics of Test
Validity of test is shown by how well the test actually measures what it is supposed to measure. Validity is determined by the sensitivity and specificity of the test.
Reliability is based on how well the test does in use over time - in its repeatability.

33. Characteristics of Test: Validity
Sensitivity is the ability of a screening procedure to correctly identify those who have the disease-- the proportion of persons with the disease who have a positive test result
Specificity is the ability of a screening procedure to correctly identify the percentage of those who do not have the disease--the proportion of persons without the disease who have a negative test result

34. Sensitivity and Specificity
Disease
Present
Absent
True positive A
False positive B
False negative C
True negative D
Positive
Test
Result
Negative
Sensitivity = A / (A+C)
Specificity = D / (B+D)

35. Sensitivity and Specificity of Breast Cancer Screening Examination
Present
Absent
True positive
A = 132
False positive
B = 983
False negative
C = 45
True negative
D = 63650
Positive
Screening
Test
Negative
Sensitivity=132/(132+45)=74.6%
Specificity=63650/( )=98.5%

36. Predictive Values
Predictive Value Positive – Probability that a person actually has the disease given a positive screening test
Predictive Value Negative – Probability that a person is actually disease-free given a negative screening test

37. Predictive Values Disease Test Result Present Absent True positive A
False positive B
False negative C
True negative D
Positive
Test
Result
Negative
Sensitivity = A / (A+C)
Specificity = D / (B+D)
PPV = A / (A+B)
NPV = D / (C+D)

38. Predictive Values of Breast Cancer Screening Examination
Present
Absent
True positive
A = 132
False positive
B = 983
False negative
C = 45
True negative
D = 63650
Positive
Screening
Test
Negative
Sensitivity=132/(132+45)=74.6%
Specificity=63650/( )=98.5%
PPV=132/( )=11.8%
NPV=63650/( )=99.9%

39. Prevalence PV+ (%) Sensitivity Specificity
Effect of Prevalence on Predictive Value Positive with Constant Sensitivity and Specificity
Prevalence PV+ (%) Sensitivity Specificity
(%) (%) (%)

40. Test-treatment threshold
Do not test
Get on the treatment
Test and treat,
based on the
basis of the result
Do not test
Do not treat
Pretest Probability of disease

41. Bayesian Approach (probabilities)
Prior idea
of effect
Study Results:
effect size
Final Conclusion:
Study + prior effect
Pretest
Probability
of disease
Test
Result
Posttest
Probability
of disease

42. Clinical Reasoning (probabilities)
Sensitive test
Specific test
Before
mammogram
After
mammogram
After
FNA
0.3 13 64
Probability of Breast Ca (%)

43. Likelihood ratio of a positive test
How much more likely is a positive test to be found in a person with the condition than in a person without it?
sensitivity/ (l-specificity)

44. Likelihood Ratios Se= a a + c Sp= d b + d LR+=1 = no value
Disease
Present
Absent
Test
Positive a b
Negative c d
Se= a
a + c
Sp= d
b + d
LR+=1 = no value
LR+ = [a/a+c]
[b/b+d] probability of + test in non-diseased
= probability of + test in diseased

45. Likelihood Ratios Se= a a + c Sp= d b + d LR+>1 = valuable
Disease
Present
Absent
Test
Positive a b
Negative c d
Se= a
a + c
Sp= d
b + d
LR+>1 = valuable
LR+=1 = no value
LR+ = [a/a+c]
[b/b+d] probability of + test in non-diseased
= probability of + test in diseased

46. Bayesian Approach Prior idea of effect Study Results: effect size
Final Conclusion:
Study + prior effect
Pretest
Probability
of disease
Test
Result
Posttest
Probability
of disease
Pretest
Prob.
of disease
Posttest
Prob.
of disease X
LR+ =

48. Where are we? Definition of screening?
Whether it is always beneficial?
Types of bias in screening?
Principles for the development of screening.
The test: Validity, LR, Multiple testing, ROC curve, Kappa
The disease;
Evaluation of a screening program

49. Influence of prevalence on predictive values

50. Multiple testing
Serial (Sequential)
Simultaneous (Parallel)

51. Multiple Tests Sequential
First perform diagnostic screen with less expensive/intrusive test
If positive, use more accurate test
First test should be sensitive, but specificity less important – why?
Procedure makes false positives less problematic – more specific – why?

52. Sequential Screening Example
Text example - Diabetes
1st stage screening, blood sugar
Easy, quick, inexpensive, non-intrusive
2nd stage screening, glucose tolerance
Greater time investment, more expensive
Disease prevalence = 5%
Sample size = 10,000

53. 1st Stage Screening Present Absent Positive True (+) 350 False (+)
Diabetes
Present
Absent
Blood
Sugar
Positive
True (+)
350
False (+)
1900
Negative
False (-)
150
True (-)
7600

54. Results Sensitivity = True positives/all with disease = 350/(350+150)
=70.0%
Specificity = True negatives/all without disease
=7600/( )
=80.0%

55. Next Step Total with diabetes = 500 Total without diabetes = 9500
All those with positive tests are brought back for second stage screening
i.e., true positives plus false positives =
2250 people brought back for second stage screening

56. Again, 1st Stage Screening
Diabetes
Present
N=500
Absent
N=9500
Blood
Sugar
Positive
N=2250
True (+)
350
False (+)
1900
Negative
N=7750
False (-)
150
True (-)
7600

57. 2nd Stage Screening Present N=350 Absent N=1900 Positive N=505
Diabetes
Present
N=350
Absent
N=1900
Glucose
Tolerance
Positive
N=505
True (+)
315
False (+)
190
Negative
N=1745
False (-) 35
True (-)
1710

58. Results
Net Sensitivity = Number correctly identified as diabetic (2nd screening)/total with diabetes
= 315/500
=63%
*Note decrease in sensitivity
Net Specificity = Number correctly identified as not diabetic (1st screening + 2nd screening)/all without diabetes
=( )/9500
=98%
*Note increase in specificity

59. Simultaneous Testing
Multiple tests for one disease administered at same time
“positive” test result – positive result any of the tests
“negative” test result – negative results on all tests
Sensitivity?
Specificity?

60. ROC Curves
(PD- x CFP) / PD- x CFN)

61. Posterior probability
Comparing two tests
Posterior probability
Prior probability

62. Spectrum of severity

63. Where are we? Definition of screening?
Whether it is always beneficial?
Types of bias in screening?
Principles for the development of screening.
The test: Validity, LR, Multiple testing, ROC curve, Kappa
The disease;
Evaluation of a screening program

64. Reliability

65. Percent agreement

66. Cohen’s Kappa Reported in 1960
Kappa corrects for the chance agreement that would be expected to occur if the 2 classifications were completely unrelated

67. K = N° - Ne 1 - Ne Kappa N° = Observed number of agreement
Ne= Number of agreement expected to occur by chance alone
Varies from -1 to 1

68. Population One (Prevalence = 0.05) Table for true positives
Observer B
From Szklo and Nieto, 2000

69. Interpretation of Kappa

70. Interpretation of Kappa
Below  Poor
 Slight
 Fair
 Moderate
 Substantial
 Almost Perfect
Landis & Koch (1977a)

71. Where are we?

72. Principles for the development of screening;
The test must have high levels performance.
The condition screened for is an important cause of morbidity, disability, or mortality.
The natural history of the disease is sufficiently well known.
The test must be acceptable to the target population and their health care providers, and appropriate follow-up of positive findings must be ensured.
Sam Shapiro in Epidemiology and Health Services

73. Where are we? Definition of screening?
Whether it is always beneficial?
Types of bias in screening?
Principles for the development of screening.
The test: Validity, LR, Multiple testing, ROC curve, Kappa
The disease;
Evaluation of a screening program

74. Characteristics of Disease
Serious
Treatable
Pre-clinical detectable period
Early treatment is better than late
Prevalent

75. Characteristics of Disease
Why do we screen for serious disease?
why we screen for Phenyl ketonuria?
Why do we screen for treatable disease?
why don’t we screen for pancreatic cancer?

76. Characteristics of Disease
Why do we screen for diseases with a pre-clinical detectable period?
Ex: HIV vs. Legionella
Why do we screen for diseases where early treatment is better than late?
Ex: breast cancer

77. Characteristics of Disease
Why do we screen for prevalent disease?
Prevalence affects predictive value
Tayssachs
Breast cancer

78. Characteristics of Disease
The more prevalent a condition, the fewer false positive tests there will be
The less prevalent a condition, the fewer false negative tests there will be
No matter how good the test is!

79. Why do we test for prevalent diseases?
Although the combination ELISA/Western Blot test for HIV has extremely high sensitivity and specificity, predictive value is dependent on prevalence.
High risk population: prevalence = 40%
PPV = NPV = 0.993
Low risk population: prevalence = 0.01% PPV = NPV = 0.999

80. Principles for the development of screening;
The test must have high levels performance.
The condition screened for is an important cause of morbidity, disability, or mortality.
The natural history of the disease is sufficiently well known.
The test must be acceptable to the target population and their health care providers, and appropriate follow-up of positive findings must be ensured.
Sam Shapiro in Epidemiology and Health Services

81. Where are we? Definition of screening?
Whether it is always beneficial?
Types of bias in screening?
Principles for the development of screening.
The test: Validity, LR, Multiple testing, ROC curve, Kappa
The disease;
Evaluation of a screening program

82. Evaluation of Screening
What are the costs of
treatment for the disease?
stage by stage
a false negative?
how are cases usually found? Does missing the case on screen mean it is missed forever?
false positive?
risks of confirmatory test
psychological risks / harms: worry, etc.
labeling

83. Characteristics of Test
Safety
Cost
Acceptability
Validity
Reliability

84. Evaluation of Screening Outcomes
Study Designs
RCT
Compare disease-specific cumulative mortality rate between those randomized (or not) to screening
Eliminates confounding and lead time bias
But, problems of:
Expense, time consuming, logistically difficult, ethical concerns, changing technology.

85. Evaluation of Screening Outcomes
Study Designs
Observational Studies
Cohort:
Compare disease-specific cumulative mortality rate between those who choose (or not) to be screened
Case-control:
Compare screening history between those with advanced disease (or death) and healthy.
Ecological:
Compare screening patterns and disease experience (both incidence and mortality) between populations

86. Problems with Observational Studies
Confounding due to health awareness - screenees are more healthy (selection bias)
More susceptible to effects of lead-time bias and length-biased sampling
Poor quality, often retrospective data

87. Evaluation of Screening Outcomes
b) Measures of Effect of Screening
Disease-specific Mortality Rate (MR)
the number of deaths due to disease
Total person-years experience
The only gold-standard outcome measure for screening
NOT affected by lead time,
A comparison of the disease-specific MR between screened and unscreened populations in a RCT is the best and really only valid measure of the true efficacy of screening.
The disease-specific MR will not be changed by early diagnosis (i.e., lead time), or other common biases (selection, length-biased sampling)
However, as a direct result of the screening program, the time and manner of diagnosis may affect the cause-of-death attribution as recorded on death certificate (concept of labeling).
Important: The MT also does not incorporate adverse effects of screening.

89. NBSS1 (National Breast Screening Study), Canada 1980
Age at entry: 40 to 49.
Randomization: Individual volunteers, with names entered successively on allocation lists. Although criticisms of the randomization procedure have been made, a thorough independent review found no evidence of subversion and that subversion on a scale large enough to affect the results was unlikely.
Sample size: 25,214 study) and 25,216 control.
Cause of death attribution: Death certificates, with review of questionable cases by a blinded review panel. Also linked with the Canadian Mortality Data Base, Statistics Canada.
Follow-up duration: 13 years.
Relative risk of breast cancer death, screening versus control (95% CI): 0.97 ( ).

90. Where are we? Definition of screening?
Whether it is always beneficial?
Types of bias in screening?
Principles for the development of screening.
The test: Validity, LR, Multiple testing, ROC curve, Kappa
The disease;
Evaluation of a screening program

91. Session objectives
Screening basics
Evaluation of screening programs