1. for the Veterans Aging Cohort Study
MEASURES OF UNHEALTHY ALCOHOL USE AMONG HIV INFECTED AND UNINFECTED PATIENTS IN THE VETERANS AGING COHORT STUDY (VACS)
Kathleen McGinnis
Amy Justice
Richard Saitz
Kevin Kraemer
Kendall Bryant
David Fiellin
for the Veterans Aging Cohort Study

2. Objectives
Summarize and compare alcohol use in HIV+ and HIV- patients enrolled in VACS
Evaluate the AUDIT and AUDIT-C for identifying alcohol dependence and unhealthy use among HIV-infected and uninfected patients in care

3. Background
As persons with HIV live longer due to effective antiretroviral therapies, it has become more important to understand the effects of alcohol on health
HIV researchers and clinicians need valid and feasible measures for assessing alcohol use
This is the first study to evaluate the AUDIT and AUDIT-C for identifying alcohol dependence and unhealthy use among HIV-infected patients in care

4. VACS Ongoing 8 Site Study
3,410 HIV-infected veterans
3,409 HIV-uninfected veterans
Enrolled between
Self-complete surveys – Baseline and yearly follow-ups
Telephone interviews –
~1/2 completed at baseline
~1/4 completed at follow-up

5. Methods

6. VACS Alcohol Instruments
Self-complete survey –
10 item Alcohol Use Disorders Identification Test (AUDIT)
3 item Alcohol Use Disorders Identification Test (AUDIT-C)
3rd item of AUDIT asks about having 6 or more drinks on one occasion (heavy drinking episode)
Telephone interview FU1 – approx ¼ of VACS
Composite International Diagnostic Interview (CIDI)–Substance Abuse Module (SAM)
30 day time line follow-back (TLFB)

7. Analytic Sample
970 men completed the Composite International Diagnostic Interview – Substance Abuse Module (CIDI-SAM)
837 with alcohol data from all sources at Year 1 follow-up and with no more than 1 year in between self-completed survey and telephone interview
Because the majority (over 95%) of the sample is male and cutoff’s for unhealthy drinking vary for males and females, we limited this analysis to males only

8. Reference (“Gold”) Standard Measures
Alcohol dependence - based on a CIDI diagnosis of current (past year) alcohol dependence
Unhealthy alcohol use
CIDI diagnosis of current abuse or dependence
Risky drinking based on 30 day TLFB
>65 years: >7 drinks over 7 consecutive days or >3 drinks on one day
<65 years: >14 drinks over 7 consecutive days or >4 drinks on one day

9. Comparison Measures
AUDIT & AUDIT-C cut points with and without heavy drinking episode in past year

10. Analysis
Various cutoffs of AUDIT and AUDIT-C were compared to the reference standards using
Measures of performance
Sensitivity
Specificity
Positive predictive value (PPV)
Negative predictive value (NPV)
Kappa statistics to measure agreement

11. Considering Measures of Performance
Sensitivity – % with positive score of those with alcohol problem
Specificity– % without positive score of those without alcohol problem
PPV – % with alcohol problem of those with positive score
NPV – % without alcohol problem of those with without positive score
It is important to consider which measure is most important for the purpose of the study.
For an intervention study, sensitivity (identifying a high percent of those with the problem) would be very important, although one may also want to use a second screen so that those w/out the problem aren’t receiving the intervention.
Whereas to evaluate the proportion of people who were treated for a problem, PPV would be very important because one would want to make sure that those identified as having the problem actually have the problem.

12. Evaluating Kappa Statistics
Poor
Fair
Moderate
Good
Excellent
This is our criteria for evaluating kappa statistics
Landis, J.R. and Koch, G. G. (1977) "The measurement of observer agreement for categorical data" in Biometrics. Vol. 33, pp. 159—174

13. Results & Interpretation

14. Analytic Sample by HIV Status
Characteristic
HIV-infected
(n=444)
HIV-uninfected
(n=393)
P-value
Age in years, mean (SD)
50 (8.4)
54 (9.8)
<0.001
Race/ethnicity NS
African American
56%
51%
White
31%
36%
Hispanic/other
13%
Alcohol Measures
Dependence (CIDI) 8% 7%
Unhealthy Use (CIDI/TLFB)
22%
20%
The HIV-infected are younger than the uninfected in this sample
The race/ethnicty distribution is similar between groups
Prevalence of dependence and unhealthy use in the past year is similar between the HIV infected and uninfected

15. Alcohol Dependence Agreement “Gold Standard” based on CIDI
Comparison Measure
Sensitivity
Specificity
PPV
NPV
Kappa
Overall
AUDIT 16+ 32 98 57 95
.38
AUDIT 8+/4+* 58 89 30 96
.33
AUDIT 8+/4+ or heavy drinking episode 82 77 22
.26
AUDIT-C 6+ 38 93 29
.27
AUDIT-C 6+ or heavy drinking episode 74 79 97
Sensitivity is highest using AUDIT 8+/4+ or heavy drinking episode in past year; or using AUDIT-C 6+ or heavy drinking episode in past year. However, for those measures/cutpoints, PPV (% w/ alcohol dependence of those with a positive score) is low indicating a high percent of false positives.
AUDIT-16+ provides the highest/best PPV, but is still not very good.
There is a clear trade off between sensitivity and PPV depending on the measure/cutpoint used.
Tradeoff between sensitivity and PPV

16. Alcohol Dependence Agreement “Gold Standard” based on CIDI
Comparison Measure
Sensitivity
Specificity
PPV
NPV
Kappa
Overall
AUDIT 16+ 32 98 57 95
.38
AUDIT 8+/4+* 58 89 30 96
.33
AUDIT 8+/4+ or heavy drinking episode 82 77 22
.26
AUDIT-C 6+ 38 93 29
.27
AUDIT-C 6+ or heavy drinking episode 74 79 97
Kappa statistics represent “fair” agreement.
All Kappa Statistics fall into the fair category
Tradeoff between sensitivity and PPV

17. Alcohol Dependence Agreement “Gold Standard” based on CIDI
Comparison Measure
Sensitivity
Specificity
PPV
NPV
Kappa
Overall
AUDIT 16+ 32 98 57 95
.38
AUDIT 8+/4+* 58 89 30 96
.33
AUDIT 8+/4+ or heavy drinking episode 82 77 22
.26
AUDIT-C 6+ or heavy drinking episode 74 79 97
HIV+ 31 94
.37 66 90 36
.41 78 76 19
.22 81 25
.29
HIV- 33 56
.39 48 88 23
.24 86
.30 20
.23
The pattern is similar when we examine alcohol dependence measures of agreement and performance by HIV status.

18. Alcohol Dependence Agreement “Gold Standard” based on CIDI
Comparison Measure
Sensitivity
Specificity
PPV
NPV
Kappa
Overall
AUDIT 16+ 32 98 57 95
.38
AUDIT 8+/4+* 58 89 30 96
.33
AUDIT 8+/4+ or heavy drinking episode 82 77 22
.26
AUDIT-C 6+ or heavy drinking episode 74 79 97
HIV+ 31 94
.37 66 90 36
.41 78 76 19
.22 81 25
.29
HIV- 33 56
.39 48 88 23
.24 86
.30 20
.23
All Kappa Statistics fall into the fair (.2-.4) range
Kappa statistics fall into the “fair” range.

19. Unhealthy Drinking Agreement “Gold Standard” based on CIDI and TLFB
Comparison Measure
Sensitivity
Specificity
PPV
NPV
Kappa
Overall
AUDIT 8+/4+* 46 94 69 87
.46
AUDIT 8+/4+ or heavy drinking episode* 72 85 56 92
.52
AUDIT-C 5+ 52 95 74 88
.53
AUDIT-C 5+ or heavy drinking episode 86 58 91
Sensitivity is using AUDIT 8+/4+ or heavy drinking episode in past year; or using AUDIT-C 5+ or heavy drinking episode in past year, although these measures would still miss picking up around 30% with unhealthy drinking. Additionally, for those measures/cutpoints, PPV (% w/ alcohol dependence of those with a positive score) is low indicating a high percent (42%-44%) of false positives. Again, choosing measures with higher PPV will result in lower sensitivity. Again, there is a clear trade off between sensitivity and PPV depending on the measure/cutpoint used.
Clear tradeoff between sensitivity and PPV

20. Unhealthy Drinking Agreement “Gold Standard” based on CIDI and TLFB
Comparison Measure
Sensitivity
Specificity
PPV
NPV
Kappa
Overall
AUDIT 8+/4+* 46 94 69 87
.46
AUDIT 8+/4+ or heavy drinking episode* 72 85 56 92
.52
AUDIT-C 5+ 52 95 74 88
.53
AUDIT-C 5+ or heavy drinking episode 86 58 91
All Kappa Statistics fall into the moderate agreement range.
Clear tradeoff between sensitivity and PPV
All Kappa Statistics fall into the moderate agreement category

21. Unhealthy Drinking Agreement “Gold Standard” based on CIDI and TLFB
Comparison Measure
Sensitivity
Specificity
PPV
NPV
Kappa
Overall
AUDIT 8+/4+* 46 94 69 87
.46
AUDIT 8+/4+ or heavy drinking episode* 72 85 56 92
.52
AUDIT-C 5+ 52 95 74 88
.53
AUDIT-C 5+ or heavy drinking episode 86 58 91
HIV+ 42 65
.41 68 90
.50 49
.49 64 59
HIV- 51 73 78
.55 55 96 77 89
.57 57 93
The pattern is similar when we examine unhealthy alcohol use measures of agreement and performance by HIV status.

22. Unhealthy Drinking Agreement “Gold Standard” based on CIDI and TLFB
Comparison Measure
Sensitivity
Specificity
PPV
NPV
Kappa
Overall
AUDIT 8+/4+* 46 94 69 87
.46
AUDIT 8+/4+ or heavy drinking episode* 72 85 56 92
.52
AUDIT-C 5+ 52 65 74 88
.53
AUDIT-C 5+ or heavy drinking episode 86 58 91
HIV+ 42
.41 68 90
.50 49
.49 64 59
HIV- 51 95 73 78
.55 55 96 77 89
.57 57 93
All Kappa Statistics fall into the moderate (.4-.6) range
Kappa statistics fall into the “moderate” range.

23. Summary
Prevalence of alcohol dependence and unhealthy alcohol use is similar for HIV-infected and HIV-uninfected male patients (8% vs. 7%; and 22% vs. 20%)
Of unhealthy drinking, the percent who are also alcohol dependent is similar between HIV+ and HIV- (36% vs. 35%)

24. Summary
Performance measures and Kappa statistics were similar between HIV-infected and uninfected male patients for both alcohol dependence and unhealthy use for most measures
Specificity & NPV was 85%+ for most measures
Tradeoff between sensitivity and PPV
Overall measure of agreement may not be as important as prioritizing sensitivity vs. PPV for the purpose of the study

25. Limitations
Data for the self-completed survey and the telephone interview (TLFB and CIDI-SAM) were not collected at exactly the same time
Analysis limited those with no more than 1 year in between their self-completed survey and telephone interview
For 75% of these, the survey and interview were no more than 6 months apart
Women were not included in the analysis because the % of women in VACS is small and there were no women in the HIV group with unhealthy drinking

26. Strengths Large sample of both HIV-infected and HIV-uninfected
First study to evaluate alcohol measures among HIV-infected
Ability to compare the measures between HIV-infected and HIV-uninfected
Population is racially/ethnically diverse

27. Conclusions
The SAM-CIDI and 30-day TLFB were collected with extensive effort (and therefore, cost), but for many studies a briefer measure is necessary
Measures weren’t substantially different between HIV-infected and uninfected patients indicating that AUDIT is valid to use for persons with HIV
When choosing between AUDIT vs. AUDIT-C and cut points, it’s important to consider
the purpose of the study/identification
Which measure of performance is more important

28. Note
We do not show all AUDIT and AUDIT-C cut points evaluated due to space limitations
We did not find substantial differences between the various recommended cut points shown earlier in this presentation
Stay tuned for corresponding manuscript that will include comparing additional cut points for AUDIT and AUDIT-C not shown here

29. Acknowledgements PI and Co-PI: AC Justice, DA Fiellin
Scientific Officer (NIAAA): K Bryant
Participating VA Medical Centers: Atlanta (D. Rimland), Baltimore (KA Oursler, R Titanji), Bronx (S Brown, S Garrison), Houston (M Rodriguez-Barradas, N Masozera), Los Angeles (M Goetz, D Leaf), Manhattan-Brooklyn (M Simberkoff, D Blumenthal, H Leaf, J Leung), Pittsburgh (A Butt, E Hoffman), and Washington DC (C Gibert, R Peck)
Core Faculty: K Mattocks (Deputy Director), K Akgun, S Braithwaite, C Brandt, K Bryant, R Cook, K Crothers, J Chang, S Crystal, N Day, R Dubrow, M Duggal, J Erdos, M Freiberg, M Gaziano, M Gerschenson, A Gordon, J Goulet, N Kim, M Kozal, K Kraemer, V LoRe, S Maisto, P Miller, P O’Connor, C Parikh, C Rinaldo, J Samet
Staff: H Bathulapalli, T Bohan, D Cohen, A Consorte, P Cunningham, A Dinh, C Frank, K Gordon, J Huston, F Kidwai, F Levin, K McGinnis, C Rogina, J Rogers, L Sacchetti, M Skanderson, J Tate, E Williams
Major Collaborators: VA Public Health Strategic Healthcare Group, VA Pharmacy Benefits Management, Massachusetts Veterans Epidemiology Research and Information Center (MAVERIC), Yale Center for Interdisciplinary Research on AIDS (CIRA), Center for Health Equity Research and Promotion (CHERP), ART-CC, NA-ACCORD, HIV-Causal
Major Funding by: National Institutes of Health: NIAAA (U10-AA13566), NIA (R01-AG029154), NHLBI (R01-HL095136; R01-HL090342; RCI-HL100347) , NIAID (U01-A ), NIMH (P30-MH062294), and the Veterans Health Administration Office of Research and Development (VA REA ) and Office of Academic Affiliations (Medical Informatics Fellowship).

30. Appendix (Extra Slides)

31. VACS Alcohol Use Compared to Other Samples
Characteristic
VACS
HIV+
(n=444)
HIV-
(n=393)
Saitz
Rubinsky et al
(n=392 outpatients at family medicine clinic)
Alcohol Measures
Dependence (CIDI) 8% 7% 4%
12%
Unhealthy Use (CIDI/TLFB)
22%
20%
7-20%
33%
Percent Dependent of Unhealthy Users
36%
35%
1/3 risky users are at risk for dependence
38%

32. Summary of Some Recommended AUDIT Cut Points to Use
Risky drinking
(Gual, et al, Alcohol and Alcoholism, 2002)
7+ for men 5+
Positive screen
NIAAA Clinical Support Materials, (
8+ for men up to 60
4+ for women, adolescents, and men over 60
Can also include binging in past year (5+ drinks for men; 4+ drinks for women) 6+
Hazardous Drinking
(Gordon et al, J Fam Prac, 2001) 8+ 3+
Heavy drinking and/or active alcohol abuse or dependence
(Bush et al, Arch Intern Med, 1998)
See Table 3 (9) 4+
Five Categories
(Bryson, Ann Intern Med, 2008)
0=nondrinker
1-2=low-level use
4-5=mild misuse
6-7=moderate misuse
8-12=severe misuse
Zones of Risk
(WHO,
I alc educ
II simple advice
III simple advice, brief counseling, and continued monitoring
20+ IV Ref to specialist for dx eval and trt
Alcohol Misuse (Bradley et al, ACER, 2007)

33. Considering Measures of Performance
Sensitivity – % with positive score of those with alcohol problem
Specificity– % without positive score of those without alcohol problem
PPV – % with alcohol problem of those with positive score
NPV – % without alcohol problem of those with without positive score
To provide an intervention, need ability to identify those with the problem!

34. Considering Measures of Performance
Sensitivity – % with positive score of those with alcohol problem
Specificity– % without positive score of those without alcohol problem
PPV – % with alcohol problem of those with positive score
NPV – % without alcohol problem of those with without positive score
To evaluate treatment utilization, important that those with positive screen actually have an alcohol problem!

35. Considering Measures of Performance
Sensitivity – % with positive score of those with alcohol problem
Specificity– % without positive score of those without alcohol problem
PPV – % with alcohol problem of those with positive score
NPV – % without alcohol problem of those with without positive score
We don’t want too many false positives!

36. Considering Measures of Performance
Sensitivity – % with positive score of those with alcohol problem
Specificity– % without positive score of those without alcohol problem
PPV – % with alcohol problem of those with positive score
NPV – % without alcohol problem of those with without positive score
We don’t want to mis-identify those with an alcohol problem!

37. Considering Measures of Performance
Sensitivity – % with positive score of those with alcohol problem
Specificity– % without positive score of those without alcohol problem
PPV – % with alcohol problem of those with positive score
NPV – % without alcohol problem of those with without positive score
To provide an intervention, need ability to identify those with the problem!
We are going to focus on sensitivity and PPV
To evaluate treatment utilization, important that those with positive screen actually have an alcohol problem!

38. Alcohol Dependence Agreement “Gold Standard” based on CIDI
Comparison Measure
Sensitivity
Specificity
PPV
NPV
Kappa
Overall
AUDIT 16+ 32 98 57 95
.38
AUDIT 8+/4+* 58 89 30 96
.33
AUDIT 8+/4+ or heavy drinking episode 82 77 22
.26
AUDIT-C 6+ 38 93 29
.27
AUDIT-C 6+ or heavy drinking episode 74 79 97

39. Unhealthy Drinking Agreement “Gold Standard” based on CIDI and TLFB
Comparison Measure
Sensitivity
Specificity
PPV
NPV
Kappa
Overall
AUDIT 8+/4+* 46 94 69 87
.46
AUDIT 8+/4+ or heavy drinking episode* 72 85 56 92
.52
AUDIT-C 5+ 52 95 74 88
.53
AUDIT-C 5+ or heavy drinking episode 86 58 91
Sensitivity is using AUDIT 8+/4+ or heavy drinking episode in past year; or using AUDIT-C 5+ or heavy drinking episode in past year, although these measures would still miss picking up around 30% with unhealthy drinking. Additionally, for those measures/cutpoints, PPV (% w/ alcohol dependence of those with a positive score) is low indicating a high percent (42%-44%) of false positives. Again, choosing measures with higher PPV will result in lower sensitivity. Again, there is a clear trade off between sensitivity and PPV depending on the measure/cutpoint used.