1. Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut Health Systems, Bloomington, IL and Richard Lennox, Psychometric Technologies, Hillsborough, NC Workshop at the Joint Meeting on Adolescent Treatment Effectiveness (JMATE), Baltimore, MD, March 28, 2006.

2. The content of this presentations are based on treatment & research funded by the Center for Substance Abuse Treatment (CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA) under contract 270-2003-00006 using data provided by the CYT grantees (Nos. TI11317, TI11320, TI11321, TI11323, and TI11324). The authors thank Sarah Knecht, Michelle White and the GAIN QA team for helping to identify common inconsistencies, Sandra McGuinness for the code for the inconsistencies and Barth Riley of UI-C for Rasch code. The opinions are those of the author and do not reflect official positions of the consortium or government. Available on line at www.chestnut.org/LI/Posters or by contacting Joan Unsicker at 720 West Chestnut, Bloomington, IL 61701, phone: (309) 827- 6026, fax: (309) 829-4661, e-Mail: junsicker@Chestnut.Org Acknowledgement

3. CYT Cannabis Youth Treatment Randomized Field Trial Sponsored by: Center for Substance Abuse Treatment (CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA), U.S. Department of Health and Human Services Coordinating Center: Chestnut Health Systems, Bloomington, IL, and Chicago, IL University of Miami, Miami, FL University of Conn. Health Center, Farmington, CT Sites: Univ. of Conn. Health Center, Farmington, CT Operation PAR, St. Petersburg, FL Chestnut Health Systems, Madison County, IL Children’s Hosp. of Philadelphia, Phil.,PA

4. Design Target Population: Adolescents with marijuana disorders who are appropriate for 1 to 3 months of outpatient treatment. Inclusion Criteria: 12 to 18 year olds with symptoms of cannabis abuse or dependence, past 90 day use, and meeting ASAM criteria for outpatient treatment Data Sources: self report on the GAIN (Dennis et al 2003), collateral reports, on-site and laboratory urine testing, therapist alliance and discharge reports, staff service logs, and cost analysis. Random Assignment: to one of three treatments within site in two research arms and quarterly follow-up interview for 12 months, and long term follow up 30 to 42 months later. Source: Dennis et al 2002, 2004

5. Adolescent Cannabis Users in CYT were as or More Severe Than Those in TEDS* 100% 15intake 85% 46% 26% 78% 26% 47% 26% 71% 0% 20% 40% 60% 80% First used under age DependenceWeekly or more use at Prior Treatment % of Admissions. CYT Outpatient(n=600)TEDS Outpatient (n=16,480) * Adolescents with marijuana problems admitted to outpatient treatment Source: Tims et al, 2002

6. Demographic Characteristics 62% 15% 55% 50% 30% 83% 17% 0% 20% 40% 60% 80% 100% FemaleMaleAfrican American CaucasianUnder 1515 to 16Single parent family Source: Tims et al, 2002

7. Institutional Involvement 25% 87% 47% 62% 0% 20% 40% 60% 80% 100% In schoolEmployedCurrent CJ Involvement Coming from Controlled Environment Source: Tims et al, 2002

8. Patterns of Substance Use 9% 17% 71% 73% 0% 20% 40% 60% 80% 100% Weekly Tobacco Use Weekly Cannabis Use Weekly Alcohol Use Significant Time in Controlled Environment Source: Tims et al, 2002

9. Multiple Problems are the NORM Self-Reported in Past Year Source: Dennis et al, 2004

10. Co-occurring Problems are Higher for those Self-Reporting Past Year Dependence Source: Tims et al., 2002 * p<.05

11. Validity Measures Number of Inconsistencies –Count of 49 paired items consistently answered by over 90% of the clients, but that are inconsistent Denial/Misrepresentation – Sum of staff rating over 8 sections on a scale of 0-no problem, 1-estimating, 2- misunderstanding, 3-denial, 4-misrepresentation Context Effect – staff report of problems that might effect the interview (e.g.., someone present, interruptions, in juvenile justice setting) Proportion of Missing Data on 86 Items used in the GAIN’s core 10 Change measures: Substance Frequency Scale, Current Withdrawal Scale, Substance Problem Scale, Health Problem Scale, Emotional Problem Scale, Recovery Environment Risk Index, Social Risk Index, Illegal Activity Scale, Training Activity Scale and the Employment Activity Scale

12. Validity Measures (Continued) This last two measures are based on the residual (actual vs expected answers) of On the 123 items of the GAIN’s 4 main psychopathology and psychopathy scales. They are based the outfit and infit statistics under the Rasch (1960) measurement model and are reported in logits Atypicalness a measure of endorsing high severity items without first endorsing the typical prior items (e.g.., suicide without depression) Randomness a measure of answers that are more random than expected on the GAIN’s 4 main psychopathology and psychopathy scales Atypicalness Residual Randomness Where y ni is the observed response of person n to item i and p ni is the probability of a correct response for person n on item i.

13. Correlation of Validity Measures Bold indicates p <.05. Denial/Misrepr. 0.07-- Context Effect 0.10 0.31 -- %Missing Data -0.060.030.05-- Atypicalness 0.03 0.20 0.05-0.02-- Randomness 0.00 0.240.14 -0.04 0.57 Inconsistencies -- Denial/Misrepr. Context Effect %Missing Data AtypicalnessRandomness Inconsistencies -- While there is some overlap, for the most part these measures capture different aspects of validity

14. Trichotomization of Validity Measures Like all GAIN scales we trichotomized the validity measures into low/mod/high range to help line staff interpret them. Because they were close to normally distributed, we divided Inconsistencies, Atypicalness and Randomness into: Low 0-50%, Mod 51-90% and High 91-100% Because they were sharply right skewed, Missing data, Denial/misrepresentation and Context effects were divided into Low 0%, Mod 1-90%, and High 91-100%

15. Overview of Validation Test Results Validity Measures Denial/Misrep Rating False Negatives Relative To Urine at Intake X Missing Data Context Inconsistencies Randomness (aka infit) Bias 3 Month outcomes X X X Atypicalness (aka outfit) Construct Validity at Intake X X X X X X Test-Retest at Intake X X X X X X Internal Consistency at Intake X X X X X X: Continuous or trichotomous version of validity measure is a statistically significant (p<.05) predictor of worse values on the criterion in this column

16. Internal Consistency Results 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 Average Cronbach’s Alpha* Low 0.87 Mod 0.88 0.860.880.860.87 High 0.890.840.800.830.820.84 Denial/Misrep Rating Missing DataContextInconsisteciesRandomnessAtypicalnessAverage *average alpha across Substance Problem Scale, Internal Mental Distress Scale, Behavior Complexity Scale & Crime/Violence Scale Inconsistencies are the best predictor of low alpha

17. Test-Retest Results Randomness is the best predictor of low test-retest Rho

18. Correlations with Intake Variables Bold indicates p <.05. Spearman Rho Days of Any AOD use Days of Alcohol Use Days of Marj.Use Days of AOD Problems Past Month Problems Days of Arguing/Violence Days of Illegal Activity Days of Illegal Activity for $ Denial/Misrep-0.06-0.07-0.04 -0.10-0.13 -0.02-0.040.00 Missing Data0.000.02-0.01-0.020.03-0.030.050.07 Context0.04-0.010.06 -0.09 -0.02-0.040.00 Inconsistencies-0.01 -0.040.02-0.02-0.05 -0.14-0.09 Randomness -0.17-0.12-0.14-0.25-0.23 -0.050.01 0.09 Atypicalness0.02 0.03-0.030.02 0.110.170.20 Randomness on symptoms generally predicts lower than expected values Atypicalness predicts higher than expected values

19. Staff Ratings of Denial/Misrepresentation Predict False Negatives Relative to Urine Screens Only significant predictor of False Negatives, OR=4.0, 95% CI (2.03, 7.96) But not all Denial/ Misrepresen tation is about drug use

20. Predicting 3-Month Outcomes Predicted 3-month variables with intake only and in second step, added the validity measures Dependent Variables: Substance Frequency, Substance Problems, Emotional Problems, Recovery Environment, Social Risk and Illegal Activity Context and Inconsistencies had a small significant positive relationship with Recovery Environment Risk at 3 months Randomness had a small significant positive relationship with Substance Frequency and Illegal Activity

21. Limitations We had very little missing data, denial/ misrepresentation, and false negatives in the data from this multi-site clinical trial This data set was for outpatient and limited in severity and diversity We plan to replicate the analyses with several larger data sets that are more diverse in terms of clinical severity, geography, demographics, level of care and type of service providers

22. Conclusions The 6 GAIN validity measures are good markers for predicting problems with internal consistency, reliability, and validity Even where there were problems, self report was still generally reliable and valid The small correlations between measures and differences in what they predicted demonstrate that they are measuring different facets of the problem Having developed metrics for identifying problem cases, the next step is to develop interventions to reduce the likilihood of these problems.

23. References Dennis, M. L., Godley, S. H., Diamond, G., Tims, F. M., Babor, T., Donaldson, J., Liddle, H., Titus, J. C., Kaminer, Y., Webb, C., Hamilton, N., & Funk, R. (2004). The Cannabis Youth Treatment (CYT) Study: Main Findings from Two Randomized Trials. Journal of Substance Abuse Treatment, in press Dennis, M. L., Titus, J. C., Diamond, G., Donaldson, J., Godley, S. H., Tims, F., Webb, C., Kaminer, Y., Babor, T., Roebeck, M. C., Godley, M. D., Hamilton, N., Liddle, H., Scott, C., & CYT Steering Committee. (2002). The Cannabis Youth Treatment (CYT) experiment Rationale, study design, and analysis plans. Addiction, 97, 16-34. Dennis, M. L., Titus, J. C., White, M. K., Unsicker, J., & Hodgkins, D. (2003). Global Appraisal of Individual Needs: Administration Guide for the GAIN and Related Measures. Bloomington, IL: Chestnut Health Systems. Retrieved from http://www.chestnut.org/li/gain. Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danmarks Paedogogiske Institut. (Republished Chicago: The University of Chicago Press: 1980). Tims, F. M., Dennis, M. L., Hamilton, N., Buchan, B. J., Diamond, G. S., Funk, R., & Brantley, L. B. (2002). Characteristics and problems of 600 adolescent cannabis abusers in outpatient treatment. Addiction, 97, 46-57.