Using Data to Inform Practice Michael L. Dennis, Ph.D. Chestnut Health Systems, Normal IL Presentation at SAMHSA/CSAT Satellite Session, “ Implementing Evidenced Based Treatment for Adolescent Based Treatment,” College of Problems on Drug Dependence, Reno, NV, June 20, The opinions in this presentation are those of the author and do not reflect positions of the government. This presentation was supported by SAMHSA/CSAT contract no Available from or by contacting author at or

Common Questions in Local Program Evaluation and Development 1.Who is being served? 2.What services are they receiving? 3.To what extent are services being targeted at those in need? 4.To what extent are services being delivered as expected? 5.Which is the most effective of several services delivered? 6.What does it cost, cost effectiveness? Source: Dennis, Fetterman & Sechrest (1994)

139/146=95% unmet need 7/28=25% to targeted 146/1222=40% in need Exploring Need, Targeting & Unmet Need Size of the Problem Extent to which services are currently being targeted Extent to which services are not reaching those in most need At Intake. After 3 mon No/Low Need Mod/High Need Total Any Treatment21728 No Treatment Total

Mental Health Problem (at intake) vs. Any MH Treatment by 3 months *3+ on ASAM dimension B3 criteria Source: Ives & Moritz 2009 CSAT Juvenile and Family Drug Court Presentation

MH Issues at Intake vs. MH Treatment at 3 months Source: Ives & Moritz 2009 CSAT Juvenile and Family Drug Court Presentation

Other MH Issues at Intake vs. MH Treatment at 3 months Source: Ives & Moritz 2009 CSAT Juvenile and Family Drug Court Presentation

GRRS Treatment Planning Needs: Mental Health Source: Ives & Moritz 2009 CSAT Juvenile and Family Drug Court Presentation

Why Do We Care About Unmet Need? If we subset to those in need, getting mental health services predicts reduced mental health problems Both psychosocial and medication interventions are associated with reduced problems If we subset to those NOT in need, getting mental health services does NOT predict change in mental health problems

Residential Treatment need (at intake) vs. 7+ Residential days at 3 months Opportunity to redirect existing funds through better targeting Source: Ives & Moritz 2009 CSAT Juvenile and Family Drug Court Presentation

GRRS Treatment Planning Needs: Substance Use and Treatment Source: Ives & Moritz 2009 CSAT Juvenile and Family Drug Court Presentation

Impact of Intake Severity on Outcome Source: ATM Main Findings data set SPSM groupings Dot/Lines show Means 06 Wave 8 10 Substance Problem Scale (0-16 Past Month Symptoms) No problems (0-25%ile) 1-3 problems (25-50%ile) 4-8 problems (50-75%ile) 9+ problems (75-100%ile) OVERALL Intake Severity Correlated -.66 with amount of change Programs with low severity look better with absolute outcomes (e.g. abstinence) Programs with high severity look better with amount of change

Different than Regression to the Mean Source: ATM Main Findings data set SPSM groupings Dot/Lines show Means 06 Wave 8 10 Substance Problem Scale (0-16 Past Month Symptoms) No problems (0-25%ile) 1-3 problems (25-50%ile) 4-8 problems (50-75%ile) 9+ problems (75-100%ile) OVERALL In its most basic form, the mean & variance are the same at both time points; no correlation between intake & amount of change

Example of Multi-dimensional HIV Subgroups A. Low Risk B. Mod. Risk W/O Trauma C. Mod. Risk With Trauma D. High Risk Total Cohen's Effect Size d Unprotected Sex Acts (f=.14) Days of Victimization (f=.22) Days of Needle Use (f=1.19) Source: Lloyd et al 2007

Possible Comparison Groups published data site over time subsites, staff, or clinics compare site to larger program (all sites) compare site to similar level of care, geography, demographic subgroup, or clinical subgroup match clinical subgroups from GAIN related presentations or papers formal matching or propensity scoring to make groups more statistically comparable formal randomized experiments path or mediation models to test whether it is actually the dosage or key ingredient driving the change

Evaluation of an OTI Waiting List Reduction Grant from Appointment and Admission Log Source: Dennis, Ingram, Burks & Rachal, 1994 Grant allowed program to add 100 slots and reduced time to readmission Used up slots in 2 months and (unexpectedly) had 200 person waiting list

Implementation is Essential ( Reduction in Recidivism from.50 Control Group Rate) The effect of a well implemented weak program is as big as a strong program implemented poorly The best is to have a strong program implemented well Thus one should optimally pick the strongest intervention that one can implement well Source: Adapted from Lipsey, 1997, 2005

Major Predictors of Effective Programs that we have to be cognizant of… 1.Triage to focus on the higher severity subgroups of individuals 2.An explicit intervention protocol (typically manualized) with a priori evidence that it works when followed with targeted population 3.Use of monitoring, feedback, supervision and quality assurance to ensure protocol adherence and project implementation 4.Use proactive case supervision at the individual level to ensure quality of care Source: Adapted from Lipsey, 1997, 2005