1. A National Model for Funder-Researcher-Provider EBP Implementation Randolph D. Muck, SAMHSA, CSAT Michael L. Dennis, Susan H. Godley, and Mark D. Godley, Chestnut Health Systems Presentation at the 2 nd Annual NIH Conference on the Science of Dissemination and Implementation, Natcher Conference Center, Bethesda, MD, January 29, 2009. Support for this project was provided by SAMHSA contract 270-07-0191 and NIH grants RO1 DA018183 and 2 RO1 AA010368. The opinions are those of the authors and do not reflect official positions of the government.

2. 2 Goals 1.Provide a brief historical overview of CSAT’s commitment to science-to-service initiatives within its adolescent treatment grant portfolio; 2.Illustrate this commitment by describing how training and quality assurance methods employed in the Assertive Adolescent and Family Treatment initiative increase model- fidelity and clinical outcomes; and 3.Discuss conclusions and next steps for the transfer of evidence-based practices to those who treat substance use disorders.

3. 3 Background For the past decade SAMHSA’s Center for Substance Abuse Treatment (CSAT) has funded a series of initiatives to replicate evidence-based practices and collect information on their effectiveness in the community Since September 2006, the Center for Substance Abuse Treatment (CSAT) has awarded 32 Assertive Adolescent Family Therapy (AAFT) grants Purpose of AAFT grants: To provide support for the initial training and implementation of evidenced based approaches to assessment and treatment for adolescents and their families

4. 4 Assertive Adolescent Family Treatment (AAFT) Grant Sites by Funding Cohort Aurora Boston Cambridge Columbia Columbus Downey Fitchburg Fresno Ft Worth Houston Huntington Huntsville Laredo Little Rock Los Angeles Manchester Nashville Oak Ridge Oakland Orlando Phoenix Pinellas Park Reno San Antonio San Francisco Syracuse Tarzana Thornton Tucson AAFT Cohort 1 (2006-2009) AAFT Cohort 2 (2007-2010) WA OR CA NV ID UT WY MT CO AZ NM TX ND SD NE KS OK MN IO MO AR LA WI IL IN KY TN MS AL GA FL SC NC VA WV OH PA NY ME. CT VT NH NJ DE MD D.C RI MI AK HI PR San Jose

5. 5 Evidence Based Practices in AAFT Formal training, quality assurance, certification, monitoring and technical assistance implementing – the Global Appraisal of Individual Needs (GAIN) to improve intake assessment, clinical interpretation, monitoring, and data management – the Adolescent Community Reinforcement Approach (A-CRA) and Assertive Continuing Care (ACC) to improve clinical practice and supervision Average of $328,000/yr x 3 years grant award to help with initial implementation ($27.5 million across grantees)

6. 6 GAIN Components Training (4 day), feedback on taped interviews, and multi-level certification program (administration, local trainer, data management) Additional training (4 day), feedback and multi-level certification on advanced clinical interpretation Access to PC and web based computer applications to support front line use Cleaning/access to SPSS files to support analysis for local, cross site or secondary research Participate in topical workgroups (e.g., Females, African American, Spanish language, GLBTQ, Young Adults, Opioid) that share ideas and resources

7. 7 GAIN Training and Certification Requirements Local GAIN Trainers – Attend a 4 day training – Receive written and oral feed back on 2-6 taped assessments until passing “certification” that they are at least satisfactory on documentation, handling questions, validity and rapport – Attend coaching calls with other sites led by experts GAIN Advanced Clinical Staff – pass knowledge pre-tests on DSM & ASAM – Attend additional 4 day training on advanced clinical interpretation – Receive written feedback on 3-6 biopsychosocial summaries – Attend coaching calls with other sites led by experts

8. 8 Performance Indicator Reports TA contractor provides CSAT monthly performance indicators by Site Sample Performance Indicators – % of expected case flow for recruitment, each data collection wave – % initiating within 2 weeks, using evidenced based practice, engaging in treatment at least 4 weeks, having continuing care over 90 days out, – Completion of certification, installation of software, status of other implementation problems, and who is addressing any outstanding issues Management report also indicates who is following up on any problems, including potential need to escalate to the project officer

9. 9 GAIN Certification Progress

10. 10 Bi-Weekly Coaching calls Session recordings & data are uploaded to the web; Experts provide ratings and narrative feedback 4-day centralized training session A-CRA/ACC Technical Assistance A-CRA/ACC Certification Requirements are clearly delineated & monitored Clinical and supervision sessions are digitally recorded Treatment Manual and Knowledge Test

11. 11 Clinician & Supervisor Certification Requirements for A-CRA/ACC Pass knowledge tests Attend a 3.5 day training Attend coaching calls with other sites led by experts Clinicians record clinical sessions for experts to rate and must pass 9 core procedures Supervisors pass criteria for a recorded supervision session and show rating reliability across 6 therapy sessions

12. 12 Upload Digital Session Recordings Read Reviews of sessions by expert raters

13. 13 Sample Procedure Rating 1 2 3 4 5 | | | | | poor needs satisfactory very excellent improvement good Caregiver Overview, Rapport Building, and Motivation: 48. ____ ____ Provided an overview of ACRA 49. ____ ____ Set positive expectations 50. ____ ____ Reviewed research regarding parenting practices 51. ____ ____ Identified CG reinforcers for continued work 52. ____ ____ Kept discussion (about adolescent) positive

14. 14 Feedback to Clinicians during certification process After each session is reviewed, the clinician is notified to check a personal certification workbook that shows: – How she/he did on the most recent session – Ratings on all prior recordings – Clear feedback on where he/she is in the certification process (e.g., has passed 3 of 9 procedures) and which procedures are left to pass Clinicians also get narrative feedback to accompany the ratings with specific feedback about – What they did well (with praise) – What needs improvement and how it can be improved Clinicians and supervisors are encouraged to discuss questions on frequent coaching calls and also hear feedback from coaches on sessions that have been reviewed.

15. 15 Performance Indicator Reports TA contractor provides CSAT monthly performance indicators by Site, Supervisors, and individual Clinicians Sample Performance Indicators – Cases Opened to website – Digital Session Recordings Uploaded – Digital Session Recordings Reviewed – Certification Progress (# of procedures passed for each clinician) – Attendance at Coaching Calls – # of Supervisors & Clinicians Certified If PI data suggests site problems, action is taken by project officer (e.g., phone call, site visit)

16. 16 A-CRA/ACC Certification Progress

17. 17 Change in 6 Month Abstinence Relative to Other CSAT Adolescent Treatment Grant Programs Notes: \a GAIN Required, \b A-CRA/ACC required, \c MET/CBT5 required, \d Juvenile justice grant encouraging use of GAIN, ACRA/ACC and/or MET/CBT5 but without as much training/oversight Source: SAIS System (GPRA) 2008 data

18. 18 Conclusions The CSAT Adolescent Treatment program has demonstrated the ability to replicate evidence based assessment and treatment approaches in community based settings with superior outcomes to previous grant initiatives using less structured approaches. Appears that including the research team who developed and tested the EBPs increases the likelihood of successful implementation (less model- drift in training and certification, more insight and understanding to help providers learn and solve implementation problems). Quality Assurance monitoring and feedback to providers and management reports to funder are necessary to initiate and maintain EBP fidelity.

19. 19 Next Steps Currently collaborating with NIAAA (R01 AA017625-01) to experimentally test the effect of adding P4P incentives to further improve both quality and quantity of A-CRA implementation Future research is necessary to study the sustainability of these EBPs in provider organizations after CSAT grant funds end.

20. For more information or a copy of this presentation please contact: Muck, Randy (SAMHSA/CSAT) Randy.Muck@samhsa.hhs.gov Questions?