1. Using Observation to Enhance Supervision CIMH Symposium Supervisor Track Oakland, California April 27, 2012

2. FFT is an Evidence-Based Practice
Four decades of research support establishing the efficacy of FFT with behavior problem youth
Recidivism/arrest, including violent felony crimes
Drug use
Out of home placement
Prevention of arrest in siblings of referred youth
Reduction in youth, mother, father interpersonal distress/somatic complaints
Improvement in family functioning
Significant cost effectiveness (up to $14.87 return for each dollar spent)
Findings replicated across research teams, with therapists with diverse clinical experience, and with culturally diverse populations

3. Science to Practice
Challenge in dissemination is to ensure that therapists implement FFT with the same rigor and fidelity established in controlled trials

4. Fidelity and Outcomes: State of Washington Implementation
38%* reduction in felony recidivism
$10.67 return for each $1 invested
$2100 per
family cost to implement
* Statistically significant outcome as compared to control condition
Wash State Institute for Public Policy, 2005

5. The Effect of Supervised Experience: Felony Recidivism by Level of Therapist Competence
The data for this slide was obtained from the Statewide Washington State FFT Project. The case experience was clustered into three levels: 1-7, 8-14, and The high competence therapists are the competent and High competent group combined while the low competent group are the “borderline and non competent” groups combined.
Case Experience
Aos et al., 2004 5

6. Summary of Washington Findings
Only half the therapists were actually implementing FFT in a competent manner
Reductions in felony recidivism were evident only for therapists implementing the model competently
For the subset of competent FFT therapists, the financial benefits of the dissemination were estimated at $7.50 for each dollar of program cost (Aos et al., 2004)

7. Modifications to Training and Dissemination System
Intensive engagement of stakeholders and buy-in during site set-up process
Revisions to the Client Services System
Web-based tracking system
Implementation of new procedures for training and supervising therapists
Increased accountability for practice using data-informed practice guidelines

8. What Works to Reduce Crime?
Juvenile Offenders
Change in Crime
(# of EB Studies)
Benefits Minus Costs
(per-person, life cycle)
Probability: you lost $
Functional Family Therapy
-18% (7)
$32,021 (<1%)
Multisystemic Therapy
-13% (10)
$18,120 (<1%)
Family Integrated Transitions
-10% (1)
$29,721 (~5%)
Multi-Dimensional Treatment
Foster Care
-18% (3)
$64,486 (<1%)
Adapted from Aos (2010)

9. Return-on-Investment of Juvenile Offender Programs on Crime Reduction (WSIPP)
Juvenile Program
Program Cost Per Program Participant
Victimizations Avoided Per Program Participant
Taxpayer Benefits Per Program Participant
Victim Benefits Per Program Participant
Per $/1000 Program Cost
State Percent of Benefits
Multi-Dimensional Treatment Foster Care
$7418
0.10
50%
$24,068
Family Integrated Transitions
$10795
0.04
$13,050
Coordination of Services
$379
0.19
45%
$2,135
Functional Family Therapy
$3134
0.22
$20,623
Multi-Systemic Therapy
$7076
0.05
$11,027

10. Next Steps Is FFT effective in real world settings?
Moving from efficacy to effectiveness research
What procedures are necessary to achieve the most potent effects?
Observation-based supervision has been critical in the development of clinical practices and in efficacy trials
The costs associated with supervision based on a direct review of therapy session recordings are often viewed as prohibitive .
Does observation-based supervision influence therapist competence and subsequent treatment outcomes?
The sustainability of community implementations must be considered in the context of empirical evidence for the procedures in relation to outcomes achieved.

11. Primary Objective
To examine the effects of observation-based supervision (BOOST) versus the standard supervision as usual (SAU) approach currently used by FFT LLC on the therapists’ ability to facilitate changes in the family.
BOOST will examine the following outcomes:
adolescent drug use
conduct/delinquent behaviors.
engaging youth/families in treatment
enhancing therapist competence 

12. Engagement-Retention
Mediating Variables
Outcome Variables
Engagement-Retention
in Treatment
AIM 2
AIM 4
Therapist
Competence
AIM 3a
Supervision
Condition
AIM 4
AIM 3b
AIM 4
Family
Functioning
Adolescent
Substance Use
AIM 1a, 1b
Adolescent Conduct /
Delinquency
Conceptual Model of Hypothesized Relationships

13. Random Assignment of FFT Supervision Teams
Supervision as Usual
8 Teams - 3 Therapists Each
(n=24)
Observation-Based Supervision
8 Teams – 3 Therapists Each
(n=24)
Family Referral (n=288)
Screening and Informed Consent
Baseline Assessment
Functional Family Therapy
Follow-Up Assessments: 5-and 12-months after Baseline

14. Implementation at Each Site
Participants will be 18 adolescents/families per team
FFT teams will include 3 therapists (6 families per therapist)
Duration of study at each site
Month 1: Start Up; Therapist Consent; Randomization of FFT teams
Month 2-8: Enrollment of family participants
Months 2 to 12: Clinical services *
Months 6-12: Completion of 5-month assessment
Months 14-20: Completion of 12-month assessment
*Supervisor and Therapist activities completed. Therapists in SAU eligible to participate in BOOST training

15. Features of the Study
One of the only formal examinations of dissemination outcomes in evidence-based practices for youth with disruptive behavior problems
First study of supervision procedures!
Inclusion of a diverse sample of family and therapist participants across multiple community agencies
Tracking treatment outcomes and post-treatment outcomes
Protocol is fully bi-lingual (including computerized assessment procedures)
BOOST procedures will be transported into participating agencies to enhance sustainability after study
Includes procedures to train therapists and site supervisors in SAU after clinical phase of study is completed at each site

16. Cautions Intensity of review can be overwhelming Rule of 5 to 1
Try not to blow smoke
Specificity and brevity
Matching to therapist and family
Let therapist and team make comments and recommendations