Ensuring Effective Treatment and Supervision Operationalizing Fidelity in Real World Settings Kimberly Gentry Sperber, Ph.D.

Workshop Objectives Review literature on fidelity Identify opportunities for assessing fidelity Identify resources for monitoring and improving fidelity Address barriers to monitoring fidelity Begin to think through implementation plan

What Does Science Say About Fidelity? Objective 1 What Does Science Say About Fidelity?

Latessa, Cullen, and Gendreau (2002) Article notes 4 common failures of correctional programs: Failure to use research in designing programs Failure to follow appropriate assessment and classification practices Failure to use effective treatment models Failure to evaluate what we do

The Role of Fidelity Lowenkamp and Latessa (2005) Examined data from 38 residential correctional programs for adults Looked at relationship between program fidelity and program effectiveness. Program fidelity was assessed using the CPAI. Found significant correlation between fidelity and effectiveness CPAI scores correlated to reincarceration

Lowenkamp and Latessa Findings Continued Differences in recidivism rates based on CPAI scores: Scores of 0-49% demonstrated 1.7% reduction compared to comparison group. Scores of 50-59% demonstrated 8.1% reduction. Scores of 60-69% demonstrated 22% reduction.

CPAI Data Continued Holsinger (1999) Examined data from Adolescent Community Correctional Facilities in Ohio Looked at relationship between program fidelity and program effectiveness. Program fidelity was assessed using the CPAI. Outcome measures examined included any court contact, felony or misdemeanor, felony, personal offense, and commitment to a secure facility

CPAI Data Continued Total composite score significantly correlated with all outcome measures. Each individual domain of the CPAI also significantly correlated with all of the outcomes Program Implementation Client Assessment Program Characteristics Staff Quality Evaluation

More Fidelity Research Landenberger and Lipsey (2005) Brand of CBT didn’t matter but quality of implementation did. Implementation defined as low dropout rate, close monitoring of quality and fidelity, and adequate training for providers. Sexton (2001) Direct linear relationship between staff competence and recidivism reductions. Schoenwald et al. (2003) Therapist adherence to the model predicted post-treatment reductions in problem behaviors of the clients. Henggeler et al. (2002) Supervisors’ expertise in the model predicted therapist adherence to the model.

More Fidelity Research Cont’d. Schoenwald and Chapman (2007) A 1-unit increase in therapist adherence score predicted 38% lower rate of criminal charges 2 years post-treatment A 1-unit increase in supervisor adherence score predicted 53% lower rate of criminal charges 2 years post-treatment.

Washington State Example (Barnoski, 2004) For each program (FFT and ART), an equivalent comparison/control group was created Felony recidivism rates were calculated for each of three groups, for each of the programs Youth who received services from therapists deemed ‘competent’ Youth who received services from therapists deemed ‘not competent’ Youth who did not receive any services (control group)

Functional Family Therapy Results: % New Felony Results calculated using multivariate models in order to control for potential differences between groups

Washington State Study Continued When FFT was delivered competently, the program reduced felony recidivism by 38% When considering how much the program costs, substantial savings in ‘avoided crime’ were observed – particularly for the competent therapists When ART was competently delivered, felony recidivism was reduced by 24% Also resulted in substantial savings

Project Greenlight Short-term prison-based reentry program in New York CBT Skills Training Employment Services Housing Services Drug Education and Awareness Family Counseling Practical Skills Training Community-Based Networks Familiarity With Parole Individualized Release Plans

Project Greenlight Benefits Participants received more service referrals Participants reported more contacts with community services after release Participants demonstrated significantly more familiarity with parole conditions Participants were more positive about parole

But Did It Work?

What Went Wrong? Violation of the risk principle Ceased use of risk assessment instrument when staff deemed process too cumbersome Violation of the need principle All offenders received same services whether needed or not Violation of the fidelity principle Staff modified delivery of the CBT curriculum (shortened the duration, increased frequency, increased class size) Differential staff competence Certain case managers produced worse outcomes

2010 UC Halfway House/CBCF Study in Ohio: Adherence to CBT in Groups and Changes in Recidivism

Fidelity and the Validity of the LSI-R Flores, Lowenkamp, Holsinger, & Latessa, (2006) Higher correlations among scores produced by trained staff and future incarceration Lower correlations among scores produced by untrained staff and future incarceration Higher correlations among agencies using the LSI-R for at least 3 years and future incarceration What are the implications of these findings?

What Do We Know About Fidelity? Fidelity is related to successful outcomes (i.e., reductions in recidivism, relapse, and MH instability). Poor fidelity can lead to null effects or even iatrogenic effects. Fidelity cannot be assumed Fidelity can be measured and monitored.

Relationship Between Evaluation and Treatment Effect (based on UC Halfway House and CBCF study)

NPC Research on Drug Courts Significant at p<.05

Opportunities for Assessing Fidelity Objective 2 Opportunities for Assessing Fidelity

Opportunities to Monitor Fidelity Training Assessments Treatment groups Individual sessions Case Management Milieu Documentation Review Program Assessments

Ensuring Training Transfer Use of knowledge-based pre/post-tests Use of knowledge-based proficiency tests Use of skill-based rating upon completion of training Mechanism for use of data Staff must meet certain criteria or score to be deemed competent. Failure to meet criteria results in consequent training, supervision, etc.

Assessments Desktop Reviews Observations Accurate scores Reviews of overrides Integration with service plan/dosage Observations Use of standardized audit sheet Assess interviewing skills Assess accuracy of item ratings

Treatment Groups Observation-based ratings of adherence to treatment model. CBT: Frequency of role-plays Structure of role-plays Appropriateness of role-plays Use of behavioral reinforcers Effective use of authority and disapproval Teaches the thought-behavior chain Teaches structured skill building Follows curriculum

Individual Sessions Observation-based ratings of adherence to treatment model. CBT: Teaches thought-behavior chain Teaches problem-solving Teaches structured skill building Conducts role-plays Appropriate use of thinking reports/homework Graduated practice Appropriate use of reinforcers

Case Management Observation-based ratings of adherence to treatment model. CBT: Teaches thought-behavior chain Teaches problem-solving Teaches structured skill building Appropriate use of reinforcers Helps client to integrate skills learned into real world environment (e.g., employment)

Milieu Observation-based ratings of competence in core correctional practices Focus is more on effective use of authority and disapproval and appropriate use of reinforcers and sanctions. Standardized list of behavioral indicators Structure for observing and rating staff interacting with clients in milieu Can also review incident data for trends

Documentation Review: Why Do It? Clinical Implications Documentation is not separate from service delivery. Did the client receive the services he/she needed? Operational Implications Good documentation should drive decision-making. Means of communication Risk Management Implications If it isn’t documented, it didn’t happen. Permanent record of what occurred in the facility Source of Staff Training Reflection of the provider and organization’s competency: EBP Outcome of care

Program Assessments Correctional Program Checklist (CPC) Correctional Program Assessment Inventory (CPAI) ICCA Treatment Survey

Sample Measures Percentage of groups containing role-plays Percentage of successful completers receiving appropriate dosage based on risk/needs assessment Percentage of staff achieving 4:1 ratio Percentage of groups observed where staff modeled the skill prior to having clients engage in role-play Percentage of role-plays containing practice of the correctives Percentage of role-plays that required observers to identify skill steps and report back to the group

Resources Required to Monitor and Improve Fidelity Objective 3 Resources Required to Monitor and Improve Fidelity

Observation-Based Ratings Creation of audit sheets Schedule for conducting the reviews Staff qualified to conduct and rate the observations Time for staff to conduct observations Mechanism to record and use the data Supervision and individual staff development QI and training initiatives

Documentation Review Staff to conduct the review Schedule for review rotation Audit sheet Time to conduct the review Mechanism for recording and using the data Action planning

Barriers to Monitoring Fidelity Objective 4 Barriers to Monitoring Fidelity

Common Barriers Strength of conceptual understanding of the EBP to be measured Resources Setting priorities Understanding/skill sets required for measurement Conflicting philosophies (helper vs. evaluator) Time!

Potential Strategies Start small For example, desk top review of assessments versus observation-based ratings Use technology to increase efficiencies For example, videotape interactions for observation-based ratings Take the time to build expertise Train on model Train on evaluation methodology Insure understanding of purpose (e.g., QI versus punishment)

Planning for Implementation Objective 5 Planning for Implementation

Considerations What services do you say you deliver? What does your contract say? What do referral sources expect? List all programming components What is the model? What curricula are in use? Identify which component of service delivery is most important to monitor at current time Make a selection for measurement Identify the specific component of the model to evaluate (e.g., CBT groups in a CBT model)

Considerations Investigate possible methods for measuring the component you have selected Look for existing tools Guidelines for measurement tools: Scale should adequately sample the critical ingredients of the EBP Need to be able to differentiate between programs/staff that follow the model versus those that don’t Scale should be sensitive enough to detect progress over time Need to identify who will be observing, measuring, and documenting Need to identify how results will be calculated For example, if a percentage, what is numerator and what is denominator Define frequency of observations and who will be observed

Considerations Decide how data will be coded, stored, aggregated, reported Determine responsible parties for all parts of the process Determine how data are to be used (e.g., are there expectations that program staff will document improvement plans) Agree on common definitions for key terminology contained in measurement tools (e.g., modeling) Determine training needs for assigned evaluators

Considerations Create training package for all staff involved To address with all staff: Defining fidelity, outlining importance, details of fidelity initiative, who’s impacted and how, expectations for all staff To address with evaluators: Clarify role of evaluation and how this might differ from their assigned operational role in the program Define all terminology contained in measurement tools/process Agree on what constitutes evidence that staff are utilizing the model correctly Specify how they are to document results and any responsibilities they have for data coding, storage, etc.

Conclusions Many agencies are allocating resources to selection/implementation of EBP with no evidence that staff are adhering to the model. There is evidence that fidelity directly affects client outcomes. There is evidence that internal evaluation processes directly affect client outcomes. Therefore, agencies have an obligation to routinely assess and assure fidelity to EBP’s. Requires a formal infrastructure to routinely monitor fidelity performance.

Questions and Answers Contact Information: kimberly.sperber@talberthouse.org