1. Ensuring Effective Treatment and Supervision
Operationalizing Fidelity in Real World Settings
Kimberly Gentry Sperber, Ph.D.

2. Workshop Objectives Review literature on fidelity
Identify opportunities for assessing fidelity
Identify resources for monitoring and improving fidelity
Address barriers to monitoring fidelity
Begin to think through implementation plan

3. What Does Science Say About Fidelity?
Objective 1
What Does Science Say About Fidelity?

4. Latessa, Cullen, and Gendreau (2002)
Article notes 4 common failures of correctional programs:
Failure to use research in designing programs
Failure to follow appropriate assessment and classification practices
Failure to use effective treatment models
Failure to evaluate what we do

5. The Role of Fidelity Lowenkamp and Latessa (2005)
Examined data from 38 residential correctional programs for adults
Looked at relationship between program fidelity and program effectiveness.
Program fidelity was assessed using the CPAI.
Found significant correlation between fidelity and effectiveness
CPAI scores correlated to reincarceration

6. Lowenkamp and Latessa Findings Continued
Differences in recidivism rates based on CPAI scores:
Scores of 0-49% demonstrated 1.7% reduction compared to comparison group.
Scores of 50-59% demonstrated 8.1% reduction.
Scores of 60-69% demonstrated 22% reduction.

7. CPAI Data Continued Holsinger (1999)
Examined data from Adolescent Community Correctional Facilities in Ohio
Looked at relationship between program fidelity and program effectiveness.
Program fidelity was assessed using the CPAI.
Outcome measures examined included any court contact, felony or misdemeanor, felony, personal offense, and commitment to a secure facility

8. CPAI Data Continued
Total composite score significantly correlated with all outcome measures.
Each individual domain of the CPAI also significantly correlated with all of the outcomes
Program Implementation
Client Assessment
Program Characteristics
Staff Quality
Evaluation

9. More Fidelity Research
Landenberger and Lipsey (2005)
Brand of CBT didn’t matter but quality of implementation did.
Implementation defined as low dropout rate, close monitoring of quality and fidelity, and adequate training for providers.
Sexton (2001)
Direct linear relationship between staff competence
and recidivism reductions.
Schoenwald et al. (2003)
Therapist adherence to the model predicted post-treatment reductions in problem behaviors of the clients.
Henggeler et al. (2002)
Supervisors’ expertise in the model predicted therapist adherence to the model.

10. More Fidelity Research Cont’d.
Schoenwald and Chapman (2007)
A 1-unit increase in therapist adherence score predicted 38% lower rate of criminal charges 2 years post-treatment
A 1-unit increase in supervisor adherence score predicted 53% lower rate of criminal charges 2 years post-treatment.

11. Washington State Example (Barnoski, 2004)
For each program (FFT and ART), an equivalent comparison/control group was created
Felony recidivism rates were calculated for each of three groups, for each of the programs
Youth who received services from therapists deemed ‘competent’
Youth who received services from therapists deemed ‘not competent’
Youth who did not receive any services (control group)

12. Functional Family Therapy Results: % New Felony
Results calculated using multivariate models in order to control for potential differences between groups

13. Washington State Study Continued
When FFT was delivered competently, the program reduced felony recidivism by 38%
When considering how much the program costs, substantial savings in ‘avoided crime’ were observed – particularly for the competent therapists
When ART was competently delivered, felony recidivism was reduced by 24%
Also resulted in substantial savings

14. Project Greenlight Short-term prison-based reentry program in New York
CBT Skills Training
Employment Services
Housing Services
Drug Education and Awareness
Family Counseling
Practical Skills Training
Community-Based Networks
Familiarity With Parole
Individualized Release Plans

15. Project Greenlight Benefits
Participants received more service referrals
Participants reported more contacts with community services after release
Participants demonstrated significantly more familiarity with parole conditions
Participants were more positive about parole

16. But Did It Work?

17. What Went Wrong? Violation of the risk principle
Ceased use of risk assessment instrument when staff deemed process too cumbersome
Violation of the need principle
All offenders received same services whether needed or not
Violation of the fidelity principle
Staff modified delivery of the CBT curriculum (shortened the duration, increased frequency, increased class size)
Differential staff competence
Certain case managers produced worse outcomes

18. 2010 UC Halfway House/CBCF Study in Ohio: Adherence to CBT in Groups and Changes in Recidivism

19. Fidelity and the Validity of the LSI-R
Flores, Lowenkamp, Holsinger, & Latessa, (2006)
Higher correlations among scores produced by trained staff and future incarceration
Lower correlations among scores produced by untrained staff and future incarceration
Higher correlations among agencies using the LSI-R for at least 3 years and future incarceration
What are the implications of these findings?

20. What Do We Know About Fidelity?
Fidelity is related to successful outcomes (i.e., reductions in recidivism, relapse, and MH instability).
Poor fidelity can lead to null effects or even iatrogenic effects.
Fidelity cannot be assumed
Fidelity can be measured and monitored.

21. Relationship Between Evaluation and Treatment Effect (based on UC Halfway House and CBCF study)

22. NPC Research on Drug Courts
Significant at p<.05

23. Opportunities for Assessing Fidelity
Objective 2
Opportunities for Assessing Fidelity

24. Opportunities to Monitor Fidelity
Training
Assessments
Treatment groups
Individual sessions
Case Management
Milieu
Documentation Review
Program Assessments

25. Ensuring Training Transfer
Use of knowledge-based pre/post-tests
Use of knowledge-based proficiency tests
Use of skill-based rating upon completion of training
Mechanism for use of data
Staff must meet certain criteria or score to be deemed competent.
Failure to meet criteria results in consequent training, supervision, etc.

26. Assessments Desktop Reviews Observations Accurate scores
Reviews of overrides
Integration with service plan/dosage
Observations
Use of standardized audit sheet
Assess interviewing skills
Assess accuracy of item ratings

27. Treatment Groups
Observation-based ratings of adherence to treatment model.
CBT:
Frequency of role-plays
Structure of role-plays
Appropriateness of role-plays
Use of behavioral reinforcers
Effective use of authority and disapproval
Teaches the thought-behavior chain
Teaches structured skill building
Follows curriculum

28. Individual Sessions
Observation-based ratings of adherence to treatment model.
CBT:
Teaches thought-behavior chain
Teaches problem-solving
Teaches structured skill building
Conducts role-plays
Appropriate use of thinking reports/homework
Graduated practice
Appropriate use of reinforcers

29. Case Management
Observation-based ratings of adherence to treatment model.
CBT:
Teaches thought-behavior chain
Teaches problem-solving
Teaches structured skill building
Appropriate use of reinforcers
Helps client to integrate skills learned into real world environment (e.g., employment)

30. Milieu
Observation-based ratings of competence in core correctional practices
Focus is more on effective use of authority and disapproval and appropriate use of reinforcers and sanctions.
Standardized list of behavioral indicators
Structure for observing and rating staff interacting with clients in milieu
Can also review incident data for trends

31. Documentation Review: Why Do It?
Clinical Implications
Documentation is not separate from service delivery.
Did the client receive the services he/she needed?
Operational Implications
Good documentation should drive decision-making.
Means of communication
Risk Management Implications
If it isn’t documented, it didn’t happen.
Permanent record of what occurred in the facility
Source of Staff Training
Reflection of the provider and organization’s competency:
EBP
Outcome of care

32. Program Assessments Correctional Program Checklist (CPC)
Correctional Program Assessment Inventory (CPAI)
ICCA Treatment Survey

33. Sample Measures Percentage of groups containing role-plays
Percentage of successful completers receiving appropriate dosage based on risk/needs assessment
Percentage of staff achieving 4:1 ratio
Percentage of groups observed where staff modeled the skill prior to having clients engage in role-play
Percentage of role-plays containing practice of the correctives
Percentage of role-plays that required observers to identify skill steps and report back to the group

34. Resources Required to Monitor and Improve Fidelity
Objective 3
Resources Required to Monitor and Improve Fidelity

35. Observation-Based Ratings
Creation of audit sheets
Schedule for conducting the reviews
Staff qualified to conduct and rate the observations
Time for staff to conduct observations
Mechanism to record and use the data
Supervision and individual staff development
QI and training initiatives

36. Documentation Review Staff to conduct the review
Schedule for review rotation
Audit sheet
Time to conduct the review
Mechanism for recording and using the data
Action planning

37. Barriers to Monitoring Fidelity
Objective 4
Barriers to Monitoring Fidelity

38. Common Barriers
Strength of conceptual understanding of the EBP to be measured
Resources
Setting priorities
Understanding/skill sets required for measurement
Conflicting philosophies (helper vs. evaluator)
Time!

39. Potential Strategies Start small
For example, desk top review of assessments versus observation-based ratings
Use technology to increase efficiencies
For example, videotape interactions for observation-based ratings
Take the time to build expertise
Train on model
Train on evaluation methodology
Insure understanding of purpose (e.g., QI versus punishment)

40. Planning for Implementation
Objective 5
Planning for Implementation

41. Considerations What services do you say you deliver?
What does your contract say?
What do referral sources expect?
List all programming components
What is the model?
What curricula are in use?
Identify which component of service delivery is most important to monitor at current time
Make a selection for measurement
Identify the specific component of the model to evaluate (e.g., CBT groups in a CBT model)

42. Considerations
Investigate possible methods for measuring the component you have selected
Look for existing tools
Guidelines for measurement tools:
Scale should adequately sample the critical ingredients of the EBP
Need to be able to differentiate between programs/staff that follow the model versus those that don’t
Scale should be sensitive enough to detect progress over time
Need to identify who will be observing, measuring, and documenting
Need to identify how results will be calculated
For example, if a percentage, what is numerator and what is denominator
Define frequency of observations and who will be observed

43. Considerations
Decide how data will be coded, stored, aggregated, reported
Determine responsible parties for all parts of the process
Determine how data are to be used (e.g., are there expectations that program staff will document improvement plans)
Agree on common definitions for key terminology contained in measurement tools (e.g., modeling)
Determine training needs for assigned evaluators

44. Considerations Create training package for all staff involved
To address with all staff:
Defining fidelity, outlining importance, details of fidelity initiative, who’s impacted and how, expectations for all staff
To address with evaluators:
Clarify role of evaluation and how this might differ from their assigned operational role in the program
Define all terminology contained in measurement tools/process
Agree on what constitutes evidence that staff are utilizing the model correctly
Specify how they are to document results and any responsibilities they have for data coding, storage, etc.

45. Conclusions
Many agencies are allocating resources to selection/implementation of EBP with no evidence that staff are adhering to the model.
There is evidence that fidelity directly affects client outcomes.
There is evidence that internal evaluation processes directly affect client outcomes.
Therefore, agencies have an obligation to routinely assess and assure fidelity to EBP’s.
Requires a formal infrastructure to routinely monitor fidelity performance.

46. Questions and Answers
Contact Information: