1. Chestnut Health Systems Bloomington-Normal, IL
Joint Meeting on Adolescent Treatment Effectiveness Washington, DC – April 10, 2012
Pay-for-performance as a method to improve delivery of high-quality care: Results from the Reinforcing Therapist Performance (RTP) experiment
Bryan R. Garner, Susan H. Godley, Michael L. Dennis, Brooke D. Hunter, Christin M. L. Bair, & Mark D. Godley
Chestnut Health Systems Bloomington-Normal, IL
R01AA (PI: Garner)
Contract #
Opinions are those of the authors and not official positions of the government

2. Learning Objectives
To learn more about…
Methods to improve the implementation of evidence-based practices
The effectiveness of pay-for-performance methods
The relationship between performance measures and client outcomes

3. Group of treatment organizations that are:
AAFT as the foundation of the Reinforcing Therapist Performance (RTP) Experiment
AAFT
Group of treatment organizations that are:
Receiving the same level of funding.
Implementing the same evidence-based treatment.
Receiving the same comprehensive training model.
Documenting treatment implementation using the same web-based program.
Documenting client outcomes using the same evidence-based assessment instrument.

4. Monitor Clinician Fidelity
A-CRA Clinician Training and Certification Process
AAFT
Participant in Coaching Calls with Supervisors/Coaches
Feedback
Monitor Clinician Fidelity
Treatment
Manual
and
Training
Workshop
Record
Therapy
Sessions
Achieve Basic
A-CRA Certification
In 9 procedures
Let me quickly walk you through the process
Adapted from Godley, Garner, Smith, Meyers, & Godley (2011)

5. + A-CRA Clinician Training and Certification Process
AAFT +
A-CRA Treatment Manual
A-CRA Training Workshop

6. Monitor Clinician Fidelity
A-CRA Clinician Training and Certification Process
AAFT
Participant in Coaching Calls with Supervisors/Coaches
Feedback
Monitor Clinician Fidelity
Treatment
Manual
and
Training
Workshop
Record
Therapy
Sessions
Achieve Basic
A-CRA Certification
In 9 procedures

7. Digital Session Recordings (DSRs)
A-CRA Clinician Training and Certification Process
AAFT
Digital Session Recordings (DSRs)

8. Upload DSR to EBTx.org website

9. Monitor Clinician Fidelity
A-CRA Clinician Training and Certification Process
AAFT
Participant in Coaching Calls with Supervisors/Coaches
Feedback
Monitor Clinician Fidelity
Treatment
Manual
and
Training
Workshop
Record
Therapy
Sessions
Achieve Basic
A-CRA Certification
In 9 procedures

10. How might we improve the process?
Improving the Implementation Process within AAFT
AAT 1:
First Year
AAFT 1:
Second Year
Third Year
15 sites funded in October 2006
AAFT 2:
First Year
Second Year
Third Year
17 sites funded in October 2007
CSAT can’t fund research. Only services.
NIH funds research!
How might we improve the process?

11. Rewarding Provider Performance: Aligning Incentives in Medicare (IOM, 2007)
Recommended Pay-for-Performance (P4P) as a promising method to improve the delivery of high quality of care.
“The literature evaluating the effectiveness of pay for performance consists of fewer than 20 studies, yielding mixed conclusions on overall impact.”

12. How might we improve the process?
Improving the Implementation Process within AAFT
RTP study funded in
October 2008
AAFT 1:
First Year
Second Year
Third Year
15 sites funded in October 2006
AAFT 2:
First Year
Second Year
Third Year
17 sites funded in October 2007
CSAT can’t fund research. Only services.
NIH funds research!
How might we improve the process?

13. Aims and Design of RTP Experiment
AAFT
RTP
Aims: Examine the effectiveness and cost-effectiveness of providing monetary incentives to therapists as a method to improve treatment service delivery and subsequent treatment outcomes.
Design: Randomly assigned AAFT grantees and their therapists to either: ) Implementation-As-Usual (IAU) or 2) Pay-For-Performance (P4P)
$50 for each month that a randomly selected digital session recording (DSR) has at least one A-CRA procedure rated at or above the minimum level of competence required for A-CRA certification.
$200 for each adolescent who they deliver an empirically derived level of A-CRA treatment that has been shown to be significantly related improved treatment outcomes (Garner et al., 2009, 2010).
Targets: Monthly A-CRA Competence Target A-CRA

14. RTP hypotheses to be tested
Assignment to Experimental
Pay-For-Performance
Intervention + + +
A-CRA
Competence
Treatment Effectiveness
Treatment Implementation Performance Measures
Target A-CRA
6-month
Remission
Status

15. Sample size & Data structure
Level 3: Sites 1 2 3 4 5 6 7 8 9
Therapist 1
Site 1
Therapist 3
Therapist 2 …
Site 29
Level 2: Therapists …
Therapist 104
Therapist 105
Level 1: Clients …
982
983
984
985
986

16. Therapist Characteristics (N=105)
% or Mean (SD)
IAU
(n = 52)
P4P
(n = 53)
Therapists (N=105)
Female
67%
81%
White
58%
51%
Age
37.2 (11.7)
36.0 (10.7)
Master’s degree or higher
60%
49%
Months experience
60.3 (80.2)
35.9 (37.5)a
In Recovery 6%
Mini-IPIP personality factorsb
Extraversion
3.6 (0.73)
3.5 (0.77)
Agreeableness
4.4 (0.47)
4.4 (0.50)
Conscientiousness
3.9 (0.73)
4.2 (0.59)a
Neuroticism
2.5 (0.63)
2.5 (0.69)
Intellect/Imagination
4.0 (0.67)
3.9 (0.72)
a p < .05; b See Donnellan, Oswald, Baird, & Lucas (2006) for more information about
Mini-International Personality Item Pool (Mini-IPIP)

17. Client Characteristics (N=986)
% or Mean (SD)
IAU
(n = 534)
P4P
(n = 452)
Clients (N=986)
Female
22%
27%a
White
36%
28%a
Age
15.8 (1.34)
15.9 (1.47)
Criminal Justice Involved
64%
68%
Prior substance use treatment
37%
Percent-of-days-abstinent
.58 (.37)
.66 (.34)a
a p < .05;

18. Results
AAFT
RTP
Aims: Examine the effectiveness and cost-effectiveness of providing monetary incentives to therapists as a method to improve treatment service delivery and subsequent treatment outcomes.
Design: Randomly assigned AAFT grantees and their therapists to either: ) Implementation-As-Usual (IAU) or 2) Pay-For-Performance (P4P)
$50 for each month that a randomly selected digital session recording (DSR) has at least one A-CRA procedure rated at or above the minimum level of competence required for A-CRA certification.
$200 for each adolescent who they deliver an empirically derived level of A-CRA treatment that has been shown to be significantly related improved treatment outcomes (Garner et al., 2009, 2010).
Targets: Monthly A-CRA Competence Target A-CRA

19. Results: A-CRA Competence
IAU
P4P
9.2%
23.7%
Odds Ratio = 2.2 p = .02

20. Results
AAFT
RTP
Aims: Examine the effectiveness and cost-effectiveness of providing monetary incentives to therapists as a method to improve treatment service delivery and subsequent treatment outcomes.
Design: Randomly assigned AAFT grantees and their therapists to either: ) Implementation-As-Usual (IAU) or 2) Pay-For-Performance (P4P)
$50 for each month that a randomly selected digital session recording (DSR) has at least one A-CRA procedure rated at or above the minimum level of competence required for A-CRA certification.
$200 for each adolescent who they deliver an empirically derived level of A-CRA treatment that has been shown to be significantly related improved treatment outcomes (Garner et al., 2009, 2010).
Targets: Monthly A-CRA Competence Target A-CRA

21. Results: Target A-CRA
IAU
P4P
3.3%
15.9%
Odds Ratio = 5.2 p = .01

22. Results
AAFT
RTP
Aims: Examine the effectiveness and cost-effectiveness of providing monetary incentives to therapists as a method to improve treatment service delivery and subsequent treatment outcomes.
Design: Randomly assigned AAFT grantees and their therapists to either: ) Implementation-As-Usual (IAU) or 2) Pay-For-Performance (P4P)
$50 for each month that a randomly selected digital session recording (DSR) has at least one A-CRA procedure rated at or above the minimum level of competence required for A-CRA certification.
$200 for each adolescent who they deliver an empirically derived level of A-CRA treatment that has been shown to be significantly related improved treatment outcomes (Garner et al., 2009, 2010).
Targets: Monthly A-CRA Competence Target A-CRA

23. Results: Remission Status

24. No Significant Difference
Results: Remission Status ?
No Significant Difference
50.8%
Average Remission Status was 24% in Cannabis Youth Treatment (CYT) study (Dennis et al., 2004)
41.8%
IAU
P4P

25. Differential Client Attrition!
Poor Follow-up Rates
Remission
Status
Unknown
for 35%
Remission
Status
Unknown
for 44%
65%
56%
Differential Client Attrition!
IAU
P4P

26. Poor Follow-up Rates IAU P4P No Significant Significant Differences
Remission
Status
Unknown
for 35%
Remission
Status
Unknown
for 44% No
Significant
Differences
Significant
Differences
65%
56%
Follow-up
group
was
significantly
more severe
at baseline
IAU
P4P

27. Post hoc analyses Not Supported Supported Assignment to Experimental
Pay-For-Performance
Intervention
A-CRA
Competence +
Not Supported
Treatment Effectiveness
Treatment Implementation Performance Measures
Target A-CRA
6-month
Remission
Status +
Supported

28. Take home messages
Modest-sized incentives can have large impacts on improving treatment implementation
Despite the large impact of P4P on the two treatment implementation measures, there remained considerable room for improvement even in the P4P condition.

29. Next Step
Examine the extent to which the improvements in A-CRA Competence and Target A-CRA were Cost-Effective.

30. Thank You.