1. Performance Improvement Projects: From Idea to PIP
Amy McCurry Schwartz, Esq., MHSA
EQRO Consultant
Behavioral Health Concepts, Inc.
Emily Trask, Ph.D.
Senior Mental Health Consultant
San Diego County Behavioral Health
University of California, San Diego

2. Performance Improvement Projects
The purpose of PIPs
To assess and improve processes, and thereby outcomes, of care.
42 CFR (d) defines PIPs as having a “focus on clinical and non-clinical areas.”

3. Performance Improvement Projects
Clinical PIPs Might Target
Prevention and care of acute and chronic conditions
High-volume services
High-risk procedures
Special health care needs

4. Performance Improvement Projects
Non-Clinical PIPs Might Target
Coordination of Care
Appeals, Grievances Process
Access or Authorization
Member Services

5. Step 1: Select the Study Topic
Therapeutic Homework
Why is this an important topic?

6. Impact Affects a significant portion of consumers
Study Topic Impact
Affects a significant portion of consumers
Has a potentially significant impact on consumer health, functional status or satisfaction
Desire to provide client centered care
Affects youth with most presenting problems (anxiety, trauma, depression, ADHD, ODD). Therapeutic HW is considered a common element across all evidence-based practices for youth.

7. Data Sources of data used to select this topic
Nationwide summaries report large treatment effects for youth therapy (range: .76 – 1.08)
In San Diego County treatment effects for youth are small to moderate (range: for emotional / behavioral problems)
Need intervention to improve client outcomes

8. Use of HW is associated with better client outcomes nationwide and in San Diego

9. Step 2: Define the Study Question
Is training clinical staff on use of therapeutic homework associated with an increased use of HW and improvement in client outcomes (i.e., behavioral/emotional symptoms, length of time in treatment)?
Does this question support San Diego’s ability to determine whether the intervention has a measurable impact for clients?
This is the original study question…

10. Step 2: Define the Study Question
This study question should focus on the outcomes for the client:
Decreased treatment length
Improved client emotional (internalizing) outcomes
Improved client behavioral (externalizing) outcomes

11. Step 2: Define the Study Question
Better to have 2 Study Questions?
Will providing training on therapeutic homework result in a 10% increase in the use of therapeutic homework?
Will the increased use of therapeutic homework result in improved client outcomes as measured by the Child and Adolescent Measurement System (behavioral/emotional symptoms) and length of time in treatment?
These are the final questions

12. Step 3: Identify a Representative and Generalizable Study Population
San Diego
Apply HW intervention to representative pilot program
Apply HW intervention system wide

13. Step 4: Select the Study Indicator(s)
A study indicator is:
A quantitative or qualitative characteristic reflecting a discrete event or continuous status to be measured
Used to track performance and improvement over time

14. Step 4: Select the Study Indicator(s)
If goal is: Better client outcomes following HW intervention:
Indicator – “Clinicians assign HW in more sessions”
Indicator – “At discharge, clients behavioral / emotional outcomes show greater improvement”
Indicator – “Shorter length of treatment”

15. Table 1. Performance Indicator Description#
Describe Performance Indicator
Numerator
Denominator
Baseline for Performance Indicator
(number)
Goal 1
Clinicians assigned HW at all or most therapy sessions
106
267
40%
10% increase (50%) 2
Caregiver report disruptive behavior outcomes discharge score (PCAMS Externalizing Score)
Sum of PCAMS externalizing scores = 31,488
Total # of clients with intake and discharge PCAMS = 1,035
Mean = 30.42
10% reduction (mean score of 27 on PCAMS discharge externalizing scale) 3
Caregiver report depression / anxiety outcomes discharge score (PCAMS Internalizing Score)
Sum of PCAMS internalizing scores = 15,035
Total # of clients with intake and discharge PCAMS = 1034
Mean = 14.54
10% reduction (mean score of 13 on PCAMS discharge internalizing scale) 4
Treatment Length (Close date – open date)
13,160 months
3375 clients in
Q4 FY
Mean = 3.9 months
3.5 months = 10% reduction in average treatment length
Focus on indicators and baseline on this slide.

16. Step 5: Sampling
If sampling applies here – is sampling technique reliable and valid?
Probability sampling
Non-probability sampling

17. Step 6: Study Design & Data Collection
Data must be valid and reliable.
Valid – indication of accuracy
The data measure what is intended to be measured
Reliable – indication of repeatability
Produces consistent results

18. Step 6: Study Design & Data Collection
Develop a data collection plan:
Clearly identify data to be collected
Identify data sources and how/when the baseline and repeat indicator data will be collected
Specify who will collect the data and that they are qualified to collect the data
Identify the data collection instruments to be used

19. Step 6: Study Design & Data Collection
HW Baseline data
This may be an indicator of homework usage…but, is it an outcome?
How was this data collected?
Online clinician survey
Outpatient programs
Who collected this data?
Mental health research consultants
Plan to collect data for this and other indicators in the future?
Therapeutic HW usage in past month: 40% of clinicians use HW in “most” or “all” outpatient therapy sessions with youth

20. Step 6: Study Design & Data Collection
Child and Adolescent Services Measurement System (CAMS)
Normed
Psychometrically sound
Given to all youth at intake and discharge
Administered by clinicians
Measures emotional and behavioral problems

21. Step 6: Study Design & Data Collection
Average treatment length
Billing data
Calculated by subtracting the intake date from the discharge date
Clinician enters data
Entered and analyzed using the same method at each timepoint

22. Step 7: Implement Intervention and Improvement Strategies
Interventions should:
Relate to causes/barriers identified through data analysis
Be culturally and linguistically appropriate
Be “implementable” system-wide
Not be “one time” efforts

23. Step 7: Implement Intervention and Improvement Strategies
Phase I. Systemwide Educational Handout
Adding use of HW to medical records review
Presentations to program managers and other stakeholder meetings
ed educational handout systemwide
Online clinician follow-up survey on HW use
Preparation for phase II: trained pilot program to track and enter HW data

24. Step 7: Implement Intervention and Improvement Strategies
Phase II. Pilot Supervisor Training on Therapeutic HW
Provision of half-day supervisor training
Summarize attendee feedback and revise training based on their suggestions
Collect CAMS, treatment length, and HW usage data
Evaluate change in HW usage and client outcomes 6 months after training

25. Step 7: Implement Intervention and Improvement Strategies
Phase III. Pilot Clinician Training on Therapeutic HW
Provision of half-day clinician training
Summarize attendee feedback and revise training based on their suggestions
Collect CAMS, treatment length, and HW usage data
Evaluate change in HW usage and client outcomes 6 months after training
Decide how to roll-out system-wide (e.g., web-based, just provide to supervisors, follow-up consultation)
Phase IV. System-wide trainings offered

26. Step 8: Analyze Data and Interpret Study Results
Valid interpretations occur when:
Analysis is conducted according to the data analysis plan
Results present numerical data that is accurate, clear and easily understood
Analysis addresses the comparability of initial and repeat measurements, including factors that threaten validity
A conclusion is drawn as to the success of the PIP and planned follow-up activities are described

27. Table 2. Performance Indicator Data Analyses
Date of Baseline Measurement
Baseline Measurement
(numerator/
denominator)
Goal for % Improvement
Intervention Applied & Date
Date of
Re-measurement
Results+
(Numerator/
Denominator)
% Improvement Achieved
1. Clinicians assigned HW at all or most therapy sessions
May 2016
106 / 267 = 40%
10%
July 2016
September 2016 (different sample of clinicians)
70 / 175 = 40% 0%
2. Caregiver report disruptive behavior outcomes discharge score (PCAMS Externalizing Score) Q4 FY
31,488 / 1035 = 30.42
10% = 27
Q1 FY
27,594 / 916 = 30.12 1%

28. Table 2. Performance Indicator Data Analyses
Date of Baseline Measurement
Baseline Measurement
(numerator/
denominator)
Goal for % Improvement
Intervention Applied & Date
Date of
Re-measurement
Results+
(Numerator/
Denominator)
% Improvement Achieved
3. Caregiver report depression/ anxiety outcomes discharge score (PCAMS Internalizing Score) Q4 FY
15,035 / 1034 = 14.54
10% = 13
July 2016
Q1 FY
13,461 / 915 = 14.71 0%
4. Average Treatment Length (Close date – open date)
13,160 months / 3375 clients = 3.9 months
3.5 months
16,311 / 4003 =
4.07 months

29. Step 9: Plan for “Real” Improvement
When a change in performance occurs, determine whether the change is real, attributable to an event unrelated to the intervention, or random chance
Results of the intervention must be “statistically significant”

30. Step 10: Achieve Sustained Improvement
If real change has occurred, the project should be able to achieve sustained improvement.
Sustained improvement is demonstrated through repeated measurements over time.
We'll be measuring change after phase I, II, III. More frequently is not possible because we are collecting data for clients that intake and discharge from a program after each phase. Is there anything you want added here?