Performance Improvement Projects: From Idea to PIP Amy McCurry Schwartz, Esq., MHSA EQRO Consultant Behavioral Health Concepts, Inc. Emily Trask, Ph.D. Senior Mental Health Consultant San Diego County Behavioral Health University of California, San Diego

Performance Improvement Projects The purpose of PIPs To assess and improve processes, and thereby outcomes, of care. 42 CFR 438.240(d) defines PIPs as having a “focus on clinical and non-clinical areas.”

Performance Improvement Projects Clinical PIPs Might Target Prevention and care of acute and chronic conditions High-volume services High-risk procedures Special health care needs

Performance Improvement Projects Non-Clinical PIPs Might Target Coordination of Care Appeals, Grievances Process Access or Authorization Member Services

Step 1: Select the Study Topic Therapeutic Homework Why is this an important topic?

Impact Affects a significant portion of consumers Study Topic Impact Affects a significant portion of consumers Has a potentially significant impact on consumer health, functional status or satisfaction Desire to provide client centered care Affects youth with most presenting problems (anxiety, trauma, depression, ADHD, ODD). Therapeutic HW is considered a common element across all evidence-based practices for youth.

Data Sources of data used to select this topic Nationwide summaries report large treatment effects for youth therapy (range: .76 – 1.08) In San Diego County treatment effects for youth are small to moderate (range: .39 - .56 for emotional / behavioral problems) Need intervention to improve client outcomes

Use of HW is associated with better client outcomes nationwide and in San Diego

Step 2: Define the Study Question Is training clinical staff on use of therapeutic homework associated with an increased use of HW and improvement in client outcomes (i.e., behavioral/emotional symptoms, length of time in treatment)? Does this question support San Diego’s ability to determine whether the intervention has a measurable impact for clients? This is the original study question…

Step 2: Define the Study Question This study question should focus on the outcomes for the client: Decreased treatment length Improved client emotional (internalizing) outcomes Improved client behavioral (externalizing) outcomes

Step 2: Define the Study Question Better to have 2 Study Questions? Will providing training on therapeutic homework result in a 10% increase in the use of therapeutic homework? Will the increased use of therapeutic homework result in improved client outcomes as measured by the Child and Adolescent Measurement System (behavioral/emotional symptoms) and length of time in treatment? These are the final questions

Step 3: Identify a Representative and Generalizable Study Population San Diego Apply HW intervention to representative pilot program Apply HW intervention system wide

Step 4: Select the Study Indicator(s) A study indicator is: A quantitative or qualitative characteristic reflecting a discrete event or continuous status to be measured Used to track performance and improvement over time

Step 4: Select the Study Indicator(s) If goal is: Better client outcomes following HW intervention: Indicator – “Clinicians assign HW in more sessions” Indicator – “At discharge, clients behavioral / emotional outcomes show greater improvement” Indicator – “Shorter length of treatment”

Table 1. Performance Indicator Description # Describe Performance Indicator   Numerator Denominator Baseline for Performance Indicator (number) Goal 1 Clinicians assigned HW at all or most therapy sessions 106 267 40% 10% increase (50%) 2 Caregiver report disruptive behavior outcomes discharge score (PCAMS Externalizing Score) Sum of PCAMS externalizing scores = 31,488 Total # of clients with intake and discharge PCAMS = 1,035 Mean = 30.42 10% reduction (mean score of 27 on PCAMS discharge externalizing scale) 3 Caregiver report depression / anxiety outcomes discharge score (PCAMS Internalizing Score) Sum of PCAMS internalizing scores = 15,035 Total # of clients with intake and discharge PCAMS = 1034 Mean = 14.54 10% reduction (mean score of 13 on PCAMS discharge internalizing scale) 4 Treatment Length (Close date – open date) 13,160 months 3375 clients in Q4 FY 2015-16 Mean = 3.9 months 3.5 months = 10% reduction in average treatment length Focus on indicators and baseline on this slide.

Step 5: Sampling If sampling applies here – is sampling technique reliable and valid? Probability sampling Non-probability sampling

Step 6: Study Design & Data Collection Data must be valid and reliable. Valid – indication of accuracy The data measure what is intended to be measured Reliable – indication of repeatability Produces consistent results

Step 6: Study Design & Data Collection Develop a data collection plan: Clearly identify data to be collected Identify data sources and how/when the baseline and repeat indicator data will be collected Specify who will collect the data and that they are qualified to collect the data Identify the data collection instruments to be used

Step 6: Study Design & Data Collection HW Baseline data This may be an indicator of homework usage…but, is it an outcome? How was this data collected? Online clinician survey Outpatient programs Who collected this data? Mental health research consultants Plan to collect data for this and other indicators in the future? Therapeutic HW usage in past month: 40% of clinicians use HW in “most” or “all” outpatient therapy sessions with youth

Step 6: Study Design & Data Collection Child and Adolescent Services Measurement System (CAMS) Normed Psychometrically sound Given to all youth at intake and discharge Administered by clinicians Measures emotional and behavioral problems

Step 6: Study Design & Data Collection Average treatment length Billing data Calculated by subtracting the intake date from the discharge date Clinician enters data Entered and analyzed using the same method at each timepoint

Step 7: Implement Intervention and Improvement Strategies Interventions should: Relate to causes/barriers identified through data analysis Be culturally and linguistically appropriate Be “implementable” system-wide Not be “one time” efforts

Step 7: Implement Intervention and Improvement Strategies Phase I. Systemwide Educational Handout Adding use of HW to medical records review Presentations to program managers and other stakeholder meetings Emailed educational handout systemwide Online clinician follow-up survey on HW use Preparation for phase II: trained pilot program to track and enter HW data

Step 7: Implement Intervention and Improvement Strategies Phase II. Pilot Supervisor Training on Therapeutic HW Provision of half-day supervisor training Summarize attendee feedback and revise training based on their suggestions Collect CAMS, treatment length, and HW usage data Evaluate change in HW usage and client outcomes 6 months after training

Step 7: Implement Intervention and Improvement Strategies Phase III. Pilot Clinician Training on Therapeutic HW Provision of half-day clinician training Summarize attendee feedback and revise training based on their suggestions Collect CAMS, treatment length, and HW usage data Evaluate change in HW usage and client outcomes 6 months after training Decide how to roll-out system-wide (e.g., web-based, just provide to supervisors, follow-up consultation) Phase IV. System-wide trainings offered

Step 8: Analyze Data and Interpret Study Results Valid interpretations occur when: Analysis is conducted according to the data analysis plan Results present numerical data that is accurate, clear and easily understood Analysis addresses the comparability of initial and repeat measurements, including factors that threaten validity A conclusion is drawn as to the success of the PIP and planned follow-up activities are described

Table 2. Performance Indicator Data Analyses Date of Baseline Measurement Baseline Measurement (numerator/ denominator) Goal for % Improvement Intervention Applied & Date Date of Re-measurement Results+ (Numerator/ Denominator) % Improvement Achieved 1. Clinicians assigned HW at all or most therapy sessions May 2016 106 / 267 = 40% 10% July 2016 September 2016 (different sample of clinicians) 70 / 175 = 40% 0% 2. Caregiver report disruptive behavior outcomes discharge score (PCAMS Externalizing Score) Q4 FY 2015-16 31,488 / 1035 = 30.42 10% = 27 Q1 FY 2016-17 27,594 / 916 = 30.12 1%

Table 2. Performance Indicator Data Analyses Date of Baseline Measurement Baseline Measurement (numerator/ denominator) Goal for % Improvement Intervention Applied & Date Date of Re-measurement Results+ (Numerator/ Denominator) % Improvement Achieved 3. Caregiver report depression/ anxiety outcomes discharge score (PCAMS Internalizing Score) Q4 FY 2015-16 15,035 / 1034 = 14.54 10% = 13 July 2016 Q1 FY 2016-17 13,461 / 915 = 14.71 0% 4. Average Treatment Length (Close date – open date) 13,160 months / 3375 clients = 3.9 months 3.5 months 16,311 / 4003 = 4.07 months

Step 9: Plan for “Real” Improvement When a change in performance occurs, determine whether the change is real, attributable to an event unrelated to the intervention, or random chance Results of the intervention must be “statistically significant”

Step 10: Achieve Sustained Improvement If real change has occurred, the project should be able to achieve sustained improvement. Sustained improvement is demonstrated through repeated measurements over time. We'll be measuring change after phase I, II, III. More frequently is not possible because we are collecting data for clients that intake and discharge from a program after each phase. Is there anything you want added here?