Behavioral Health Concepts, Inc. Performance Improvement Projects 101 Drug MediCal – Bringing Ideas to Successful PIP Concept Amy McCurry Schwartz, Esq., MHSA California EQRO Consultant

PIPs – 101Background The purpose of PIPs To assess and improve processes, and thereby outcomes, of the substance use disorder continuum of care. How is a PIP different from a PM (Performance Measure)? A PIP focuses on program improvement, a PM reflects current performance of an measure.

PIPs – 101 Background Clinical vs. Non-Clinical PIPs The requirement for DMC programs to do both clinical and non-clinical PIPs comes from the federal regulatory definition of a PIP. 42 CFR 438.240(d) defines PIPs as having a “focus on clinical and non-clinical areas.”

PIPs – 101 Background Clinical PIPs Might Target Prevention and care of acute and chronic SUD conditions including relapse High-volumes of services – such as 180 days or more of residential in a year High-risk procedures – detox w/mixed drugs Reducing clients leaving residential detox and not continuing treatment

PIPs – 101 Introduction Non-Clinical PIPs Might Target Coordination of Care across SUD continuum and other treatment or support resources Appeals, Grievances Processes Timely Access to Assessment & Treatment Client Services and experience of care

PIPs – 101 Introduction EQR PROTOCOL 3: OMB Approval No. 0938-0786 EQR PROTOCOL 3: VALIDATING PERFORMANCE IMPROVEMENT PROJECTS (PIPs) A Mandatory Protocol for External Quality Reviews (EQR) by the Federal Government for all managed care plans

CMS Protocol: Validating Performance Improvement Projects – Key Job of EQRO 10 Protocol Validation Activities each with important steps and processes: Identify a potential the study topic(s) Define the study question(s) Use a Representative and Generalizable study population Select the study variable(s) If sampling used, valid approach (not recommended without sophisticated stat capacity) just use one site or program

CMS Protocol: Validating Performance Improvement Projects 10 Protocol Steps Continued: Reliably collect data Analyze data and interpret study results Implement intervention/improvement strategies Plan for “real” improvement Achieve sustained improvement

Step 1: Identify the potential study topic(s) Look for topics that: Impact a significant portion of clients Get client input, data on topic/approach whenever possible Have a potentially significant impact on client SUD recovery, functional status or satisfaction Consider high-volume or high-risk conditions of the population served

Step 1: Identify the study topic(s) Data sources that may help: MediCal eligibility database DMC encounter/claims data DMC performance on standardized measures on CalOMS by program External data – Medicaid FFS, other health plans, local or national public health reports, research review of improvement efforts in this area (be sure to show this data has a direct correlation to the DMC-ODS’s population) Special studies to examine your study question itself

CALEQRO PIP Submission Tool

Step 2: Define the Study Question(s) Make the Question: Obvious Simple Answerable with the data and design you are considering

Step 2: Define the Study Question(s) State the Question in a way that: Supports the ability to determine whether the intervention has a measurable improvement impact for a clearly defined population

Step 2: Define the Study Question(s) Start small and simple (incremental approaches are easier to manage well) and then if successful you can bring to scale

Step 2: Define the Study Question(s) #1 What’s the problem? Try to define the problem as systems issues, clinical or process oriented, that relate to individual clients' treatment, outcomes, or satisfaction. IF the problem is No-Show rates: The MHP may see this problem as: Fiscal Efficiency Inconvenience Provider communication/partnership with client

Step 2: Define the Study Question(s) Study questions are measurable and manageable issues that reflect the goal of the PIP. “To assess and improve processes, and thereby outcomes, of the substance use disorder continuum of care”—it is important to the agency, the client, and to the provision of quality Substance Use health care services.

Step 2: Define the Study Question(s): Don’t write a question that asks: “Will providing pre-appointment reminder calls for all assessment appointments reduce the DMC’s cost of ASAM Assessment Services?” OR “Will providing training to all front line staff get them to enter appointments in a more timely manner into the EHR?”

Step 2: Define the Study Question(s): INSTEAD: Think or ASK? What is keeping clients from their appointments? What are the barriers to coming to the assessment appointments? Transportation Time off work or school during the clinic hours Childcare Remembering because it was too far from request Frame the PIP around these barriers. Provide interventions that mitigate these barriers.

Step 2: Define the Study Question(s) Vague Study Question Example: An “implied” study question “Is a mail intervention reminder adequate to improve SUD medication adherence over time?” or “Can we evaluate the use of appointment reminders?”

Step 2: Define the Study Question(s): Clearer Study Question Example: Will providing educational materials to doctors, counselors, and clients increase the number and percentage of clients who participate in a MAT (medication assisted treatment) clinic, and decrease the number and percentage of clients who use counseling services without MAT supports?

Step 2: Define the Study Question(s): Clearer Study Question Example: “Will reducing the number or incidents of multiple intake/assessment appointments prior to the first treatment session increase the number of clients successfully engaging in treatment?” Even better if you add a factor by which to increase (i.e. 10%, 5%...)

Step 2: Define the Study Question(s): This step is critical because it sets the framework for data collection and analysis of results. So take the time to develop a strong study question! Consult with your assigned EQRO staff to help refine the question and bring it to concept!

CALEQRO PIP Submission Tool

Step 3: Use a Representative and Generalizable Study Population Each PIP must potentially represent the entire SUD Medi-Cal population to which the study indicators apply. For Example, a youth question might be studied only at one program but have potential to improve all such programs.

Step 3: Use a Representative and Generalizable Study Population The DMC must decide whether to review data for the entire population or a single program or unit or set of that population. It is good to start small and incrementally bring to scale if the intervention works

Step 3: Use a Representative and Generalizable Study Population Reasons to use a unit, set or single program: Limited resources are available to analyze the data The intervention(s) and data collection can be very resource intensive and you do not know yet if it is effective

Step 3: Use a Representative and Generalizable Study Population Population Example: “All DMC clients ages 0-18 years with a primary or secondary diagnosis of XXXXXXX as identified through claims data”

CALEQRO PIP Submission Tool Same questions

Step 4: Select the Study Indicator(s) A study indicator is: A quantitative or qualitative characteristic reflecting a discrete event or continuous status to be measured Used to track performance and improvement over time

Step 4: Select the Study Indicator(s) Study Indicator “tips”: The study indicator should be objective and clearly defined The indicator must be “measurable” – do you have the necessary data (or can you get it)? client input recommended on these is good and many counties have client QI advisory group.

Step 4: Select the Study Indicator(s) Study Indicator “tips”: Use an existing indicator generally accepted in the SUD clinical guidelines such as ASAM or NIATX or SAMHSA or HEDIS or AHRQ, etc. Use an existing, relevant SUD best practice or guideline to develop the study indicator (s) to tracking improvements or change.

Step 4: Select the Study Indicator(s) Study Indicators Example: Clients enrolled in SUD outpatient clinic with 4 visits within 30 days of initial assessment. Access & engagement indicator Percentage of clients who complete residential detox and transition to next level of care within 14 days of discharge. Transitions in care indicator

Step 4: Select the Study Indicator(s) Study Indicators Example: Percentage of Methadone clients with 9 months or more of regular treatment visits (3 per week average) Percentage of current Methadone clients able to work at least 20 hours per week. Improved functioning indicator

Step 5: Use Sound Sampling/Target Group Techniques Proper sampling techniques are necessary to provide valid and reliable (therefore generalizable) information on the quality of care provided. CAUTION – true statistically valid sampling is difficult. At this point, consider a representative clinic or program that is small and build out unless you have statistical and data capacity.

Step 5: Use Sound Sampling Techniques Probability sampling is based on chance, it removes bias from the selection process. Examples: simple random sampling systematic random sampling Non-probability sampling is based on choice, not chance. Example: convenience sampling

Step 5: Use Sound Sampling Techniques Probability Sample: A study of how many clients received a client satisfaction survey after completion of a level of care during a specified year by randomly selecting 100 of the 1,000 of the DMC population. Non- Probability Sample: A study of the performance of an outpatient clinic by sampling all the clients that were seen in that SUD office on a specific day.

CALEQRO PIP Submission Tool Discuss why

Step 6: Reliably Collect Data Data must be valid and reliable. Valid – indication of accuracy The data measures what is intended to be measured. No other data is mixed in. An example would be good here

Step 6: Reliably Collect Data Example: Attendance may be a valid measure of engagement but not necessarily a valid measure of satisfaction with the program. Client satisfaction survey was reliable but discovered that because it wasn’t anonymous (we had them sign them) and were completed the week prior to discharge, response bias could reduce their reliability.

Step 6: Reliably Collect Data Reliable – indication of repeatability of collection method and source. Produces consistent results

Step 6: Reliably Collect Data

Step 6: Reliably Collect Data Develop a data collection plan: Clearly identify data to be collected Identify data sources and how/when the baseline and repeat indicator data will be collected Some examples that would be SUD

Step 6: Reliably Collect Data Specify who will collect the data and that they are qualified to collect the data Data collection personnel have the conceptual and organizational skills to abstract the data. Example helpful

Step 6: Reliably Collect Data Experienced clinical staff, such as SUD certified counselors, should be used to extract the appropriate data from medical records (if this is data source) to support a judgment about whether SUD clinical criteria are met. In contrast, trained assistants or medical records clerks may collect data if the abstraction involves verifying the presence of a diagnostic test or assessment report, etc. (i.e. is is present or not??)

Step 6: Reliably Collect Data Identify the data collection instruments to be used

CALEQRO PIP Submission Tool

Step 7: Analyze Data and Interpret Study Results Use the Data Analysis Plan Up Front How did the SUD identified group or program perform on the selected indicators? Many PIPs have baseline data with indicators and periodic tracking of change in indicators over a planned time period

Step 7: Analyze Data and Interpret Study Results Valid interpretations occur when: Analysis is conducted according to a well developed data analysis plan Results present numerical data that is accurate, clear and easily understood Analysis addresses the comparability of initial (or baseline) and repeat measurements, including factors that threaten validity A conclusion is drawn as to the success of the PIP and planned follow-up activities are described or refinement of the design and the PIP can be extended or continued.

CALEQRO PIP Submission Tool Goal?

Step 8: Implement Intervention and Improvement Strategies Interventions should: Relate to causes/barriers identified through data analysis – use client input Issue = No Show Rates Barrier identified = Transportation Intervention = Bus and Taxi vouchers, Gas reimbursement

Step 8: Implement Intervention and Improvement Strategies Issue = No Show Rates Barrier identified = Time off during clinic hours Offer After-hours SUD clinic Offer home visit or community Based counseling option Barrier identified = Childcare during appointment Provide childcare (vouchers or on-site)

Step 8: Implement Intervention and Improvement Strategies Interventions should: Be culturally and linguistically appropriate Be “implementable” system-wide Not be “one time” efforts

Step 8: Implement Intervention and Improvement Strategies An improvement strategy is an intervention designed to change behavior at an institutional, practitioner, or beneficiary level. Real, sustained improvement results from a continuous cycle of measuring performance and implementing system-wide improvements. (Plan, Do, Check, Act) Need examples, this is all too abstract

CALEQRO PIP Submission Tool example

Step 9: Plan for “Real” Improvement When a change in performance occurs, determine whether the change is real, attributable to an event unrelated to the intervention (s), or random chance Results of the intervention must be “statistically significant” . Consultation available to help measure this. example

CALEQRO PIP Submission Tool

Step 10: Achieve Sustained Improvement If real change has occurred, the project should be able to achieve sustained improvement. Sustained improvement is demonstrated through repeated measurements over time. Improvements can then be expanded to other clinics or other programs with data monitoring to insure they are working. example

CMS Protocols: Conducting Performance Improvement Projects and Validating Performance Improvement Projects Protocols Available Online at: http://www.medicaid.gov/Medicaid-CHIP-Program-Information/By- Topics/Quality-of-Care/Quality-of-Care-External-Quality- Review.html Click on “External Quality Review Protocols (2012 Update)”