1. Performance Improvement Projects – Tips and Common Mistakes
Behavioral Health Concepts – CAL-EQRO for MHP and DMC-ODS
Rama Khalsa, PhD
Amy McCurry Schwartz, Esq., MHSA

2. Common PIP Mistakes Lack of facts to support the problem Lack of data
Support your position

3. Common PIP Mistakes Lack of analysis of the problem
Analyze existing data to identify “targets”

4. Common PIP Mistakes
Indicators/Interventions do not “line up” with the identified problem

5. Avoiding Common PIP Mistakes - Data
Do a thorough data review before beginning the PIP
County specific data
Review local data linked to your challenge/problem
Review system procedures and process linked to your challenge/problem
National data
Helpful background, benchmarks and possible interventions
SAMHSA, NIDA, NIMH, NIATX, AHRQ, NCQA, ASAM, NSDUH, etc

6. Avoiding Common PIP Mistakes - Facts
Identify a problem area or opportunity for improvement in treatment or process, but do not take it “to the bank” without first reviewing your facts and data
Include information from clients, families, staff and community partners
Client, families, community partner and staff input should be obtained on problem
What do they see as the barriers to better care?
Why do they think the problem is happening?
What ideas do they have to fix it?

7. Avoiding Common PIP Mistakes - Indicators
Set up a solid foundation to identify the issues leading to the problem
A special survey or focus group may be necessary to understand the full extent of the issues surrounding the problem area
Key indicators will flow from a solid analysis
Collect baseline data for your indicators
Identify effective and meaningful interventions/actions to mitigate the problem.

8. CMS Protocol: Validating Performance Improvement Projects
OMB Approval No.
EQR PROTOCOL 3:
VALIDATING PERFORMANCE IMPROVEMENT PROJECTS (PIPs)
A Mandatory Protocol for External Quality Reviews (EQR)

9. CMS Protocol: Validating Performance Improvement Projects
10 Protocol Steps:
Select the study topic(s)
Define the study question(s)
Use a Representative and Generalizable study population
Select the study variable(s)
Use sound sampling techniques

10. CMS Protocol: Validating Performance Improvement Projects
10 Protocol Steps Continued:
Reliably collect data
Analyze data and interpret study results
Implement intervention/improvement strategies
Plan for “real” improvement
Achieve sustained improvement

11. Framework Create an objective, empirical foundation for your PIP -
start the process with an exploratory analysis –
verify and quantify the problems
(when does it occur, how often, how severe is it,
what does the problem impact, etc.)
Indicators are the data by which you measure success

12. Framework Interventions are actions to impact the problem
or improvement you want to make.
Understand how they link together!
AND how they link to the study question.
Don’t make the PIP overly complex, start small and if successful bring to scale. Make the process manageable so it can be done well.

14. Assess Ask: Do we have a way to measure that?
What data do we have that would show us if there is in fact an issue?
Do we know if our issue is “out of whack” with other
similarly sized counties?

15. Assess Ask: Do we know if this is problem nationally?
What are the national averages for
this problem/issue/condition?
Is there a norm we are not meeting?
Is this a problem for our county?

16. Assess
If the MHP or DMC-ODS feels there is an issue, a baseline measurement must be established.

17. Continuous Quality Improvement
How not to Assess:
“I think we have an issue with X”.
“Oh, ok, let’s make that a PIP.”
How to begin to Assess:
“I think we have an issue with X.”
“Why do you think there is an issue?”
“Well, Dr. Jones says people are never here for their scheduled appointment.” OR
“I keep seeing the same patients back in 10 days after discharge.” OR
“Clinic X’s patients’ ANSA scores don’t seem to be improving”.
Not yet - A PIP doesn’t start here.

18. Plan
The MHP or DMC- ODS should gather input from stakeholders to explore solutions to the problem.
What barriers exist?
Can the MHP effect those barriers?
How will the MHP/ DMC-ODS measure their effect?
Not yet - A PIP hasn’t started yet.

19. Improve
Identify interventions (Do not start a PIP with the intervention and then try to make it work)
New treatment we want to try
New training we want to try
The MHP or DMC- ODS should decide what interventions they want to use to try to improve the issue after reviewing the causes and barriers.
Then put those interventions in to place.
A PIP has started.

20. Key Issues to Remember in PIP Design
*Preparation and ground work are important for PIP success – train staff on why the PIP is important and what their role is…
*Training is important but is not an intervention. The activity you are training for is the intervention, such as a new treatment program or new process to improve access.

21. Key Issues to Remember in PIP Design
*Create a baseline data set to measure against for improvement
*Measure similar time periods (like quarterly) to understand the changes occurring because of your interventions.
*A PIP IS NOT ACTIVE UNTIL ONE INTERVENTION has started.

23. EXAMPLE PIP

24. Data – There is a problem!
Average Quarterly Percent Meeting Benchmark (Adult Only)
FY 10/11
FY 11/12
FY 12/13
FY 13/14
Target %
Meet
Avg
Days
Benchmark 1 – Admit to 1st Face to Face
14 days
57.2% 20
24.5% 33
38.4% 27
23.2% 36
Benchmark 2 – Admit to 1st Psychiatric/Med Appt
28 days
44.2% 42
27.8% 52
25.2% 60
20.8% 64
Benchmark 3 - Intake to 2nd Face to Face
30 days
82.0% 18
82.3% 19
64.4%
76.3% 22
Benchmark 4 – Hospital discharge to 1st
Psychiatric/Med Appt
72.1% 29
67.5% 34
65.6%
71.6% 26
Benchmark 5 – Hospital discharge to 1st Face to Face
7 days
65.2% 13
58.4% 17 28
48.3%

25. Formulate a Study Question
Does creating a Care Coordination Team with strategies to engage and provide timely access to outpatient services increase engagement and mental health treatment and increase overall client satisfaction?

26. Baseline for Performance Indicator
Indicators
Describe Performance Indicator
Numerator
Denominator
Baseline for Performance Indicator
Goal 1
Timeliness to first face to face appointment at RST
Number of episodes where clients had their initial face-to-face service at the RST’s within 14 days of referral to
RST
Total number of referrals to the RST’s where the client received a service
24.8%
50% 2
Timeliness to first outpatient psychiatric service
Number of episodes where clients had their first outpatient psychiatric service at the RST’s within 28 days of
referral to RST
21.5% 3
Timeliness from Acute care discharge to first face- to-face outpatient appointment
Number of acute discharges with a
face-to-face service at the RST’s within 7 days of acute discharge
Number of acute discharges for RST clients
38.5%
60% 4
Timeliness from acute care discharge to first psychiatric medication appointment
Number of acute discharges with an outpatient psychiatric service at the RST’s within 30 days of acute
discharge
62.9%
80% 5
No show and cancellations rates prior to first appointments
Number of episodes where clients had
a no-show or cancellation prior to first outpatient appointment
Total number of referrals to the RST’s
3.7%
TBD

27. Baseline for Performance Indicator
Indicators
Describe Performance Indicator
Numerator
Denominator
Baseline for Performance Indicator
Goal 6
% of referred clients that engage in RST services
Number of clients referred who receive a service from the RST’s
Total number of referrals to the RST’s
62.8%
80% 7
% of clients that are hospitalized while waiting for 1st appointment (includes clients that eventually received outpatient services and clients that never
showed for services)
Number of RST clients referred who had an inpatient hospitalization in between being authorized for services
and their first face to face appointment
Number of clients referred to the RSTs
3.4% 2% 8
Level of satisfaction
Scores on the General Satisfaction domain of the CPS for RST clients meeting benchmark #1 and those not meeting benchmark #1
Total number of RST clients who completed the CPS and meet benchmark #1 and those not meeting benchmark #1
Timely Access=4.51 Not timely= 4.03
TBD 9
% of referred clients that receive engagement services prior to first face to face appointment
Number of episodes where clients had an engagement service prior to their
first appointment
10.4%
95% 10
% discharged to health provider/lower level of care for behavioral health treatment
Number of clients discharged from the RSTs to a health provider/lower level
of care
Number of clients discharge from the RSTs in the FY

28. Specific Intervention
Interventions
Number of
Intervention
Specific Intervention
Barriers/Causes Addressed
Corresponding
Indicator
Date
Applied 1
Hire additional staff to create a Community Care Team
(CCT)
Staff Resources/Capacity/
Engagement
1-10 2
RSTs to implement orientation groups to complete
intake appointments to facilitate more timely access to services
Staff Resources/Engagement
1/8
9/1/2015 3
CCT will outreach and provide engagement services to
clients prior to their first appointment
1/6/8/9
7/1/2015 4
CCT staff to call the client within a 14 days of access opening the episode 5
Increase documentation of engagement services provided prior to first appointment
Measurement/Engagement
1/6/9

29. Specific Intervention
Interventions
Number of
Intervention
Specific Intervention
Barriers/Causes Addressed
Corresponding
Indicator
Date
Applied 6
Increasing medical staff hours at each RST
Staff Resources/Engagement
2/4/8
9/1/2015 7
When the RST is notified that a client is hospitalized, the CCT/RST will engage with the client to get them into services.
Engagement
3/4/7/8 8
Increase documentation of no show and cancellations
Measurement 5
7/1/2015 9
CCT will assist stable clients to transition to a PCP or lower level of care
Capacity
8/10 10
CCT will serve as the lead contact for clients who are touched by the Navigator program in the emergency rooms and jails
1/6/8/9
10/1/2015 11
Discharge codes will be added to the EHR to track clients discharging to
lower levels of care (i.e. Primary Care and GMC providers).
Measurement/Capacity

30. The BEST PIP Concepts have….
Knowledge of what might have changed in your environment to contribute to or improve the problem (examples, opening or closing problems due to budget changes or new medications to help improve recovery for specific SUDs or diagnoses)
A clear study question based on thorough data collection and analysis of the problem and potential causes

32. HOW DO I START?

33. HELPFUL TIPS MHPs & DMC-ODS teamwork
Use the PIP library at the CalEQRO.com web site to see if other counties have similar problems and have PIPs to evaluate
Consult with county and BHC staff to adapt to your local challenges;
Consider getting outside help with using statistical significance tests if no internal resources
Give concept only draft PIP to BHC reviewer early to make sure there are no major problems.

34. HELPFUL TIPS Resources to Consider
There are many PIPs on timeliness, access, no-shows, engagement of special populations, improvements in care using quality metrics. Consult with BHC staff, and other quality resources for Behavioral Health for ideas.
Make sure you have data support to accomplish your planned PIP analysis and activities;
Picking a problem program staff care about enhances cooperation and completeness of data collection;
Do updates to staff to keep up morale and buy-in to improving the PIP problem, process or removing a barrier;

35. WHAT DO I DO IF…?

36. PIP Problems and Solutions
Sampling approach is not statistically sound and creates bias.
Data plan is not clear to staff
Importance of PIP activities not understood by stakeholders so poor cooperation with data access
Check with review staff or professional with statistical background to get consultation before using sampling
Training and buy-in from staff very important to PIP success.
Incomplete or inaccurate data can compromise PIP so this is important to clarify up from with partners and have back up plan for problems. Monitoring data quality is so important to find problems early.

37. PIP Problems and Solutions
No local data to show that the issue is a local problem not just a national one.
Need help understanding if this is good concept for PIP and how to evaluate it
Develop a local data source to confirm it is a local as well as national problem using claims, CalOMS, CSI, MMEF, Satisfaction Surveys, focus group, CHIS, etc.
Call or your assigned review leader who can help you with the process

38. PIP Problems and Solutions
Interventions not working, can I add a new intervention and extend the PIP
Data displayed in confusing fashion and not shared with stakeholders
Yes an additional or refined intervention based on learnings can extend the PIP
Display data in way that all stakeholders can participate in learnings from PIP and possible changes in strategies for improvement.

40. Contact Us Rama Khalsa, PhD Amy McCurry Schwartz, Esq., MHSA
Director Drug MediCal EQRO
CA EQRO Consultant