1. Data Entry PEI Outcomes Measures Application
John J. Flynn
Odre Miller
LAC - DMH, MHSA Implementation Unit
Barbara Filkins
WRMA

2. Learning Objectives Learn about PEI Outcomes history & purpose
Discuss how to organize and check PEI Outcome questionnaire scores from Clinicians
Learn how to enter those scores in the PEI OMA computer application
Learn how to spot and fix problems with PEI Outcome Measures Data
Learn about PEI Outcomes
What are they? How do they get collected and entered?
Discuss how to organize and check PEI Outcomes from Clinicians
What are the forms we’re going to see? What do we need to do to get them organized?
Learn how to enter PEI Outcome Data
How to logon and enter data…first, The “Happy Path” and then the “Rocky Road”
Learn how to spot and fix problems with PEI Outcome Measures Data
How to deal with errors…the “Rocky Road”
Handouts:
EBP List
Step-By-Step Guide…includes PEI OMA address
Sample Data: Happy Path
Sample Data: Rocky Road
Website/Egroup Information
We will provide a copy of this PowerPoint at the end of our session
But first -- a word about handouts…
Version August 9, 2013 2 2

3. The What and Why of PEI Outcomes
Outcomes are basically Questionnaires to help us understand what’s effective
State Mandated: we receive funds from the MHSA, so we need to provide Outcomes
State mandated under MHSA (Prop 63) – PEI outcomes are the second largest component of MHSA funding
Need to capture outcomes to justify the funding from the State.
How will this info be used? Our clinicians are supposed to use it to understand what’s working on an ongoing basis…it’s part of their toolkit.
The State and County will use this data to evaluate what’s working in PEI services.
Version August 9, 2013

4. Outcomes: History Full Service Partnership (2006)
Specialized Foster Care ( )
Field Capable Clinical Services (2009)
Wraparound FSP (2010)
In 2004, the voters of the State of California passed Prop 63 AKA the Mental Health Services Act AKA the tax on millionaires.
We started with the most intensive program, FSP, in And we built the Outcome Measures Application to track the outcomes from FSP.
Next was Specialized Foster Care, created because of the Katie A. lawsuit. They only collected data for 3 years, but you will still see them in the OMA
In 2009 we started collecting for FCCS: a step down in terms of intensity…but still Prop 63 $$$
Finally is Wraparound, some clients of which are FSP. We treat them just like FSP clients if they’re paid for by FSP.
Then in 2011 we started collecting for the last piece of Prop 63 money, PEI, which is another step down in intensity. We created a new program to collect that data, the PEI OMA.
Prevention and Early Intervention or PEI (2011)
Version August 9, 2013

5. The What and Why of PEI Outcomes
Your clinicians are being trained in collecting Outcomes for a variety of different Evidence Based Practices
DMH has built you a custom application to do Data Entry: the PEI Outcome Measures Application
(Read Slide)
The State and County will use this data to evaluate what’s working in PEI services.
Version August 9, 2013

6. What are PEI Outcome Measures?
I am not a clinician…I teach this class from the perspective of the data/admin staffer. All these EBP’s are a mystery to me on a certain level…just so many forms coming at me.
At larger agencies, clerical staff are going to be getting paperwork on all sorts of different measures…basically, different kinds of assessments.
These assessments need to be SORTED and checked out…you’re going to need to think of this in terms of two basic steps: SORTING/CHECKING the data and ENTERING the data.
COMPLICATED!
It’s up to us to sort them out!
Version 2.99 – April 7, 2014

7. The PEI Roadmap…
This complicated little table tells you all about the different programs and assessments that we’re working with. You won’t have ALL of these at your agency, but you might have 4 or 5 of them. It’s a handy little cheat sheet to help you organize what you will be entering into the application.
FOCUS OF TREATMENT – this is a general category of services your agency’s providing
EBP’s or Evidence Based Practices – these are the techniques your clinicians are using with clients
GENERAL and SPECIFIC Outcome Measures…we have GENERAL MEASURES that apply to ALL the programs. We want to know how ALL the clients are doing in ALL the programs. We have SPECIFIC measures to pick out what’s going on with individual clients in SPECIFIC Evidence Based Practices. Each of these measures represents a different questionnaire. There are over a dozen of them. Most are very similar in how they report outcomes but each has a slightly different way of entering data – different number of fields, some have different reasons that describe how the data was collected, some use negative numbers but more on this later.
AGE: Notice that for different Age Groups your clinical staff will use different Outcome Measures. Example: If you have a 10-year-old in Trauma: CBITS, then your clinician is going to turn in a “YOQ” assessment from the Parents and two assessments called the UCLA PTSD Reaction Index (one for the parent, one for the kid). If he or she were 13…what would they turn in? One last point about AGE…the exact Questionnaires that the clinician will use is based on the client’s AGE at the time of the FIRST EBP SESSION. They will use the same set of Questionnaires throughout the treatment, even if the client has a birthday.
Version August 9, 2013

8. As if this weren’t complicated enough…
SOME EBP’s are not currently entered into the PEI OMA…we are ADDING them to the PEI OMA software
Managing and Adapting Practice (MAP)
Triple Positive Parenting (Triple P)
Trauma-Focused CBT (TF-CBT)
For MAP, PPP and TF-CBT, we are re-writing the PEI OMA… Previously, CiMH was collecting this data via spreadsheets. Their contract has ended, and we’re now revising the PEI OMA computer program to collect this data
HOLD YOUR DATA until we announce the implementation of PEI OMA’s next version.
Version August 9, 2013

9. “How are we going to organize this, Team?”
When you get back in your office, you need to meet and talk about how the paper is going to go from one place to another, you need to know WHAT you’re going to be getting (in terms of questionnaires), and what the schedule and the expectations are.
Make a point of having a staff meeting that deals with these issues! Avoid traffic jams!
Make sure you know what questionnaires you’ll be getting, what the schedule is, and what the workflow is!
Version August 9, 2013

10. PEI Outcome Workflow Version 2.97 - August 9, 2013
Clinician Meets with client, goes through one or more Questionnaires, and then scores these questionnaires. Those scores are what you’re going to be entering into the PEI Outcome Measures computer application. The first one is a PRE assessment, and there are rules about the dates we’ll talk about in a minute.
We’re going to look at these different scoring worksheets in a minute, but you should know that they should only be created or corrected by the clinicians. The way they are scored can be a little strange: the ranges are not 1 to 100, for example. In some cases the possible scores for a Questionnaire could be -16 to 240, for example. Also, some scores are calculated and converted into a “T” score…in those cases the clinician will total up the responses and look up the “T” score on a special table. In other words: some TOTALS are not totals in the normal sense…they’re not just the sum of all the scores above.
2. Next up is you: Data Entry. You’re going to need to do a quick check to make sure the form looks like it’s generally OK. Do you have the client ID? Are there blank spaces? Is it legible? Are you missing important items like Diagnosis, Date of Assessment?
3. Did you find problems? Take it back to the clinician to figure out what needs to be done. You’ll probably want to talk not just to the clinician but to your team about how to make sure the process is working correctly.
4. Finally, you’ve got data entry. If you’ve got more than one or two clients to enter, you are going to want to SORT them so that you can minimize mistakes and be more efficient. You should sort all those Questionnaires by Focus and Evidence Based Practice: this makes it much easier to enter stuff into the PEI OMA.
Now you might still encounter problems here…the OMA PEI Computer Application might give you errors because the clinician didn’t fill things in correctly. In that case, back to Step 3…consult with the clinician.
Version August 9, 2013

11. Working on Outcomes with Clinicians…
CLINICIANS will collect the assessments on their clients…they are responsible for knowing the right Questionnaires according to age, EBP, Focus of Treatment…but don’t count on it!
They are also responsible for making sure that the assessments are completely filled out…no blanks! But don’t count on it!
Finally, Clinicians are responsible for making sure everything is done in a timely manner…but don’t count on it!
DATA ENTRY staff are responsible for…DATA ENTRY! But also, for organizing and checking the Measures to make sure they make some kind of sense…starting with the date.
Version August 9, 2013

12. Dates and Deadlines: The Treatment Cycle
DEFINITION: A Treatment Cycle is a single completed EBP: from “Pre” assessment to “Post” assessment. It may include Updates.
Your team needs to do at least TWO questionnaires for each Measure: a “Pre” and a “Post”
Beginning or Pre in the 21 day window: 7 days before to 14 days after
End or Post aorund the End of Treatment… in the 21 day window: 7 days before to 14 days after
In some cases, your clinicians will also hand in UPDATES. In some cases these are required (if the treatment goes for more than 6 Months), and in some cases they will do Updates just to help track progress.
Please note: these are the dates CLINICIANS need to complete the Assessments! If you enter them LATER than two weeks after the Beginning or End…that’s OK!
Version August 9, 2013

13. Rules about Dates…21 Day Window
Pre’s are done within 21 days of the first EBP Session (7 before to 14 after)
Updates can be done any time
Post’s are done within 21 days of the last EBP Session (7 before to 14 after)
These are mostly for the clinicians, but if they get them wrong then you are going to have a hard time doing the data entry!
Version August 9, 2013

14. Rules about Dates…Birthday!
The Questionnaires that the clinician will use depends on the client’s AGE at the time of the FIRST EBP SESSION.
The system will calculate the client’s age based on their age in the Integrated System at the time of the first EBP Session.
Clinicians use the same set of Questionnaires throughout the treatment, even if the client has a birthday.
These are mostly for the clinicians, but if they get them wrong then you are going to have a hard time doing the data entry!
Important point about that second one: if you have a PRE…you have to have a POST!
Version August 9, 2013

15. What will the data you get look like?
HANDOUT: Sample Worksheet for Seeking Safety
Different clinics will be using different forms…we’re going to look at TWO in our training today.
The one on the left is an example of the optional worksheets DMH is providing to clinicians.
The one on the right is a SAMPLE of the scoring sheet for the Trauma Symptom Checklist for Young Children.
You might also get scores in a spreadsheet format…or even scrawled on lined paper! There’s no set method for providing the scores to clerical staff. Though we have provided you some forms on the PEI wiki to help you organize your data…we’ll use those for our Happy Path training, but they are totally optional.
PLEASE GET TOGETHER WITH YOUR TEAM AND DISCUSS THIS! It’s important that you all agree on how the data is going to get to you!
It depends!
Version August 9, 2013

16. What will the data you get look like?
Many providers are using spreadsheets like these to communicate scores from questionnaires…
It’s important to understand how the clinicians are going to communicate these questionnaire scores to you, because they are not all using the same worksheets.
Many providers are reporting scores to others (like CIMH) using spreadsheets…your agency might be showing you scores like THIS…
A NOTE ABOUT CIMH: please know that there are other scores and other reporting situations out there…some of your scores will be reported in PEI OMA, some of them might be sent to CIMH using a spreadsheet like this one.
Version August 9, 2013

17. Sorting: what have I got here?
First…what do we have here?
We need to sort different EBP’s
We also need to sort different PROVIDER ID’s, if you have more than one.
You should also be on the lookout for problems… forms that are missing required information, dates that are incorrect, data you can’t read (DON’T guess, ask the clinician!), missing questionnaires.
Version August 9, 2013

18. Sorting…what problems have I got here?
These problems are going to make sense a little later when we start in on the application…but for now I just want to invite you to look this over…what problems do you see?
Blanks on the Therapist ID and on Diagnosis…and there’s a date wrong here. The first Questionnaire date is in the future, and it’s outside the 14 day window…
Version August 9, 2013

19. Sorting…what problems have I got here?
My Client here is 16 years of age…
Here’s the bottom half of the form…what sorts of problems do we have here?
There’s a wrong date in the first column…and there’s an extra questionnaire. With a 16 year old, the client shouldn’t have done the OQ…the third column.
Version August 9, 2013 19

20. What if the clinician is unable to collect a questionnaire?
Clinicians do have the option of marking a “Unable to Collect” and providing a reason.
These reasons may change based on the questionnaire the clinician is answering.
This (btw) is another discussion you should have with your team as part of your initial organization – everyone should have a good idea of when UTC applies and possibly discuss some those reasons that a little ‘gray’
Version August 9, 2013

21. What do you do when you find those mistakes? Do you fix them?
If you do encounter mistakes…do you fix them?
NO! It’s up to the clinician to make any corrections to the assessment! Good clinical practice demands that you NOT alter the documents…that is the responsibility of the clinician! Put a Post-It on it, and talk to the clinician!
Version August 9, 2013

22. And now…the PEI Outcome Application…
HANDOUT: Samples for Happy Path
So now we’re going to actually get in and look at the PEI Application. This time through we’re going to look at the ideal, “Happy Path” version.
Things are not always easy, though! Things go WRONG! So after we travel the Happy Path, we’ll try the Rocky Road!
The HAPPY PATH version…
Version August 9, 2013

23. And now…the Rocky Road….
Careful!
Errors on Sheets
Bad Scores!
Missing Information!
HANDOUT: TSCYC “bad” questionnaires
Version August 9, 2013

24. Data Entry Training: Happy Path
Sample Data on correct Worksheets
Data’s already sorted
Sample Client ID’s for everyone
Training Environment:
Login as User Name: jflynn
Password is: data
HANDOUT: Step By Step Guide
Now we’re going to work through the application with clean, correct worksheets and our Step-By-Step guides.
Sample Data on correct Worksheets
Data’s already sorted
Sample Client ID’s for everyone
Login as User Name: jflynn
Password is: data
Please refer to the STEP BY STEP guide for help after this class!
Version August 9, 2013

25. Avoiding Data Entry Errors
Discussion Points:
Sort By Provider Number
Sort By Evidence Based Practice
Sort By Beginning / End of Treatment
Look out for BLANKS…especially in required fields (Diagnosis, Provider ID, Client ID #)
Look out for BAD DATES…questionnaires done MORE than 14 days out, for example
If you get stopped in Data Entry…POST-IT and talk to the clinician! Don’t just correct things!
Remember: Scoring can be confusing…it’s not just about totaling things up!
Version August 9, 2013

26. Questions or Additional Information
Outcomes Project Website
PEI Outcomes address
E-group for PEI Outcome Measures Application Alerts.
HANDOUT: PEI Website, PEI Egroup
Version August 9, 2013

27. Finally… HELP DESK: 213-351-1335 The OMA is getting updated!
Issue Description, Provider #, Focus of Treatment, EBP and Client ID
Help Desk will triage your call…Odre is our point man for PEI OMA
The OMA is getting updated!
Now a couple more things for you…
Let’s review the Data Entry Checklist
Please help us with our Training Evaluation
On the OMA update
We are currently in Phase One – what you see here
We are working on Phases Two and Three. Two is going to provide additional capabilities like the ability to correct some information without having to call the helpdesk or make a request to the PEI Outcome Team. Phase 3 will be development of reports, including some administrative reports that you can use. Planned release is within the first half of 2012 but there is no official release date as of yet.
Updates will be announced to those of you who subscribe to our Application Alerts and as well as information will be posted on the project wiki. Additional training opportunities will be announced at the time of release!
Version August 9, 2013 27