Data Entry PEI Outcomes Measures Application John J. Flynn Odre Miller LAC - DMH, MHSA Implementation Unit Barbara Filkins WRMA

Learning Objectives Learn about PEI Outcomes history & purpose Discuss how to organize and check PEI Outcome questionnaire scores from Clinicians Learn how to enter those scores in the PEI OMA computer application Learn how to spot and fix problems with PEI Outcome Measures Data Learn about PEI Outcomes What are they? How do they get collected and entered? Discuss how to organize and check PEI Outcomes from Clinicians What are the forms we’re going to see? What do we need to do to get them organized? Learn how to enter PEI Outcome Data How to logon and enter data…first, The “Happy Path” and then the “Rocky Road” Learn how to spot and fix problems with PEI Outcome Measures Data How to deal with errors…the “Rocky Road” Handouts: EBP List Step-By-Step Guide…includes PEI OMA address Sample Data: Happy Path Sample Data: Rocky Road Website/Egroup Information We will provide a copy of this PowerPoint at the end of our session But first -- a word about handouts… Version 2.97 - August 9, 2013 2 2

The What and Why of PEI Outcomes Outcomes are basically Questionnaires to help us understand what’s effective State Mandated: we receive funds from the MHSA, so we need to provide Outcomes State mandated under MHSA (Prop 63) – PEI outcomes are the second largest component of MHSA funding Need to capture outcomes to justify the funding from the State. How will this info be used? Our clinicians are supposed to use it to understand what’s working on an ongoing basis…it’s part of their toolkit. The State and County will use this data to evaluate what’s working in PEI services. Version 2.97 - August 9, 2013

Outcomes: History Full Service Partnership (2006) Specialized Foster Care (2007-2010) Field Capable Clinical Services (2009) Wraparound FSP (2010) In 2004, the voters of the State of California passed Prop 63 AKA the Mental Health Services Act AKA the tax on millionaires. We started with the most intensive program, FSP, in 2006. And we built the Outcome Measures Application to track the outcomes from FSP. Next was Specialized Foster Care, created because of the Katie A. lawsuit. They only collected data for 3 years, but you will still see them in the OMA In 2009 we started collecting for FCCS: a step down in terms of intensity…but still Prop 63 $$$ Finally is Wraparound, some clients of which are FSP. We treat them just like FSP clients if they’re paid for by FSP. Then in 2011 we started collecting for the last piece of Prop 63 money, PEI, which is another step down in intensity. We created a new program to collect that data, the PEI OMA. Prevention and Early Intervention or PEI (2011) Version 2.97 - August 9, 2013

The What and Why of PEI Outcomes Your clinicians are being trained in collecting Outcomes for a variety of different Evidence Based Practices DMH has built you a custom application to do Data Entry: the PEI Outcome Measures Application (Read Slide) The State and County will use this data to evaluate what’s working in PEI services. Version 2.97 - August 9, 2013

What are PEI Outcome Measures? I am not a clinician…I teach this class from the perspective of the data/admin staffer. All these EBP’s are a mystery to me on a certain level…just so many forms coming at me. At larger agencies, clerical staff are going to be getting paperwork on all sorts of different measures…basically, different kinds of assessments. These assessments need to be SORTED and checked out…you’re going to need to think of this in terms of two basic steps: SORTING/CHECKING the data and ENTERING the data. COMPLICATED! It’s up to us to sort them out! Version 2.99 – April 7, 2014

The PEI Roadmap… This complicated little table tells you all about the different programs and assessments that we’re working with. You won’t have ALL of these at your agency, but you might have 4 or 5 of them. It’s a handy little cheat sheet to help you organize what you will be entering into the application. FOCUS OF TREATMENT – this is a general category of services your agency’s providing EBP’s or Evidence Based Practices – these are the techniques your clinicians are using with clients GENERAL and SPECIFIC Outcome Measures…we have GENERAL MEASURES that apply to ALL the programs. We want to know how ALL the clients are doing in ALL the programs. We have SPECIFIC measures to pick out what’s going on with individual clients in SPECIFIC Evidence Based Practices. Each of these measures represents a different questionnaire. There are over a dozen of them. Most are very similar in how they report outcomes but each has a slightly different way of entering data – different number of fields, some have different reasons that describe how the data was collected, some use negative numbers but more on this later. AGE: Notice that for different Age Groups your clinical staff will use different Outcome Measures. Example: If you have a 10-year-old in Trauma: CBITS, then your clinician is going to turn in a “YOQ” assessment from the Parents and two assessments called the UCLA PTSD Reaction Index (one for the parent, one for the kid). If he or she were 13…what would they turn in? One last point about AGE…the exact Questionnaires that the clinician will use is based on the client’s AGE at the time of the FIRST EBP SESSION. They will use the same set of Questionnaires throughout the treatment, even if the client has a birthday. Version 2.97 - August 9, 2013

As if this weren’t complicated enough… SOME EBP’s are not currently entered into the PEI OMA…we are ADDING them to the PEI OMA software Managing and Adapting Practice (MAP) Triple Positive Parenting (Triple P) Trauma-Focused CBT (TF-CBT) For MAP, PPP and TF-CBT, we are re-writing the PEI OMA… Previously, CiMH was collecting this data via spreadsheets. Their contract has ended, and we’re now revising the PEI OMA computer program to collect this data HOLD YOUR DATA until we announce the implementation of PEI OMA’s next version. Version 2.97 - August 9, 2013

“How are we going to organize this, Team?” When you get back in your office, you need to meet and talk about how the paper is going to go from one place to another, you need to know WHAT you’re going to be getting (in terms of questionnaires), and what the schedule and the expectations are. Make a point of having a staff meeting that deals with these issues! Avoid traffic jams! Make sure you know what questionnaires you’ll be getting, what the schedule is, and what the workflow is! Version 2.97 - August 9, 2013

PEI Outcome Workflow Version 2.97 - August 9, 2013 Clinician Meets with client, goes through one or more Questionnaires, and then scores these questionnaires. Those scores are what you’re going to be entering into the PEI Outcome Measures computer application. The first one is a PRE assessment, and there are rules about the dates we’ll talk about in a minute. We’re going to look at these different scoring worksheets in a minute, but you should know that they should only be created or corrected by the clinicians. The way they are scored can be a little strange: the ranges are not 1 to 100, for example. In some cases the possible scores for a Questionnaire could be -16 to 240, for example. Also, some scores are calculated and converted into a “T” score…in those cases the clinician will total up the responses and look up the “T” score on a special table. In other words: some TOTALS are not totals in the normal sense…they’re not just the sum of all the scores above. 2. Next up is you: Data Entry. You’re going to need to do a quick check to make sure the form looks like it’s generally OK. Do you have the client ID? Are there blank spaces? Is it legible? Are you missing important items like Diagnosis, Date of Assessment? 3. Did you find problems? Take it back to the clinician to figure out what needs to be done. You’ll probably want to talk not just to the clinician but to your team about how to make sure the process is working correctly. 4. Finally, you’ve got data entry. If you’ve got more than one or two clients to enter, you are going to want to SORT them so that you can minimize mistakes and be more efficient. You should sort all those Questionnaires by Focus and Evidence Based Practice: this makes it much easier to enter stuff into the PEI OMA. Now you might still encounter problems here…the OMA PEI Computer Application might give you errors because the clinician didn’t fill things in correctly. In that case, back to Step 3…consult with the clinician. Version 2.97 - August 9, 2013

Working on Outcomes with Clinicians… CLINICIANS will collect the assessments on their clients…they are responsible for knowing the right Questionnaires according to age, EBP, Focus of Treatment…but don’t count on it! They are also responsible for making sure that the assessments are completely filled out…no blanks! But don’t count on it! Finally, Clinicians are responsible for making sure everything is done in a timely manner…but don’t count on it! DATA ENTRY staff are responsible for…DATA ENTRY! But also, for organizing and checking the Measures to make sure they make some kind of sense…starting with the date. Version 2.97 - August 9, 2013

Dates and Deadlines: The Treatment Cycle DEFINITION: A Treatment Cycle is a single completed EBP: from “Pre” assessment to “Post” assessment. It may include Updates. Your team needs to do at least TWO questionnaires for each Measure: a “Pre” and a “Post” Beginning or Pre in the 21 day window: 7 days before to 14 days after End or Post aorund the End of Treatment… in the 21 day window: 7 days before to 14 days after In some cases, your clinicians will also hand in UPDATES. In some cases these are required (if the treatment goes for more than 6 Months), and in some cases they will do Updates just to help track progress. Please note: these are the dates CLINICIANS need to complete the Assessments! If you enter them LATER than two weeks after the Beginning or End…that’s OK! Version 2.97 - August 9, 2013

Rules about Dates…21 Day Window Pre’s are done within 21 days of the first EBP Session (7 before to 14 after) Updates can be done any time Post’s are done within 21 days of the last EBP Session (7 before to 14 after) These are mostly for the clinicians, but if they get them wrong then you are going to have a hard time doing the data entry! Version 2.97 - August 9, 2013

Rules about Dates…Birthday! The Questionnaires that the clinician will use depends on the client’s AGE at the time of the FIRST EBP SESSION. The system will calculate the client’s age based on their age in the Integrated System at the time of the first EBP Session. Clinicians use the same set of Questionnaires throughout the treatment, even if the client has a birthday. These are mostly for the clinicians, but if they get them wrong then you are going to have a hard time doing the data entry! Important point about that second one: if you have a PRE…you have to have a POST! Version 2.97 - August 9, 2013

What will the data you get look like? HANDOUT: Sample Worksheet for Seeking Safety Different clinics will be using different forms…we’re going to look at TWO in our training today. The one on the left is an example of the optional worksheets DMH is providing to clinicians. The one on the right is a SAMPLE of the scoring sheet for the Trauma Symptom Checklist for Young Children. You might also get scores in a spreadsheet format…or even scrawled on lined paper! There’s no set method for providing the scores to clerical staff. Though we have provided you some forms on the PEI wiki to help you organize your data…we’ll use those for our Happy Path training, but they are totally optional. PLEASE GET TOGETHER WITH YOUR TEAM AND DISCUSS THIS! It’s important that you all agree on how the data is going to get to you! It depends! Version 2.97 - August 9, 2013

What will the data you get look like? Many providers are using spreadsheets like these to communicate scores from questionnaires… It’s important to understand how the clinicians are going to communicate these questionnaire scores to you, because they are not all using the same worksheets. Many providers are reporting scores to others (like CIMH) using spreadsheets…your agency might be showing you scores like THIS… A NOTE ABOUT CIMH: please know that there are other scores and other reporting situations out there…some of your scores will be reported in PEI OMA, some of them might be sent to CIMH using a spreadsheet like this one. Version 2.97 - August 9, 2013

Sorting: what have I got here? First…what do we have here? We need to sort different EBP’s We also need to sort different PROVIDER ID’s, if you have more than one. You should also be on the lookout for problems… forms that are missing required information, dates that are incorrect, data you can’t read (DON’T guess, ask the clinician!), missing questionnaires. Version 2.97 - August 9, 2013

Sorting…what problems have I got here? These problems are going to make sense a little later when we start in on the application…but for now I just want to invite you to look this over…what problems do you see? Blanks on the Therapist ID and on Diagnosis…and there’s a date wrong here. The first Questionnaire date is in the future, and it’s outside the 14 day window… Version 2.97 - August 9, 2013

Sorting…what problems have I got here? My Client here is 16 years of age… Here’s the bottom half of the form…what sorts of problems do we have here? There’s a wrong date in the first column…and there’s an extra questionnaire. With a 16 year old, the client shouldn’t have done the OQ…the third column. Version 2.97 - August 9, 2013 19

What if the clinician is unable to collect a questionnaire? Clinicians do have the option of marking a “Unable to Collect” and providing a reason. These reasons may change based on the questionnaire the clinician is answering. This (btw) is another discussion you should have with your team as part of your initial organization – everyone should have a good idea of when UTC applies and possibly discuss some those reasons that a little ‘gray’ Version 2.97 - August 9, 2013

What do you do when you find those mistakes? Do you fix them? If you do encounter mistakes…do you fix them? NO! It’s up to the clinician to make any corrections to the assessment! Good clinical practice demands that you NOT alter the documents…that is the responsibility of the clinician! Put a Post-It on it, and talk to the clinician! Version 2.97 - August 9, 2013

And now…the PEI Outcome Application… HANDOUT: Samples for Happy Path So now we’re going to actually get in and look at the PEI Application. This time through we’re going to look at the ideal, “Happy Path” version. Things are not always easy, though! Things go WRONG! So after we travel the Happy Path, we’ll try the Rocky Road! The HAPPY PATH version… Version 2.97 - August 9, 2013

And now…the Rocky Road…. Careful! Errors on Sheets Bad Scores! Missing Information! HANDOUT: TSCYC “bad” questionnaires Version 2.97 - August 9, 2013

Data Entry Training: Happy Path Sample Data on correct Worksheets Data’s already sorted Sample Client ID’s for everyone Training Environment: http://clsxtrainweb/PEIOutcomeMeasures Login as User Name: jflynn Password is: data HANDOUT: Step By Step Guide Now we’re going to work through the application with clean, correct worksheets and our Step-By-Step guides. Sample Data on correct Worksheets Data’s already sorted Sample Client ID’s for everyone Login as User Name: jflynn Password is: data Please refer to the STEP BY STEP guide for help after this class! Version 2.97 - August 9, 2013

Avoiding Data Entry Errors Discussion Points: Sort By Provider Number Sort By Evidence Based Practice Sort By Beginning / End of Treatment Look out for BLANKS…especially in required fields (Diagnosis, Provider ID, Client ID #) Look out for BAD DATES…questionnaires done MORE than 14 days out, for example If you get stopped in Data Entry…POST-IT and talk to the clinician! Don’t just correct things! Remember: Scoring can be confusing…it’s not just about totaling things up! Version 2.97 - August 9, 2013

Questions or Additional Information Outcomes Project Website http://dmhoma.pbworks.com PEI Outcomes e-mail address PEIOutcomes@dmh.lacounty.gov E-group for PEI Outcome Measures Application Alerts. Peioutcomes-subscribe@yahoogroups.com HANDOUT: PEI Website, PEI Egroup Version 2.97 - August 9, 2013

Finally… HELP DESK: 213-351-1335 The OMA is getting updated! Issue Description, Provider #, Focus of Treatment, EBP and Client ID Help Desk will triage your call…Odre is our point man for PEI OMA The OMA is getting updated! Now a couple more things for you… Let’s review the Data Entry Checklist Please help us with our Training Evaluation On the OMA update We are currently in Phase One – what you see here We are working on Phases Two and Three. Two is going to provide additional capabilities like the ability to correct some information without having to call the helpdesk or make a request to the PEI Outcome Team. Phase 3 will be development of reports, including some administrative reports that you can use. Planned release is within the first half of 2012 but there is no official release date as of yet. Updates will be announced to those of you who subscribe to our Application Alerts and email as well as information will be posted on the project wiki. Additional training opportunities will be announced at the time of release! Version 2.97 - August 9, 2013 27