1. 11 Triple P Outcomes in California Arizona Child Trauma Summit April 9, 2013 Cricket Mitchell, PhD Senior Associate, CiMH

2. Summary of Breakout Session Overview of California’s Triple P Outcome Evaluation Data –Counties supported by CiMH Options to consider in developing outcome evaluation protocols Outcome evaluation for Arizona’s Triple P implementation – facilitated discussion 2

3. What is CiMH? And How is It Related to Triple P? The California Institute for Mental Health (CiMH) is a statewide non-profit that provides training, technical assistance, research, evaluation, and policy support to publicly-funded agencies –Supports the dissemination and implementation of 12 evidence-based practices Program performance and outcome evaluation is a critical implementation support Triple P was selected for dissemination by CiMH and promoted to county agencies in 2006 –Some agencies contract with CiMH, and some do not 3

4. Triple P Implementation Sites Across California Counties Mendocino Alameda Shasta Nevada Sonoma Marin San Francisco Contra Costa Santa Cruz Santa Clara San Joaquin Ventura Los Angeles Riverside Orange San Diego ______________ Also Tri-cities Area 4

5. Overview of California’s Triple P Outcome Evaluation Data Summer 2012 Triple P Data Submission to CiMH –Four Counties Los Angeles Shasta Sonoma Ventura –74 implementation sites –5,292 unique child clients served 5

6. Overview of California’s Triple P Outcome Evaluation Data Outcome evaluation protocols within each county vary –Data elements collected Demographics Service delivery information –Outcome measures used –Applications/software used for data entry 6

7. Overview of California’s Triple P Outcome Evaluation Data CiMH’s Program Performance and Outcome Evaluation Reports –Three primary domains Characteristics of clients served Description of services provided Outcomes achieved –Two-Pronged Approach to Outcome Measurement »Target-specific symptoms »General mental health functioning 7

8. Overview of California’s Triple P Outcome Evaluation Data Today’s presentation will highlight select data elements from the Summer 2012 data submission –Triple P Levels and Types –Child Client Demographics Age, Gender, Ethnicity, Primary Language Spoken in the Home, and Primary Axis I DSM-IV diagnosis –Triple P Outcomes Eyberg Child Behavior Inventory (ECBI), Parenting Scale, and Youth Outcome Questionnaire (YOQ) 8

9. Overview of California’s Triple P Outcome Evaluation Data – Level and Type of Triple P 9

10. Overview of California’s Triple P Outcome Evaluation Data – Age 10 Range:.01 – 26.05 years –Some counties serve Transition Age Youth (15-26) Mean: 7.7 –Standard Deviation: 4.1 Mode: 4.0 Frequency distribution is positively skewed –25 th percentile: 4.6 –50 th percentile: 7.2 –75 th percentile: 10.7

11. Overview of California’s Triple P Outcome Evaluation Data – Gender 11

12. Overview of California’s Triple P Outcome Evaluation Data – Ethnicity 12

13. Overview of California’s Triple P Outcome Evaluation Data – Primary Language 13

14. Overview of California’s Triple P Outcome Evaluation Data – Primary Axis I DSM-IV Dx* 14 *Two of the four Counties track mental health dx

15. CiMH Outcome Indicators Percent Improvement –Percent improvement from average pre-score to average post-score Paired t-test conducted to examine whether or not the difference is likely to be due to chance (p<.01); if not, the percent change is asterisked (*) to indicate a statistically significant improvement Effect Size Estimate: Cohen’s d –A standardized measure that estimates the magnitude, or strength, of the observed change Conventional interpretation:.8 ≈ “large” effect;.5 ≈ “moderate” effect; and,.2-.3 ≈ “small” effect 15

16. CiMH Outcome Indicators Reliable Change –The amount of change observed in an outcome measure that can be considered an actual change, and not likely to be due to the passage of time or measurement error (p<.05) Complex statistical formula that takes the measure’s reliability into consideration, as well as the variability observed among scores –Once the formula is applied, clients can be grouped into one of three categories: reliable positive change; reliable negative change; and, no reliable change 16

17. Overview of California’s Triple P Outcome Evaluation Data Target-Specific Outcome Measure Focused on Child Disruptive Behaviors –Eyberg Child Behavior Inventory (ECBI) Parent/Caregiver Report of the Intensity and Problematic extent of child behavior problems 36 items Intensity Score Range 36 – 252 –Clinical cutpoint 131 and higher Problem Score Range 0 – 36 –Clinical cutpoint 15 and higher –Used by two of the four counties 17

18. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: ECBI 18 Eyberg Child Behavior Inventory (ECBI) Percent Improvement from the Average Pre- Score to the Average Post- Score Effect Size Estimate (Cohen’s d) Percent of Clients Showing Reliable Change from Pre- to Post- Positive Change No Change Negative Change Intensity Raw Score 28.9%* (n=726) [pre=135.3] 1.05 63.4% (n=460) 31.5% (n=229) 5.1% (n=37) Problem Raw Score 51.0%* (n=744) [pre=17.7] 1.14 65.5% (n=487) 29.5% (n=220) 5.0% (n=37) *A statistically significant improvement, p <.01

19. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: ECBI Intensity 19 Solid line indicates clinical cutpoint

20. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: ECBI Problem 20 Solid line indicates clinical cutpoint

21. Overview of California’s Triple P Outcome Evaluation Data Target-Specific Outcome Measure Focused on Parenting –Parenting Scale Parent/Caregiver Report that assesses parenting and disciplinary styles that are found to be related to the development and/or maintenance of child disruptive behavior problems 30 items Total Score is a mean item response ranging from 1 – 7 –Clinical cutpoint 2.8 and higher –Used by two of the four counties 21

22. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: Parenting Scale 22 Parenting Scale Percent Improvement from the Average Pre- Score to the Average Post- Score Effect Size Estimate (Cohen’s d) Percent of Clients Showing Reliable Change from Pre- to Post- Positive Change No Change Negative Change Total Score 28.0%* (n=154) [pre=3.6] 1.25 48.7% (n=75) 49.4% (n=76) 1.9% (n=3) *A statistically significant improvement, p <.01

23. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: Parenting Scale 23 Solid line indicates clinical cutpoint

24. Overview of California’s Triple P Outcome Evaluation Data General Outcome Measure of Mental Health Functioning –Youth Outcome Questionnaire Parent/Caregiver Report that assesses multiple dimensions of child/youth mental health functioning 64 items Total Score Range -16 – 240 –Clinical cutpoint 47 and higher –Used by one of the four counties 24

25. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: YOQ Total 25 Youth Outcome Questionnaire (YOQ) Percent Improvement from the Average Pre- Score to the Average Post- Score Effect Size Estimate (Cohen’s d) Percent of Clients Showing Reliable Change from Pre- to Post- Positive Change No Change Negative Change Total Score 36.3%* (n=638) [pre=63.9].74 57.5% (n=367) 34.0% (n=217) 8.5% (n=54) *A statistically significant improvement, p <.01

26. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: YOQ Total 26 Solid line indicates clinical cutpoint

27. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: Reliable Change 27

28. 28 Overview of California’s Triple P Outcome Evaluation Data Follow-up analyses of aggregate data indicate no differences in: –change in ECBI Intensity Score outcomes; –change in ECBI Problem Score outcomes; –change in Parenting Scale outcomes; or, –change in YOQ Total Score outcomes by gender or ethnicity

29. Options to Consider in Developing Outcome Evaluation Protocols Data elements to track/collect –Parsimony –Utility Outcome measures –Relevance to treatment target/goals –Psychometric characteristics (valid, reliable) –Cost –Time (administration, scoring, data entry) –Training and technical assistance 29 Thoughtful and thorough planning is the key!

30. Options to Consider in Developing Outcome Evaluation Protocols Application/software used for data entry –System already in place that can be modified? (e.g., EHR, county- or state-level information system) –Cost –Skill level to use/employ –Utility of data elements for analysis and reporting –Training and technical assistance 30

31. Options to Consider in Developing Outcome Evaluation Protocols Frequency of analysis and reporting –Multiple stakeholders Different reports for different audiences –Processes for maximizing utility of data Clinical utility Program improvement Systems-level decisions 31 Feedback is Essential

32. Additional Considerations for Telling the Whole Story Collect minimal data on all clients referred –Determine entry rate –Determine additional need (waiting lists) Collect completion status (yes/no) –Determine dropout rate –May provide the opportunity to examine dose-response relationships Track clients who are served by more than one Level/Type of Triple P (within and across providers) Track population-level indicators (substantiated child maltreatment cases, out of home placements, emergency room visits for unexplained child injuries) 32

33. Evaluation for Arizona’s Triple P Implementation – Discussion Questions Who should be included in decision-making? Is there an overarching evaluation framework? What data are currently being collected? –What additional data elements are of interest? What outcome measures will be used? –How will they be obtained, distributed, and used? –Who will provide training and technical assistance? How will data be tracked/collected from individual Triple P providers? 33

34. Evaluation for Arizona’s Triple P Implementation – Discussion Questions How will population-level indicators be tracked? With what frequency? Responsibility for and frequency of data analysis and reporting? How will data be used to inform decisions? –Client-level –Program-level –System-level 34

35. 35 Discussion Summary

36. 36 The End Contact Information Cricket Mitchell, PhD Email: cmitchell@cimh.orgcmitchell@cimh.org Cell phone: 858-220-6355