1. Keys to Data-Driven Decision Making in Complex Service Systems: Availability, Reliability, & Palatability Alyssa M. Ward, Ph.D, LCP Director of Clinical Training & Evidence Based Practice

2. Data Integration: Basic Considerations Establish Data Routine Maximize Uptake & Integration of Findings AvailabilityReliability Mission Vision Leadership Priorities Identify Meaningful Targets Palatability Begin with Intention 2

3. Begin with Intention: Data with Purpose Every Data Point = Time Time = Effort/Burden + Money What is the mission and vision of the organization, system, or specific implementation? What are the organizational priorities? What targets are meaningful to whom? 3

4. Hathaway-Sycamores Child & Family Services Large, community-based mental health & child welfare organization serving roughly 9,000 youth & families/year in Los Angeles County, CA One of the largest Medi-Cal contracts with LAC Department of Mental Health (~50 million/year) Wide Array of Mental Health & Child Welfare Services include: – Outpatient – School-Based – In-Home – Residential – Wrap-Around – Full Service Partnership (FSP) – Therapeutic Behavioral Services (TBS)

5. Quality Performance & Outcomes Reliable Outcomes Satisfaction Fiscal Practice Process Fidelity Tracer Methodology Safety Utilization Consumer Customer Stakeholder Workforce Level of Care Functioning Symptoms Completion Cost of care 5

6. HSFCS Indicators: Child Welfare vs. MH Child Welfare (such as adoption, Wraparound, FSP, residential) Mental Health (such as Outpatient, School Based Services, Transitional Aged Youth) Demographic Number served, age gender, ethnicity Utilization Length of Stay, number of service hours/visits Safety Incident Reports Infection Control EBP Completion Track successful completion of EBP and drop out reasons Level of Care Utilize the ROLES to track living environment Clinical outcome measures In addition to the Mental Health outcome measures* Program specific: Residential and Wraparound participate in the CANS-CW CAFAS/YAFAS, Youth Outcome Questionnaire, Evidenced Based Practice measures depending on treatment focus (RCADS, UCLA PTSD Reaction Index, PHQ 9, Eyberg) Satisfaction In addition to the Mental Health satisfaction survey* Program specific: Residential and FFA have benchmarking consumer satisfaction surveys for California Benchmarking Initiative (CBI) Annual satisfaction survey utilizes the California Department of Health Care Services Program Completion Program specific: Residential and WraparoundNot tracked Occupancy Program specific: Residential and FFA Program specific: Transitional Independent Living Program Complaints and Grievances Program specific: Residential Currently working on agency wide process to track complaints, grievances and compliments Benchmarking program specific Program specific: Residential: cost per day, occupancy, incident rate, discharge level of care. FFA: cost per unit, placement stability, occupancy, discharged to permanent home, average days to reunification. Wraparound: program completion, closed due to disruption, discharge to permanent family home Program specific: NPS: cost per day, staff retention, % of youth with transition plan 6

7. Prevention & Early Intervention Indicators 7

8. Closer Look: Prevention & Early Intervention Outcomes 8

9. 9

10. Trauma Practice Analysis 10

11. Outcomes: Youth Outcomes Questionnaire * p<.0001 PEI ProgramsYOQYOQ-SR Avg. PreAvg. PostAvg. PreAvg. Post The Center for Grief and Loss38.0622.4936.3117.85 Outpatient65.2843.0340.1825.55 School Based Services, East (SBS East)42.4735.048.2342.85 School Based Services, West (SBS West)54.5440.0651.4133.29 Family Resource Center52.2635.5449.7340.45 YOQ Outcomes For Overall Agency

12. Outcomes: YOQ P <.0001

13. Evidence-Based Practices at H-S Implementation Scope EBP# Therapist Trained Total # Active Therapists Therapist Attrition Rate # Youth Served Seeking Safety21914633% 544 TF-CBT16910637% 645 DTQI1026338% 390 MAP1076638% 792 Triple P (4&5)523238% 199 Incredible Years331942% 69 Child Parent Psychotherapy 181233% 59 PCIT6517% 17 *Service delivery and training data reflect period from 7/1/2010 to 9/30/2014 Most Trained Most implemented

14. Implementation Challenges Data Staffing/Supervision

15. Satisfaction Data: Given your training experiences, how prepared do you feel to implement EBPs?

16. EBP Satisfaction by Treatment Type Number of Respondents

17. EBPs for Trauma: A Closer Look at Satisfaction

18. Our Core Practice Model: Data in the Service Delivery Context IMPLEMENTING Providing prescribed intervention tailored to build on strengths and address underlying needs of youth and families ENGAGING Creating trustful working relationships with youth and families TRANSITIONING Evaluating the effectiveness of the plans and interventions, adapting to overcome challenges and organizing after-care supports PLANNING Teaming and collaborating with youth and families to develop a tailored plan to achieve desired results

19. Core Practice Model Principles Phases of Reliable Help

20. How Do Outcomes Fit Into Our Work? Provide one way to gather information from consumers and families to hear their story Research indicates that for some people in some situations, questionnaires are a less threatening way to disclose problems Help to identify early risk factors ENGAGING Creating trustful working relationships with youth and families

21. How Do Outcomes Fit Into Our Work? Measures provide information about broad functioning as well as problem-specific impairments Some measures provide means to integrate strengths towards goals process (e.g. CAFAS) Measures may help with articulating goals What improvement do we expect to see? On what scales? Selecting appropriate evidence-based practices or general interventions PLANNING Teaming and collaborating with youth and families to develop a tailored plan to achieve desired results

22. How Do Outcomes Fit Into Our Work? Outcomes enable monitoring of effectiveness of plans and interventions Adaptation of evidence-based practices Use critical items or particularly high scales to inform ongoing conversation with youth and family about issues of focus and progress Responses from measures may be integrated into some practices For example: cognitive interventions and use of endorsed thoughts IMPLEMENTING Providing prescribed intervention tailored to build on strengths and address underlying needs of youth and families

23. How Do Outcomes Fit Into Our Work? Allow other programs or practitioners a snapshot of treatment progress to smooth after- care, transfer, or return to service – Can highlight specific scales or critical items that corresponded well to goals Outcomes enable assessment of effectiveness of plans and interventions – These may be shared with the family to celebrate progress and make plans for areas that may require further clinical or other attention TRANSITIONING Evaluating the effectiveness of the plans and interventions, adapting to overcome challenges and organizing after-care supports

24. Key Considerations Data Availability – All stakeholders in the process of care (consumers, clinicians, team members, program management, community partners, contract holders) 24

25. Data Availability at HSCFS Electronic Health Record integration Actual data sources Data entry points 25

26. Key Considerations Reliability – How is data monitored, maintained, cleaned? Before transmission to other systems Confidentiality issues 26

27. Compliancy Trend Watch 27

28. HSCFS: Compliancy Reporting Examples 28

29. HSCFS: Compliancy Reporting Examples 29

30. Key Considerations Palatability – Making data accessible to people who hate data 30

31. HSCFS: Flavors of Data Visual Presentation of Data Samples – Benchmarking 31

32. 32

33. 33

34. Stop-Lighting Reporting

35. 35

36. 36 Initiative Reporting

37. MAP Consumer Demographics

38. MAP Clinical Dashboard Practice Progress

39. For more information contact: Alyssa M. Ward, Ph.D. Director of Clinical Training & Evidence Based Practice AlyssaWard@Hathaway- Sycamores.org 626 395-7100 ext. 2559 39