1. Building a Data- Driven Culture in Nevada June 10, 2013 The Performance Indicators Project is a collaboration of the California Department of Social Services and UC Berkeley, and is supported by CDSS, and the Stuart Foundation.

2. Summit Content Areas Leading with Data: A Focus on Outcomes –Review of basic terminology –Avoiding pitfalls and data abuse –Key concepts in performance measurement –Telling the NV Story: State and National Context Connecting Data to Practice: Defining the problems From Data to Action: Strategy Development and Implementation CQI Structure and Function in NV

3. BASIC TERMINOLOGY AND FORMULAS Data Analytics 101

4. Basic Terminology Descriptive Data Point-in-time Trends Comparisons data source: AFCARS

5. Basic Terminology Process Measures - familiar to staff, relevant at a caseworker level, current Outcome Measures - the “big picture” measure of system performance, especially when looked at longitudinally

6. Measures of Central Tendency Mean: the average value for a range of data Median: the value of the middle item when the data are arranged from smallest to largest Mode: the value that occurs most frequently within the data 12 4 15 63 7 9 4 174 4 7 9 12 15 17 63 7 = 9.7 = 9

7. Measures of Variability Minimum: the smallest value within the data Maximum: the largest value within the data Range: the overall span of the data 4 4 7 9 12 15 17 63

8. Disaggregation One of the most powerful ways to work with data… Disaggregation involves dismantling or separating out groups within a population to better understand the dynamics and plan strategies for improvement Useful for identifying critical issues that were previously undetected Aggregate Permanency Outcomes Race/Ethnicity Age Region/Circuit Placement Type

9. Measuring Change How much has this measure changed over time? What will our performance be next quarter if we increase or decrease by 10% 10% increase = baseline x 1.1 10% decrease = baseline x.90

10. COMMON DATA PITFALLS Data Analytics 101

11. Common Pitfalls Small N – impact on rates and trends Seasonal variation Faulty comparisons – failing to consider demographic and policy differences Outlier impact on central tendency Data integrity/Data entry (over or under emphasized) Missing or incomplete definitions Data overload: lack of focus on and connection to key outcomes

12. Common Pitfalls: Seasonal variation Period 1 to 2: 38.5% reduction Period 2 to 5: 41.7% increase Period 1 to 7: 10.2% reduction

13. Common Pitfalls: Small n (impact on rates and trends) 100% reduction! But…from 2 children in care to 0 children in care 57% increase! But from 7 to 11 children in care

14. Common Pitfalls and Graph Interpretation Data source: UNC at Chapel Hill Jordan Institute for Families website. URL: http://ssw.unc.edu/ma/http://ssw.unc.edu/ma/

15. Avoiding data overload and Managing with data

16. One thing the modern computer age has given everyone is data. Lots and lots of data. There is a large leap, however, between having data and learning from it. W. Gregory Mankiw Professor of Economics, Harvard New York Times, Sunday Business Section (Sept. 5 2010) p. 5

17. Manage with Data Provides us the ability to: Compare metrics with agency mission and practice model Connect to evidence-based practice and link processes to desired outcomes Strategize on what work needs to be done Focus on end outcomes Identify what needs attention Tell the story

18. Manage with Data Pick the right measures for the job Prioritize reports and measures in line with agency values, mission, vision Connect process measures to outcomes/practice model Move beyond compliance and “gotcha” Make it fun! Celebrate success and tell the story Use your data to engage the community, create urgency for action, maintain support

19. Pick the Right Measures for the Job Federal Accountability Overall Performance Monitoring Outside Auditing (i.e. consent decrees, monitors) Contractor or Provider Performance Office, Supervisor, Worker Level accountability Compliance Performance Outcomes Continuum

20. Prioritize Measures in line with Agency Values To keep children safe and at home To improve a child or youth’s well-being To facilitate a child or youth’s move to swift & certain permanency

21. Performance is Guided by Your Values as an Agency: Missouri Key Outcomes

22. Key Data Reports: How are they all connected? CFSR and PIP SCRT ROM AFCARS NCANDS SEE Results COA

23. Connecting the Dots Management Report: Frequency of Visits with Caretakers Case Review Measure: Caregiver involvement in case planning Case Review Measure: Individualized Services Outcome Measure: Timely Reunifications Process Data: Accountability Relevant to workers and supervisors Process Data: Accountability Relevant to workers and supervisors Intermediate Outcomes Relevant to workers, supervisors, managers Intermediate Outcomes Relevant to workers, supervisors, managers Outcomes: “So What?” Reflect Key Priorities of Leadership Outcomes: “So What?” Reflect Key Priorities of Leadership

24. CFSR Findings: Relationship of Well-Being to Permanency Positive ratings on Services to children, parents, foster parents Involvement of parents in case planning Caseworker visits with children Caseworker visits with parents Substantial achievement on Timely achievement of permanency Preserving children’s connections while in foster care supports... Administration for Children and Families, U.S Department of Health and Human Services, Findings From the Initial Child and Family Services Reviews, 2001–2004. Available at http://www.acf.hhs.gov/programs/cb/cwmonitoring/results/index.htm

25. Factors Associated with Timely Reunification, Guardianship, and Permanent Relative Placement The strongest associations with timely permanency included: Caseworker Visits with Parents Child’s Visits with Parents and Siblings in Foster Care Services to Children, Parents, & Foster Parents Family/Child Involvement in Case Planning ASFA Requirements Regarding Termination of Parental Rights Placement Stability Administration for Children and Families, U.S Department of Health and Human Services, Findings From the Initial Child and Family Services Reviews, 2001–2004. Available at http://www.acf.hhs.gov/programs/cb/cwmonitoring/results/index.htm

26. Strongest Associations Between Visits and Other Indicators Both Caseworker Visits with Parents and Caseworker Visits with Children were strongly associated with: Risk of harm to children Needs & Services for children, parents, foster parents Child and parent involvement in case planning Administration for Children and Families, U.S Department of Health and Human Services, Findings From the Initial Child and Family Services Reviews, 2001–2004. Available at http://www.acf.hhs.gov/programs/cb/cwmonitoring/results/index.htm

27. Other Significant Associations Between Visits and Indicators Caseworker Visits with Parents and Caseworker Visits with Children were also strongly associated with: Services to protect children at home Safety Outcome 1 Safety Outcome 2 Timely permanency goals Timely reunification Child’s visits with parents and siblings Relative placements Meeting educational needs Meeting physical health needs Meeting mental health needs Administration for Children and Families, U.S Department of Health and Human Services, Findings From the Initial Child and Family Services Reviews, 2001–2004. Available at http://www.acf.hhs.gov/programs/cb/cwmonitoring/results/index.htm

28. Connecting process to outcomes

29. Use Data to Create Urgency for Action – Target improvements based on your own baseline Regional variation should generate productive discussion about differences in: Service array Community differences in reporting and tolerance for “risk” Differences across partner agencies, courts, juvenile justice, behavioral health etc… Demographic risk factors and “case mix” Case loads, turnover (staff and leadership), and training A variety of other policy/practice differences

30. Grounded in good case practice model principles Develop presentation skills Understand & demystify data Master qualitative & quantitative tools Recognize challenges Celebrate good practice Support positive change Act as a local resource Grow as managers & leaders Your Role as a Data Leader

31. Knowing when you’ve got it right No more “the data are wrong” Folks own data, know it, act on it Practice people know the data, data people know the practice Field pulls data, asks for reports, initiates actions tied to the data Constantly talking about data in a positive way After a really busy day, the data manager comments: “I think I liked it better when no one paid attention to the data”

32. Telling the Story: Key Child Welfare Indicators

33. So What’s the Story? Describe the issue with as much detail as possible, variation is key to hypothesis development. What’s happening right now for all kids? Has it always been this way? Is it true in all places, for all ages, for all racial/ethnic groups? Is this indicator correlated with any others? Does it look the same for all types of cases, or in places where practice is different?

34. interdependence between measures… Counterbalanced Indicators of System Performance Permanency Through Reunification, Adoption, or Guardianship Shorter Lengths Of Stay Stability Of Care Rate of Referrals/ Substantiated Referrals Home-Based Services vs. Out-of-Home Care Maintain Positive Attachments To Family, Friends, and Neighbors Use of Least Restrictive Form of Care Reentry to Care

35. the current placement system* (highly simplified) *adapted from Lyle, G. L., & Barker, M.A. (1998) Patterns & Spells: New approaches to conceptualizing children’s out of home placement experiences. Chicago: American Evaluation Association Annual Conference CHILD IN a bunch of stuff happens CHILD OUT the foster care system

36. Trends in Out of Home Care Data source: AFCARS Nationwide, the number of children in out of home care is declining. In NV, both the entry rate (per 1,000 children in the population) and the in-care rate are higher than the national average.

37. Safety – The Absence of Repeat Maltreatment

38. CHILDREN ENTERING CARE Managing with Data in Child Welfare

40. Children Entering Care: Nevada

41. Key Questions: Entries What is the entry rate – by age/race? Are entries increasing/decreasing? for all groups? What strategies are in place/planned to reduce entries (and re-entries) into care?

42. Possible reasons for county differences in entry rates: Service array – preventive and in home Standard of evidence Law enforcement removals Demographic risk factors A variety of other policy/practice differences Substantial variation year to year is also common in counties with few removals/small populations

43. CHILDREN IN CARE POINT IN TIME Managing with Data in Child Welfare

44. Key Questions: Children in Care What groups of children are in care NOW What types of placements? How long have they been in care? What is needed to move them to permanency?

45. Placement Type (Ages 0-17)

46. Placement Type (ages 0-10)

47. OUTCOMES: EXITS AND LENGTH OF STAY National and State Level

48. Key Questions: Permanency Outcomes What proportion of children entering care will eventually reunify? How does this differ by age at removal? What percent of children remain in care after 3 years? Are there differences by age/race? Is this trend changing over time?

49. data Know which view to use

50. the view matters… January 1, 2012December 31, 2012July 1, 2012 Source: Aron Shlonsky, University of Toronto (formerly at CSSR)

51. the view matters…

52. entries, point in time and exits views…

55. Exit Cohort View… but what about those that remain in care?

56. Timely Reunification (entry cohort)

57. Nationally, there has been almost no improvement in timely reunification

58. Re-Entry after Reunification

59. Permanency for Longer Stayers

61. Connecting Data to Practice: Using the CQI Framework

62. Continuous Quality Improvement (CQI)…an ongoing process of identifying, describing, and analyzing strengths and problems and then testing, implementing, learning from, and revising solutions.

63. CQI Relies on… An organizational culture that is proactive and supports continuous learning. A strong foundation – the mission, vision, and values of the agency. The active inclusion and participation of staff at all levels of the agency, children, youth, families, and stakeholders throughout the process.

64. Key Principles Use data and information from multiple sources, qualitative and quantitative Data have a purpose: Identify trends and anomalies; find areas for improvement; tell stories about what is happening in practice and policy CQI must support staff to improve outcomes for families

65. Key Principles If it ain’t “broke”, it can probably still be “fixed” CQI goes beyond “compliance” to “quality” Meaningful and active engagement of staff at all levels, children, youth, families, and stakeholders CQI requires training, preparation, and consistent ongoing support

66. DEMYSTIFYING THE LOGIC MODEL CQI Group Exercise 2

67. Observe We’ve noted that: Children are not exiting to permanency quickly enough Explain And we believe it is because: Case management and case consultation has not been consistent Strategy So we plan to: Improve training and supervision; ensure practice is aligned with policy Outcome Which will result in ENVISIONED OUTCOME: An increase in children exiting to permanency within three years Developed by NY OCFS

68. …if he had one hour to save the world he would spend 55 minutes defining the problem and only 5 minutes finding the solution. Before jumping right into solving a problem Step back Invest time and effort Improve understanding Source: http://litemind.com/problem-definition/ (accessed 6/3/11)http://litemind.com/problem-definition/

69. We have noted that: We believe it is because: So we plan to: Which will result in: H YPOTHESIS S TATEMENT : A H IGH L EVEL C AUSE A ND E FFECT S TATEMENT Needs and Strengths Assessment Activities Initial and Intermediate L OGIC M ODEL : D IGGING D EEPER – M ORE D ETAIL Outputs Key End Outcomes Where are we now? Observe performance on key measures: review trends and patterns Establish priorities by considering: mandates, greatest areas of need/opportunity for impact etc… Explain/Explore key underlying factors: both internal and external Consider subpopulations: is performance different by age? Race? Maltreatment type? Define strengths & areas needing improvement Administrative Data is only one part of the assessment.

70. We have noted that: We believe it is because: So we plan to: Which will result in: H YPOTHESIS S TATEMENT : A H IGH L EVEL C AUSE A ND E FFECT S TATEMENT Needs and Strengths Assessment Activities Short Term Outcomes L OGIC M ODEL : D IGGING D EEPER – M ORE D ETAIL Outputs Long Term Outcomes Where do we want to be? What are the ultimate outcomes that we hope to achieve? Reduce entries into care Improve likelihood and timeliness of a permanent exit Reduce re-entry Improve health, mental health and education indicators

71. Group Exercise!! ObserveExplainStrategyOutcome Observe We examined the data and noted that: How are you doing on key outcomes? Are they going in the right direction? Is this true everywhere, and for all children? What other indicators are related to this outcome? Explain And we believe it is because: Why? Start with brainstorming, then look to a variety of existing data – where is there variation? What more do you need to know? How will you find out? Developed by NY OCFS

72. Table Discussion 1 – Trends in Timely Permanency Review regional data packets and CQI handout: Focus on WHAT and WHY Describe the trends in timely permanency and related measures Are these indicators –Increasing? –Decreasing? –Staying about the same? What does this tell us? What more do we need to know?

73. MOVING FROM DATA TO ACTION CQI Group Exercise 2

74. We have noted that: We believe it is because: So we plan to: Which will result in: H YPOTHESIS S TATEMENT : A H IGH L EVEL C AUSE A ND E FFECT S TATEMENT Needs and Strengths Assessment Activities Short term outcomes L OGIC M ODEL : D IGGING D EEPER – M ORE D ETAIL Outputs Long Term Outcomes What will we do to address the issue? Strategies should align with the strengths and needs. What activities are supporting good performance? What are the barriers? Consider Strategies: What do you control? Where do you need to partner or advocate? Training Programs/Services Policies/practices

75. We have noted that: We believe it is because: So we plan to: Which will result in: H YPOTHESIS S TATEMENT : A H IGH L EVEL C AUSE A ND E FFECT S TATEMENT Needs and Strengths Assessment Activities Short Term Outcomes L OGIC M ODEL : D IGGING D EEPER – M ORE D ETAIL Outputs Long Term Outcomes How do we know that the strategy was implemented as planned? What are our timeframes? Examples of outputs: Often a count (and percent) # of people trained # of clients served # of referrals # of meetings held

76. We have noted that: We believe it is because: So we plan to: Which will result in: H YPOTHESIS S TATEMENT : A H IGH L EVEL C AUSE A ND E FFECT S TATEMENT Needs and Strengths Assessment Activities Short Term Outcomes L OGIC M ODEL : D IGGING D EEPER – M ORE D ETAIL Outputs Long Term Outcomes How will we know we are heading in the right direction? Short Term outcomes can be expected to change quickly. Examples of measurable improvements: Improve diligent search and engagement Reduce time to adjudication and disposition Increase timely permanency hearings Improved family engagement in case planning