1. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Data 201: The Empirical Data Strikes Back* Emily Putnam-Hornstein, MSW Center for Social Services Research University of California at Berkeley May 22, 2008 The Performance Indicators Project at CSSR is supported by the California Department of Social Services and the Stuart Foundation *Thanks to Will Sanson for the brilliant title!

2. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Agenda Brief Review Orientation to data source Using data to evaluate performance Downloading and working with the data Small group projects

3. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Data Views

4. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley January 1, 2005January 1, 2006 How long do children stay in foster care? July 1, 2005 Child 1 Child 2 Child 3 Child 4 Child 5 Child 6 Child 7 Child 8 Child 9 Child 10

5. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Data Orientation

6. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley CFSR Measures

7. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Safety Measure 1 Safety Measure 2

8. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Composite 1

9. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Composite 2

10. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Composite 3

11. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Composite 4

12. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Safety S1.1 S2.1 Permanency Composite 1: Reunification Composite 2: Adoption Composite 3: Long-Term Composite 4: Placement C1.1 C1.2 C1.3 C1.4 C2.1 C2.2 C2.3 C2.4 C2.5 C3.1 C3.2 C3.3 C4.1 C4.2 C4.3

13. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Principal Components Analysis (PCA) (the “black box” version) black box of fancy statistical tools Timeliness of Reunification Timeliness of Adoption Permanency of Reunification Placement Stability Median Time in Care Recurrence of Maltreatment Abuse in Foster Care Emancipating from Care Component #1 Component #2 Component #3 A bunch of measures… Three components based on related measures!

14. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Z-Scores? Before dumping all of the measures into the PCA “Black Box”, they were transformed into standard scores (z-scores) A z-score serves two purposes: Puts measures in the same “range” Sets measures to the same “system”

15. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley And an Example… –A researcher interested in measuring “success” in high school. –Collects the following measures for each student: Athletic Ability Good Grades Physical Attractiveness Interest in Sports Chess Club Membership Science Club Membership Social Life Principal Components Analysis…

16. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Interest in Sports Athletic Ability Good Grades Chess Club Member Science Club Member Physical Attractiveness Active Social Life Reduces the number of individual measures: VERY HIGHLY ASSOCIATED!! Explores the contribution of each part to the whole: Jock Component = Brainiac Component = Popular Kids Component = Structures the data into independent components: Athletic Ability Interest in Sports Good Grades Chess Club Member Physical Attractiveness Active Social Life

17. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Safety S1.1 S2.1 Permanency Composite 1: Reunification Composite 2: Adoption Composite 3: Long-Term Composite 4: Stability C1.1 C1.2 C1.3 C1.4 C2.1 C2.2 C2.3 C2.4 C2.5 C3.1 C3.2 C3.3 C4.1 C4.2 C4.3

18. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Evaluating Safety: S1.1 (Safety Indicator 1) Nat’l Std=94.6% S2.1 (Safety Indicator 2) Nat’l Std=99.68%

19. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Permanency Indicators Reunification (Composite 1) Nat’l Std=122.6 Adoption (Composite 2) Nat’l Std=106.4 Long Term Care (Composite 3) Nat’l Std=121.7 Placement Stability (Composite 4) Nat’l Std=101.5 Evaluating Permanency:

20. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley AB636 Measures

21. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley AB636 Measures

22. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley AB636 Measures

23. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

27. Methodology explaining each measure and filtering possibilities

28. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Filtering / Disaggregation Options

29. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

30. Using Data to Evaluate Performance Off to the website… 1.National goals / standards 2.Performance changes over time 3.Filters for disaggregation

31. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley National Standards and Goals National Standards represent the approximate performance of the 75 th percentile performing state (published by the Feds) 6 National Standards (2 Safety, 4 Permanency)…but Permanency standards are composites which are hard to interpret UCB website posts “national goals” (again, approximate performance of the 75 th or 25 th percentile performing state – depending on measure direction) as benchmarks for performance and improvement for each of the 15 individual Federal performance measures No standards / goals for state 636 measures

32. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley National Standards for each of the two federal safety measures

33. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley National “Goals” for the 15 permanency measures

34. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Examining Performance Changes Over Time Five different report (or table) viewing options… 1.Federal CFSR Summary Table 2.California CWS Outcomes System Summary Table 3.Single Time Period Selection Table 4.Multiple Time Periods Table 5.Single Time Period Summary Table 6.Multiple Time Periods Summary Table

35. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Estimate of performance for CA and all counties based on Federal methodology (i.e., includes all agencies). Users specify a single time period.

36. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Performance for CA and all counties matching CWS Outcomes Spreadsheets (i.e., limited to child welfare agency only). Users specify a single time period.

37. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Users specify a single time period for a single geography (these reports usually offer the greatest number of filtering options)

38. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Users choose an interval that allows them to view multiple time periods for a single geography

39. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Users specify a single time period for CA and all counties. Expands the Federal CFSR and California CWS reports by offering filtering options.

40. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Users choose an interval that allows them to view multiple time periods for CA and all counties.

41. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Filtering…

42. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley And off to work with the data… (see attachments)

43. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley **Thank you to Graham Dawson for sharing this slide with us!

44. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Using Excel to Manipulate Data… Column Percentages…but what if we are interested in the percentage of children in group homes who are 11-15 years old?

45. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 2006 California: Ethnicity and Path through the Child Welfare System (Missing Values & Other Race Excluded from % Calculations)

46. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 2006 San Francisco: Ethnicity and Path through the Child Welfare System (Missing Values & Other Race Excluded from % Calculations)

47. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley CWS Outcomes Spreadsheets… (70 tabs!)

48. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley CSSR.BERKELEY.EDU/UCB_CHILDWELFARE Needell, B., Webster, D., Armijo, M., Lee, S., Cuccaro-Alamin, S., Shaw, T., Dawson, W., Piccus, W., Magruder, J., Exel, M., Conley, A., Smith, J., Dunn, A., Frerer, K., & Putnam Hornstein, E., (2007). Child Welfare Services Reports for California., from University of California at Berkeley Center for Social Services Research website. URL: http://cssr.berkeley.edu/UCB_CHILDWELFARE/ Emily Putnam-Hornstein eputnamhornstein@berkeley.edu Barbara Needell bneedell@berkeley.edu