CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley CFSR2 Data Indicators: The Good, the Bad, and the Ugly Center for Social Services Research University of California at Berkeley March 2008 The Performance Indicators Project at CSSR is supported by the California Department of Social Services and the Stuart Foundation

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Overview of Key Changes… Move from individual measures to data composites for evaluating permanency (no such change for evaluating safety) Adjustment of National Standards based on most recent data available Measures reversed (where necessary) so that higher score equals better performance (easier to interpret) Some changes in time periods for given measures

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley CFSR: Seven Outcomes Safety Children are, first and foremost, protected from abuse and neglect. Children are safely maintained in their homes whenever possible and appropriate. Permanency Children have permanency and stability in their living arrangements. The continuity of family relationships and connections is preserved for children. Child and Family Well-Being Families have enhanced capacity to provide for their children's needs. Children receive appropriate services to meet their educational needs. Children receive adequate services to meet their physical and mental health needs.

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley CFSR: Seven Systemic Factors Statewide information system Case review system Quality assurance program Staff and provider training Service array Agency responsiveness to the community Foster and adoptive parent licensing, recruitment and retention

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley CA CWS Outcomes System Round 1 of the CFSRs –2 of the “outcomes” = 6 items (2 for safety, 4 for permanency) –National Standards attached: based on the 75 th %tile of reporting states –States failing to meet a given standard had to include that item in their Performance Improvement Plans (PIPs) Round 2 of the CFSRs –Also comprised of 6 items with standards attached –BUT…this time the permanency standards are comprised of 15 different measures distilled into four composites –TOTAL of 17 FEDERAL MEASURES

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Safety Indicator 1 Indicator 2 Permanency Composite 1 Composite 2 Composite 3 Composite 4 Component A Component B Component A Component B Component C Component A Component B Measure 1 Measure 2 Measure 3 Measure 4 Measure 1 Measure 2 Measure 3 Measure 4 Measure 5 Measure 1 Measure 2 Measure 3

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Safety S1.1 S2.1 Permanency Composite 1: Reunification Composite 2: Adoption Composite 3: Long-Term Composite 4: Placement C1.1 C1.2 C1.3 C1.4 C2.1 C2.2 C2.3 C2.4 C2.5 C3.1 C3.2 C3.3 C4.1 C4.2 C4.3

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley National Standards National standards for both the safety indicators and permanency composites are based on State performance in 2004, 75 th percentile In California, we at CSSR will attempt to replicate each of the measures and composite scores, break them out by child welfare and probation agencies, and report/update quarterly. Although national standards have been set for the composites rather than individual measures… –The goal is to improve State performance on all measures (every improvement reflects a better outcome for children) –Improvement on any given measure will result in an increase in the overall composite score Analogous to Academic Achievement Test Scoring…

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Evaluating Safety: S1.1 (Safety Indicator 1) Nat’l Std=94.6% Of all children who were victims of a substantiated maltreatment allegation during the first 6 months of a year, what percent were not victims of another substantiated maltreatment allegation within the next 6- month period? Of all children served in foster care during the year, what percent were not victims of a substantiated maltreatment allegation by a foster parent or facility staff member? S2.1 (Safety Indicator 2) Nat’l Std=99.68%

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Permanency Indicators Reunification (Composite 1) Nat’l Std=122.6 Adoption (Composite 2) Nat’l Std=106.4 Long Term Care (Composite 3) Nat’l Std=121.7 Placement Stability (Composite 4) Nat’l Std=101.5 Evaluating Permanency:

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Principal Components Analysis (PCA) (the “black box” version) black box of fancy statistical tools Timeliness of Reunification Timeliness of Adoption Permanency of Reunification Placement Stability Median Time in Care Recurrence of Maltreatment Abuse in Foster Care Emancipating from Care Component #1 Component #2 Component #3 A bunch of measures… Three components based on related measures!

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Z-Scores? Before dumping all of the measures into the PCA “Black Box”, they were transformed into standard scores (z-scores) A z-score serves two purposes: Puts measures in the same “range” Sets measures to the same “system”

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley And an Example… –A researcher interested in measuring “success” in high school. –Collects the following measures for each student: Athletic Ability Good Grades Physical Attractiveness Interest in Sports Chess Club Membership Science Club Membership Social Life Principal Components Analysis…

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Interest in Sports Athletic Ability Good Grades Chess Club Member Science Club Member Physical Attractiveness Active Social Life Reduces the number of individual measures: VERY HIGHLY ASSOCIATED!! Explores the contribution of each part to the whole: Jock Component = Brainiac Component = Popular Kids Component = Structures the data into independent components: Athletic Ability Interest in Sports Good Grades Chess Club Member Physical Attractiveness Active Social Life

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Permanency Indicators Reunification (Composite 1) Adoption (Composite 2) Long Term Care (Composite 3) Placement Stability (Composite 4) Composite 1: Timeliness and Permanency of Reunification

C1.2 C1.3 C1.4 Reunification within 12 months (exit cohort) Median time to reunification (exit cohort) Reunification within 12 months (entry cohort) Reentry following reunification C1.1 Composite 1: Timeliness and Permanency of Reunification Timeliness (Component A) Permanency (Component B)

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Permanency Indicators Reunification (Composite 1) Adoption (Composite 2) Long Term Care (Composite 3) Placement Stability (Composite 4) Composite 2: Timeliness of Adoption

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Permanency Indicators Reunification (Composite 1) Adoption (Composite 2) Timeliness (Component A) Progress (Component B) “Legally Free” (Component C) Long Term Care (Composite 3) Placement Stability (Composite 4) Composite 2: Timeliness of Adoption C2.1 C2.2 C2.4 C2.5 Adoption within 24 months (exit cohort) Median time to adoption (exit cohort) Adoption Within 12 Months (17 Months In Care) Legally Free Within 6 Months (17 Months In Care) Adoption Within 12 Months (Legally Free) C2.3

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Permanency Indicators Reunification (Composite 1) Adoption (Composite 2) Long Term Care (Composite 3) Placement Stability (Composite 4) Composite 3: Permanency for Children in Long Term Care

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Permanency Indicators Reunification (Composite 1) Adoption (Composite 2) Long Term Care (Composite 3) Permanency (Component A) “Growing-up” (Component B) Placement Stability (Composite 4) Composite 3: Permanency for Children in Long Term Care C3.1 C3.3 C3.2 Exits To Permanency (Legally Free At Exit) Exits To Permanency (24 Months In Care) In Care 3 Years Or Longer (Emancipated or Age 18 In Care)

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Permanency Indicators Reunification (Composite 1) Adoption (Composite 2) Long Term Care (Composite 3) Placement Stability (Composite 4) Composite 4: Placement Stability

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Permanency Indicators Reunification (Composite 1) Adoption (Composite 2) Long Term Care (Composite 3) Placement Stability (Composite 4) Composite 4: Placement Stability C4.1 C4.2 C4.3 Placement Stability (8 Days To 12 Months In Care) Placement Stability (12 To 24 Months In Care) Placement Stability (At Least 24 Months In Care)

Std Measure1 x Weight (0.462) Std Measure2 x Weight (0.451) Std Measure3 x Weight (0.295) Std Measure4 x Weight (0.129) Std Measure1 x Weight (0.085) Std Measure2 x Weight (0.070) Std Measure3 x Weight (-0.005) Std Measure4 x Weight (1.025) Component A Timeliness of Reunification (Sum of Weighted Measures) Component B Permanency of Reunification (Sum of Weighted Measures) (Un-Weighted) County Composite Score (Component A + Component B / 2) (Weighted) County Composite Score (# of Children in Foster Care x Score) (Un-Scaled) State Composite Score (County weighted scores summed and divided by total # of children served) Timeliness & Permanency of Reunification (Scaled) State Composite Score (Transformed to position on national scale)

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley What else is new? Move from static to dynamic website Possible to still review change over time 75 th percentile for measures posted (national goal) Several 636 Measures have been dropped –‘Old’ to ‘New’ Document (posted with county spreadsheets)

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley CSSR.BERKELEY.EDU/UCB_CHILDWELFARE Needell, B., Webster, D., Armijo, M., Lee, S., Cuccaro-Alamin, S., Shaw, T., Dawson, W., Piccus, W., Magruder, J., Exel, M., Conley, A., Smith, J., Dunn, A., Frerer, K., & Putnam Hornstein, E., (2007). Child Welfare Services Reports for California. Retrieved [month day, year], from University of California at Berkeley Center for Social Services Research website. URL: Emily Putnam-Hornstein Barbara Needell