Taking Research to Practice: Rethinking Outcomes and Performance Measures for the Child and Family Service Reviews John D. Fluke, Child Protection Research Center American Humane Association; Erin Dalton, Allegheny County Department of Human Services; Barbara Needell, University of California, Berkeley; Fred H. Wulczyn, University of Chicago; Mark E. Courtney, University of Chicago; Donald Baumann, Texas Department of Family and Protective Services; Society for Social Work Research, January 15, 2011

The Child and Family Service Review Outcomes: Do Research Principles Matter? “We will restore science to its rightful place” – Barack Obama, January 21, 2009 Framework for Impact – Mission Driven Framework for Outcomes: Safety, Permanency, Well Being – Embedded within systems Entries and Exits States and Processes – Quality of Care – Focus on Improvement Research – Rigorous Design/Sample Validity Reliability – Parsimony – Risk Adjustment

How do the Basic Principles of Outcome Based Research Line up With the Child and Family Services Review Process? Strengths – Effort to focus on outcomes – Use of Data Driven Framework – Development of Infrastructure – Focus on system improvement Issues - Outcome Research Principles – Measures are not necessarily Valid Parsimonious Rigorously Designed or Implemented Risk Adjusted – Interpretation of Review Findings are not necessarily informed by research – Remedies (Program Improvement Plans) are not demonstrably tied to research – Measures of change are focused on unadjusted normative standards False Choices Along the continuum of Rigor and Feasibility – Risk of Type I and Type II errors

The Way Forward: What Needs Attention in the CFSR Outcomes and Evaluation? Measures – State and National Data Infrastructure – Focus on Validity and Parsimony – Range of Data – Adequate Samples Standards – Based on Risk Adjustment – Based on Changes Improvement – Performance monitoring – Improving the scientific rigor Reducing the Type I and Type II errors (same as administrators)

Examples of Measurement

A Local Perspective Child welfare -- leaders in this area Unnecessary complexity Competition amongst measures Different, inconsistent definitions Data quality Error Serious Penalties – Doing the wrong things – Financial

Unnecessary Complexity Permanency Composite 1 Measures Timeliness and Permanency of Reunification, and consists of four measures National standard:122.6 National median: Pennsylvania score: 85.2

Inappropriate Complexity: Competition Amongst Measures Within Composites Permanency Composite 1 Measures Timeliness and Permanency of Reunification, and consists of four measures Exits to reunification: Of those reunified during the report period, what percent went home within 12 months of removal? Median length of stay: Of those reunified, what was the median months in care? Of those entering care for first time in the 6 months prior to the period, what percent were reunified within 12 months of removal? Of those exiting care in the 12 months prior to the report period, what percent returned to care within 12 months of being discharged?

Different Populations - Lack of Risk Adjustment Recurrence of Maltreatment- Of the children who were victims of substantiated abuse during first six months of the report period, what percentage were not victims of another substantiated report within six months. – National standard: 94.6% – PA: 97.0% Why are we doing so well? Are we doing well?

Even within State

But are we doing better? CFSR case review suggests otherwise Our analysis suggests

Different Populations - Lack of Risk Adjustment Of those exiting care in the 12 months prior to the report period, what percent returned to care within 12 months of being discharged? PA: 28.5% Median: 15.0% 75th: 9.9% Why are we performing so differently than the nation? Could be serving a more difficult population? An older population? More likely it’s because juvenile justice population included in our counts, not in others.

Returns to Care Insert returns to care w/in 12 mos from perm exits Look at age composition over time

Data Quality Placement Stability Composite Consists of three measures National standard: National median: 93.3 Pennsylvania score: 102.4

Opportunity for Reform Interest at the federal level APHSA obtaining suggestions from the states and “experts” Casey Family Programs convening states to discuss No reason to sacrifice science in developing measures and recommending approaches to improving services