Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, & University Collaboration National Association of Welfare Research and Statistics 45 th Annual Workshop Madison, Wisconsin August 31, 2005 Kelly Cross San Bernardino County HS Terry V. Shaw Center for Social Services Research University of California at Berkeley The Performance Indicators Project is funded by the California Department of Social Services and the Stuart Foundation

Measuring Child Welfare Outcomes Child InChild Out A bunch of stuff happens *adapted from Lyle, G. L., & Barker, M.A. (1998) Patterns & Spells: New approaches to conceptualizing children’s out of home placement experiences. Chicago: American Evaluation Association Annual Conference

u Government Performance Results Act of 1993 (GPRA) u Annual Outcomes Report to Congress mandated by Adoption and Safe Families Act (ASFA) of 1997 u Statewide Data Indicators in Child and Family Services Reviews -- a subset of the Annual Outcomes—from National Child Abuse and Neglect Data System (NCANDS) and Adoption and Foster Care Analysis and Reporting System (AFCARS) u California Child Welfare System Improvement and Accountability Act (AB636) became law in 2001 and went into effect in January 2004 Outcomes, outcomes, everywhere

Quarterly distribution of county specific outcome indicators data Includes national standards (from AFCARS), but also draws heavily on previous work done by CWDA and UCB using entry cohort measures Mirrors Family to Family Outcomes Retains key process measures (e.g., child visits, time to investigation)

Statewide Data Indicators from AFCARS u Stability Of Foster Care Placement u Length Of Time To Reunification u Foster Care Re-entries u Length Of Time To Adoption

Why do we use entry cohort measures in addition to measures from AFCARS?

Who is in AFCARS?  AFCARS contains data on children in foster care during a federal fiscal year  Each reporting period’s submission is a separate dataset. Reporting periods are linked together by the Children’s Bureau to form the annual databases. ANNUAL DATABASES ARE NOT LINKED TO EACH OTHER.

11/0211/0311/04 Data snapshots can be biased Source: Aron Shlonsky, University of Toronto (formerly at CSSR)

California EXAMPLE: Age of Foster Children (2003 first entries, 2003 exits, July caseload)

California EXAMPLE: Median Length of Stay in Months ( first entries, 2001 first spell exits, July first spell caseload)

Federal Measure: Of all children who were adopted during the year, what % had been in care for less than 24 months? (national standard = 32%) State enriched: Of all children entering care for the first time, what % are adopted in less than 24 months? (we do not have state standards)

Percent of children exiting care to finalized adoption in less than 24 months (32% National Standard) Baseline: 100 kids exiting to adoption, 33 of them within 24 months=33%. Substantial conformity achieved! Two pronged approach (1) Faster adoption for 100 children, 50 of them within 24 months=50%, (2) adoptions for 100 kids in long term care 2 years later: 200 kids exiting to adoption, 50 within 24 months=25%. Substantial conformity NOT achieved!?!

Are you getting better or worse? Data from the Multi State Data Archive Adoption within 24 Months year Source: Chapin Hall Center for Children

Why don’t we have state standards ?

The Cycle of Experiences in the Child Welfare System The Cycle of Experiences in the Child Welfare System Counterbalanced Indicators of SystemPerformance PermanencyThroughReunification, Adoption, or Guardianship Length Of Stay Stability Of Care Rate of Referrals/ Substantiated Referrals Home-Based Services vs. Out-of-HomeCare Positive Attachments To Family, Friends, and Neighbors Use of Least Restrictive Form of Care Source: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L. (2002). Measuring Outcomes in Child Welfare. Chapel Hill: Jordan Institute for Families, Reentry to Care

Lack of understanding about the limitations of the national standards, and pressure to achieve “substantial conformity” (pass), could drive changes in policy and practice that may not be best for children and families.

Kelly Cross San Bernardino County HS

AB636 Components Quarterly distribution of county specific outcome indicators data County Self Assessment Peer Quality Case Review County Self Improvement Plan Continuous monitoring of outcomes

Limitations of Administrative Data Adapting a case management system into a mechanism for tracking longitudinal outcomes. A three-year cycle of outcome evaluation Still are not ready to set a baseline Must refine measurement methods, data clean up and training.

Examples of County Work to Examine Data and Improve Practice Changing policy Social worker monthly contacts Out of home abuse. Confounding policy and data issues ICWA status Recurrence of maltreatment Associated referrals “Substantial risk,” & “Sibling abused, child at risk”

Examples of County Work to Examine Data and Improve Practice continued Complex outcomes—Required health visits One of several complex measures in development Identify CHDP standards Allow data entry lag Revise code to account for different time periods when children are in compliance An example of improved practice—Reentry Examining data helped target those children most at risk of returning to care after reunification Expanded use of Public Health Nurses

AB636=State / County Partnership Shifts focus from process measured compliance to outcome based review system, but requires linking outcomes to related processes. Data are our friends, not our dictators. Requires county collaboration with community partners (SIPs signed by Boards of Supervisors). Promotes sharing of promising practices among counties.

More Advantages of University Involvement Participate regularly on state and county workgroups and committees. Share programming code and seek input from county partners on its continual improvement. Ensure availability to answer ad hoc questions

Data and Policy Committees Continual refining of measurement process requires both Data and Policy Committees. Policy committee—interprets regulatory implications and decides general structure that measurement will conform to (e.g., inclusion of guardianship in measures). Data committee—determines specific data collection & analysis steps necessary to implement measurement guidelines decided by policy group.

Examples of County Work to Examine Data and Improve Practice continued Complex outcomes—Required health visits One of several complex measures in development Identify CHDP standards Allow data entry lag Revise code to account for different time periods when children are in compliance An example of improved practice—Reentry Examining data helped target those children most at risk of returning to care after reunification Expanded use of Public Health Nurses

UCB Website cssr.berkeley.edu (Child Welfare Services Reports) includes age, ethnicity, gender breakouts kin vs non-kin for all AB636 measures and more use “Datadude” to examine performance over time

Lessons Learned It takes time. Keep accurate records of development & policy decisions. Participant turn over (state, county, & university) Discovering new populations

State Websites (Child Welfare Systems Improvements) (AB636 Quarterly Reports) Kelly Cross (909)