CHILD AND FAMILY SERVICES REVIEWS (CFSRs) Sustaining the Momentum: The Next Round of Reviews Children’s Bureau Plan for CFSR Statewide Data Indicators and National Standards

Child and Family Services Reviews Statewide Data Indicators CFSR Round 3

Measurement Focus of this presentation: Statewide Data Indicators

Sources of data AFCARS (Adoption and Foster Care Analysis and Reporting System) Contains case level data on all children in foster care and receiving title IV-E funding or who are adopted with child welfare agency involvement. Federally mandated reporting by state child welfare agencies Submitted every six months

Sources of data NCANDS (National Child Abuse and Neglect Data System Child File contains case level detail on children for whom agencies received a screened-in report of alleged maltreatment. Voluntary dataset reported by state child welfare agencies Submitted every year Or.. Alternate source data Safety data only Must be approved by the Children’s Bureau

CFSR Round 2 Safety measures: Permanency Composites Absence of recurrence of maltreatment Absence of maltreatment in foster care Permanency Composites Composite 1: Timeliness and permanency of reunification Composite 2: Timeliness of adoptions Composite 3: Permanency for children/youth in care long periods of time Composite 4: Placement stability

We still want to know… Are children getting to permanent homes quickly, and remaining there, without coming back into care? Are they safe while in the care of the state? Are children reported to the agency for safety concerns being reported subsequently? And are they in stable placements, keeping the number of moves to a minimum?

Changes to measurement Greater reliance on entry cohorts Fewer and simpler indicators No composites

Cohorts Entry Cohorts Children placed in foster care during a 12 month period Children with a screened-in report of maltreatment during a 12 month period Children/youth already in care 1st day of the year, who have been in foster care for 2 or more years

Statewide Data Indicators: Safety Maltreatment in Foster Care Re-report of Maltreatment

Maltreatment in Foster Care Of all children in care during a 12-month period, what is the rate of victimization per day of foster care? How it’s changed: Links AFCARS and NCANDS using child ID Includes all perpetrator types Controls for amount of time spent in foster care

Re-Report of Maltreatment Of all children with a screened-in report of alleged maltreatment in a 12-month period, what percent had another screened-in report within 12 months of their initial report? How it’s changed: Rather than limit the indicator to include victims only, we include all screened-in reports of alleged maltreatment that have reached disposition. Expanded timeframes: from 6 months to 12 months

Statewide Data Indicators: Permanency Entry Cohort Permanency in 12 months Re-entry Children in care 2+ years Permanency in 12 months Placement Stability

Exclusions/Adjustments Age 18 or older Foster care episodes less than 8 days

Entry Cohort Permanency in 12 months Of all children who enter care in a 12-month period, what percent discharged to permanency within 12 months of entering care? How it’s changed: Includes all entries, not just first-time Expanded from 6 to 12 months Expanded from reunification/live with relative to also include adoption and guardianship

Re-entry Of all children who entered foster care in a 12-month period who were discharged within 12 months of that entry to reunification, living with a relative, or guardianship, what percent re-entered foster care within 12 months of their discharge? How it’s changed: Limited to the entry cohort Expanded to include re-entry after guardianship

Companion measures

Children in Care 2+Years: Permanency in 12 months Of all children in foster care the first day of the year who had been in foster care (in that episode) for 2 years or longer, what percent discharged to permanency within the next 12 months? How it’s changed: This measure was a part of Composite 3 in Round 2. It serves to ensure that children and youth in care long periods of time achieve permanency

Placement Stability Of all children who enter care in a 12-month period, what is the rate of placement moves per day of foster care? How it’s changed: Entry cohort Controls for time in care Cumulative across episodes Does not count initial placement, but counts every move

How is performance on these statewide data indicators used for monitoring? Initial determination of compliance with National Standards Performance on statewide data indicators tracked as it relates to Program Improvement Plans

CFSR Round 3 : National Standards & Measuring State Performance

National Standards Methodology Set at the national observed performance for each statewide indicator e.g., Permanency by 12 months 95,000 230,000 =41.3% Will often be similar to national average

National Standards Why national performance? Easily communicated and understood Rooted in strategies central to an effective performance management system Ambitious – Most states will need to improve on at least one indicator Feasible – Most states will do well on at least one indicator

Measuring States’ Performance Methodology Calculate each state’s performance using a multi-level model Yields performance that takes into account: Characteristics of each state’s case mix e.g., risk factors unique to different groups of children Random measurement error e.g., due to small sample sizes (in small states or for rare outcomes) States’ long-run ability to achieve the desired outcome Risk Adjustment

Measuring States’ Performance Multi-level Analysis Widely accepted statistical technique Education (schools) Health care (hospitals) Child welfare (states, counties, etc.) Enables fair evaluation of relative performance among groups with different characteristics

Measuring States’ Performance Risk-Adjustment Goal: Minimize differences in outcomes due to factors over which the state has little control e.g., a state’s children are at greater risk for poorer outcome simply because of their age or history Takes into account factors that: differ across states AND can influence outcomes regardless of the quality of care a state provides

Measuring States’ Performance Risk-Adjustment e.g., Maltreatment in foster care Need to control for variables that Differ across states  Can affect outcomes

Measuring States’ Performance Risk-Adjustment Variables Research literature, expert panel and consultants Availability of data Statistically significant relationship to the outcome e.g., Age of child

Measuring States’ Performance Possible Risk-Adjustment Variables Child’s age Child’s sex Number of prior removals State foster care entry rate . . . The Children’s Bureau will finalize variables after receiving public comments

Measuring States’ Performance Observed vs. Risk-Adjusted State’s Performance Nation Observed Risk-adjusted

Measuring States’ Performance Observed vs. Risk-Adjusted e.g., Permanency by 12 Months (Entry Cohort) State A

Measuring States’ Performance Confidence Intervals 95% interval estimate around each state’s risk-adjusted performance e.g. 45% + or - 2%

Measuring States’ Performance Categorizing States No different than the nation Higher than the nation Lower than the nation States that fail to meet the National Standard will be required to include that indicator in a program improvement plan

Final Thoughts Approach similar to that used by: Centers for Medicare & Medicaid Services Education Health care

Child and Family Services Reviews Setting Targets and Tracking Performance CFSR Round 3

Objectives Establish meaningful and achievable performance goals for states Create a system that is statistically grounded Data points are simple to understand and easy to replicate Monitor and assess performance over time

CFSR Round 2 All states had to improve by the same percentage, regardless of baseline performance Did not take into account variability in state performance over time

Changes for CFSR Round 3 Goals set based on state’s own past performance Driven by variability in performance shown in a state over the last three years

Changes for CFSR Round 3 Performance will be easier to track Simple rates or percentages for individual measures

What if you meet the national standard? States meeting national standards during the PIP monitoring period will be credited as having met their goals

Setting Baselines Baseline= State’s observed performance on the indicator for the most recent year of data available before the beginning of PIP implementation.

Example Indicator: Percent of Children in care 2 or more years to Permanency in 12 months

How much to improve? Disclaimer: this is a high level overview and does not fully describe all steps involved.

How much to improve? Improvement Factor= Percent difference between the grand mean of the resampled population and 4 times the standard deviation from the grand mean Apply improvement factor to the baseline, to get a target

Example

Caps and Floors We will impose bounds on improvement factors generated by this model

Companion measures Entry Cohort Permanency in 12 months and the Re-entry indicator are companion measures If a State must meet a target for Entry Cohort Perm in 12, they cannot get worse than a threshold for Re-entry.

Thresholds Inverse of the target Example 1- State must improve on Perm in 12 Months. Therefore, they can’t get worse than the threshold on Re-entry. Note: States strive for lower rates of Re-entry

Thresholds Example 2- State must improve on Re-entry. Therefore, they can’t get worse than the threshold on Permanency.

CFSR Round 3 Data Profiles Reports state performance on statewide data indicators Context information

Other CFSR Round 3 materials available on Notice Of Statewide Data Indicators And National Standards For Child And Family Services Reviews: Published on April 23, 2014 @ https://federalregister.gov/a/2014-09001 Comment by May 23, 2014 @ http://www.regulations.gov/#!documentDetail;D=ACF_FRDOC_0001-0047 or mail to Miranda Lynch Thomas, 1250 Maryland Ave. SW., 8th Floor, Washington, DC 20024 Other CFSR Round 3 materials available on CFSR Portal - https://www.cfsrportal.org/ (resources page) CB Website - http://www.acf.hhs.gov/programs/cb/monitoring/child-family-services-reviews/