1. CHILD AND FAMILY SERVICES REVIEWS (CFSRs)
Sustaining the Momentum: The Next Round of Reviews
Children’s Bureau Plan for CFSR Statewide Data Indicators and National Standards

2. Child and Family Services Reviews Statewide Data Indicators
CFSR Round 3

3. Measurement
Focus of this presentation:
Statewide Data Indicators

4. Sources of data
AFCARS (Adoption and Foster Care Analysis and Reporting System)
Contains case level data on all children in foster care and receiving title IV-E funding or who are adopted with child welfare agency involvement.
Federally mandated reporting by state child welfare agencies
Submitted every six months

5. Sources of data NCANDS (National Child Abuse and Neglect Data System
Child File contains case level detail on children for whom agencies received a screened-in report of alleged maltreatment.
Voluntary dataset reported by state child welfare agencies
Submitted every year
Or..
Alternate source data
Safety data only
Must be approved by the Children’s Bureau

6. CFSR Round 2 Safety measures: Permanency Composites
Absence of recurrence of maltreatment
Absence of maltreatment in foster care
Permanency Composites
Composite 1: Timeliness and permanency of reunification
Composite 2: Timeliness of adoptions
Composite 3: Permanency for children/youth in care long periods of time
Composite 4: Placement stability

7. We still want to know…
Are children getting to permanent homes quickly, and remaining there, without coming back into care?
Are they safe while in the care of the state?
Are children reported to the agency for safety concerns being reported subsequently?
And are they in stable placements, keeping the number of moves to a minimum?

8. Changes to measurement
Greater reliance on entry cohorts
Fewer and simpler indicators
No composites

9. Cohorts
Entry Cohorts
Children placed in foster care during a 12 month period
Children with a screened-in report of maltreatment during a 12 month period
Children/youth already in care 1st day of the year, who have been in foster care for 2 or more years

10. Statewide Data Indicators: Safety
Maltreatment in Foster Care
Re-report of Maltreatment

11. Maltreatment in Foster Care
Of all children in care during a 12-month period, what is the rate of victimization per day of foster care?
How it’s changed:
Links AFCARS and NCANDS using child ID
Includes all perpetrator types
Controls for amount of time spent in foster care

12. Re-Report of Maltreatment
Of all children with a screened-in report of alleged maltreatment in a 12-month period, what percent had another screened-in report within 12 months of their initial report?
How it’s changed:
Rather than limit the indicator to include victims only, we include all screened-in reports of alleged maltreatment that have reached disposition.
Expanded timeframes: from 6 months to 12 months

13. Statewide Data Indicators: Permanency
Entry Cohort Permanency in 12 months
Re-entry
Children in care 2+ years Permanency in 12 months
Placement Stability

14. Exclusions/Adjustments
Age 18 or older
Foster care episodes less than 8 days

15. Entry Cohort Permanency in 12 months
Of all children who enter care in a 12-month period, what percent discharged to permanency within 12 months of entering care?
How it’s changed:
Includes all entries, not just first-time
Expanded from 6 to 12 months
Expanded from reunification/live with relative to also include adoption and guardianship

16. Re-entry
Of all children who entered foster care in a 12-month period who were discharged within 12 months of that entry to reunification, living with a relative, or guardianship, what percent re-entered foster care within 12 months of their discharge?
How it’s changed:
Limited to the entry cohort
Expanded to include re-entry after guardianship

17. Companion measures

18. Children in Care 2+Years: Permanency in 12 months
Of all children in foster care the first day of the year who had been in foster care (in that episode) for 2 years or longer, what percent discharged to permanency within the next 12 months?
How it’s changed:
This measure was a part of Composite 3 in Round 2.
It serves to ensure that children and youth in care long periods of time achieve permanency

19. Placement Stability
Of all children who enter care in a 12-month period, what is the rate of placement moves per day of foster care?
How it’s changed:
Entry cohort
Controls for time in care
Cumulative across episodes
Does not count initial placement, but counts every move

20. How is performance on these statewide data indicators used for monitoring?
Initial determination of compliance with National Standards
Performance on statewide data indicators tracked as it relates to Program Improvement Plans

21. CFSR Round 3 : National Standards & Measuring State Performance

22. National Standards Methodology
Set at the national observed performance for each statewide indicator
e.g., Permanency by 12 months
95, ,000 =41.3%
Will often be similar to national average

23. National Standards Why national performance?
Easily communicated and understood
Rooted in strategies central to an effective performance management system
Ambitious – Most states will need to improve on at least one indicator
Feasible – Most states will do well on at least one indicator

24. Measuring States’ Performance Methodology
Calculate each state’s performance using a multi-level model
Yields performance that takes into account:
Characteristics of each state’s case mix
e.g., risk factors unique to different groups of children
Random measurement error
e.g., due to small sample sizes (in small states or for rare outcomes)
States’ long-run ability to achieve the desired outcome
Risk Adjustment

25. Measuring States’ Performance Multi-level Analysis
Widely accepted statistical technique
Education (schools)
Health care (hospitals)
Child welfare (states, counties, etc.)
Enables fair evaluation of relative performance among groups with different characteristics

26. Measuring States’ Performance Risk-Adjustment
Goal: Minimize differences in outcomes due to factors over which the state has little control
e.g., a state’s children are at greater risk for poorer outcome simply because of their age or history
Takes into account factors that:
differ across states AND
can influence outcomes
regardless of the quality of care a state provides

27. Measuring States’ Performance Risk-Adjustment
e.g., Maltreatment in foster care
Need to control for variables that
Differ across states  Can affect outcomes

28. Measuring States’ Performance Risk-Adjustment Variables
Research literature, expert panel and consultants
Availability of data
Statistically significant relationship
to the outcome
e.g., Age of child

29. Measuring States’ Performance Possible Risk-Adjustment Variables
Child’s age
Child’s sex
Number of prior removals
State foster care entry rate
. . .
The Children’s Bureau will finalize variables after receiving public comments

30. Measuring States’ Performance Observed vs. Risk-Adjusted
State’s Performance Nation
Observed
Risk-adjusted

31. Measuring States’ Performance Observed vs. Risk-Adjusted
e.g., Permanency by 12 Months (Entry Cohort)
State A

32. Measuring States’ Performance Confidence Intervals
95% interval estimate around each state’s risk-adjusted performance
e.g. 45% + or - 2%

33. Measuring States’ Performance Categorizing States
No different than the nation
Higher than the nation
Lower than the nation
States that fail to meet the National Standard will be required to include that indicator in a program improvement plan

34. Final Thoughts Approach similar to that used by:
Centers for Medicare & Medicaid Services
Education
Health care

35. Child and Family Services Reviews Setting Targets and Tracking Performance
CFSR Round 3

36. Objectives
Establish meaningful and achievable performance goals for states
Create a system that is statistically grounded
Data points are simple to understand and easy to replicate
Monitor and assess performance over time

37. CFSR Round 2
All states had to improve by the same percentage, regardless of baseline performance
Did not take into account variability in state performance over time

38. Changes for CFSR Round 3
Goals set based on state’s own past performance
Driven by variability in performance shown in a state over the last three years

39. Changes for CFSR Round 3 Performance will be easier to track
Simple rates or percentages for individual measures

40. What if you meet the national standard?
States meeting national standards during the PIP monitoring period will be credited as having met their goals

41. Setting Baselines
Baseline= State’s observed performance on the indicator for the most recent year of data available before the beginning of PIP implementation.

42. Example
Indicator: Percent of Children in care 2 or more years to Permanency in 12 months

43. How much to improve? Disclaimer: this is a high level overview
and does not fully describe all steps involved.

44. How much to improve? Improvement Factor=
Percent difference between the grand mean of the resampled population and 4 times the standard deviation from the grand mean
Apply improvement factor to the baseline, to get a target

45. Example

46. Caps and Floors
We will impose bounds on improvement factors generated by this model

47. Companion measures
Entry Cohort Permanency in 12 months and the Re-entry indicator are companion measures
If a State must meet a target for Entry Cohort Perm in 12, they cannot get worse than a threshold for Re-entry.

48. Thresholds Inverse of the target
Example 1- State must improve on Perm in 12 Months. Therefore, they can’t get worse than the threshold on Re-entry.
Note: States strive for lower rates of Re-entry

49. Thresholds
Example 2- State must improve on Re-entry. Therefore, they can’t get worse than the threshold on Permanency.

50. CFSR Round 3 Data Profiles
Reports state performance on statewide data indicators
Context information

51. Other CFSR Round 3 materials available on
Notice Of Statewide Data Indicators And National Standards For Child And Family Services Reviews:
Published on April 23,
Comment by May 23, or mail to Miranda Lynch Thomas, 1250 Maryland Ave. SW., 8th Floor, Washington, DC 20024
Other CFSR Round 3 materials available on
CFSR Portal - (resources page)
CB Website -