1. CFSR2 Data Indicators: The Good, the Bad, and the Ugly
Barbara Needell, MSW, PhD Emily Putnam-Hornstein, MSW
Center for Social Services Research University of California at Berkeley
We thank the Children's Bureau for the material contained in this presentation. 
For additional information about CFSR2 see:
The Performance Indicators Project at CSSR is supported by the California Department of Social Services and the Stuart Foundation
The purpose of today’s presentation is to acquaint you with the data indicators for the second round of the CFSR
While I will be going over how those indicators were generated using Principal Component Analysis, the key is not understanding the statistics/weights/etc. that were used (VERY complicated – that’s what statisticians are for)
Rather, the important take-away should be an understanding what different measures will be used to evaluate performance…and how these can be used to support planning and program decision making

2. Overview of Key Changes…
Move from individual measures to data composites for evaluating permanency (no such change for evaluating safety)
Adjustment of National Standards based on most recent data available
Measures reversed (where necessary) so that higher score always equals better performance (easier to interpret)
Some changes in time periods for given measures
Briefly, there have been a few key changes that will be discussed:
Move from individual measures to 4 indicators (comprised of measures, components and composites) for evaluating permanency. Evaluation of safety using two indicators (not comprised of multiple measures)…much more to follow on all this later.
National Standards have been adjusted based on the most recent state performance data available
Some measures have been reversed so that higher percentage equals better performance – allows for easier interpretation
Changes in time periods – e.g., maltreatment in care now reflects children served in foster care during a 12-month period rather than a 9 month period

3. Indicator 1 Safety Indicator 2 Composite 1 Composite 2 Permanency
Measure Measure Measure Measure 1
Component A Component B
Composite 1
Measure Measure Measure Measure Measure 1
Component A Component B Component C
Composite 2
While the terminology is a bit confusing – a visual should help.
States will be evaluated on Safety and Permanency.
Evaluation of Safety performance is fairly simple and is based on two national indicators (indicators 1 and 2). These are not created from multiple measures.
Evaluation of Permanency is more complicated. It is based on four composites which are indicators (as used in evaluating safety) except that they incorporate performance on MULTIPLE permanency-related measures. Composites are built from components and individual measures.
Can still think of six performance indicators (similar to the first round) – it’s just that four of the six indicators are comprised of multiple performance measures.
This will all become much clearer as we go through the specific indicators and composites...
Permanency
Measure Measure Measure 1
Component A Component B
Composite 3
Measure 1 Measure 2 Measure 3
Composite 4

4. Evaluating Safety: Safety Indicator 1 Safety Indicator 2
Of all children who were victims of substantiated or indicated abuse or neglect during the first 6 months of the reporting year, 94.6% or more did not experience a recurrence of maltreatment within a 6-month period
Of all children in foster care during the reporting year, 99.68% or more were not maltreated by a foster parent or facility staff member
Evaluating Safety
We’ll start with the two data “indicators” related to safety
(Composites were initially proposed, but rejected by the field.)
What is new?
National standards have been adjusted to reflect the most recent data available since the focus of the CFSR is on “recent practice”.
Indicators have been “reversed” (higher % = a higher performance level)
Safety Indicator 2 (maltreatment in foster care) now reflects children served in foster care over a 12-month period rather than a 9-month period

5. Evaluating Permanency:
There are four composites (or composite indicators) that will be used for evaluating permanency
Composite 1 = Timeliness and permanency of reunification
Composite 2 = Timeliness of adoption
Composite 3 = Achieving permanency for children in foster care for long periods of time
Composite 4 = Placement Stability
Why was decision made to use composites?
Composites incorporate a lot more information than individual measures
The hope was that this would encourage a more holistic view of performance and would better capture the experiences of all children in foster care – not just those children who are in foster care for 12 months or 24 months
What does a composite score mean?
A composite score represents performance on a whole set of measures.
It is scaled from Again, this will be much clearer as we go on, but all scaling does is make each of the composite scores a relative score. That is, each state’s composite score has no meaning apart from the other states’ scores. It simply tells us how each state performed as compared with all other states.
For each scaled composite score a percentile will also be calculated which will reflect the score’s place in the distribution of scores for that composite. For example, a percentile score of 50 would mean that the State did better than 50 percent of the states. A percentile score of 75 would mean that the state did better than 75 percent of the states.

6. Principal Components Analysis (PCA) (the “black box” version)
Three components based on related measures!
A bunch of measures…
Median Time in Care
Emancipating from Care
Recurrence of Maltreatment
black box of fancy statistical tools
Component #1
Timeliness of Reunification
Abuse in Foster Care
Component #2
Permanency of Reunification
Placement Stability
Component #3
Timeliness of Adoption
Before going through the measures used to create each of the four permanency composites, I’m going to walk you through the “black box” version of the statistical technique used to figure out which measures to use - Principle Components Analysis…
Imagine a bunch of individual foster care measures.
You have no idea how to evaluate performance based on all of these measures so you throw them all into a black box together.
Inside this box are statistical tools that reduce the number of individual measures, detect the structure of relationships between measures and combine them into components, and then tell you the contribution of each measure to the created component
That is Principal Components Analysis in a nutshell.

7. Puts measures in the same “range” Sets measures to the same “system”
Z-Scores?
Before dumping all of the measures into the PCA “Black Box”, they were transformed into standard scores (z-scores)
A z-score serves two purposes:
Puts measures in the same “range”
Sets measures to the same “system”
Z-score…
A z-score puts everything on the same scale. For example, we have some measures that, like this cat, are in the very high range. Such as our measure of permanency for children with TPR. The range is only from about 85 percent to 100 percent. But then, in that same composite, we have measures where performance ranges from 8 percent to 35.3 percent. Need to be able to equalize them. That is what a z-score does.
A z-score also makes equivalent measures that are not based on the same system. For example, we have measure of percentages and we have measures of number of months. To include them in the same analysis system, we need to make them equivalent. The z-score transformation does this.

8. Principal Components Analysis…
And an Example…
A researcher interested in measuring “success” in high school.
Collects the following measures for each student:
Athletic Ability
Good Grades
Physical Attractiveness
Interest in Sports
Chess Club Membership
Science Club Membership
Social Life
Principal Components Analysis…
And now a simple (non-child welfare) example…
A researcher is interested in studying high school success
For each student there are multiple measures: athletic ability, good grades, physical attractiveness, etc.
All of these measures are transformed to z-scores and then dumped into the Principal Components black box…

9. Determines relationships between individual measures:
Structures the data into independent components:
Explores the contribution of each part to the whole:
Determines relationships between individual measures:
Reduces the number of individual measures:
Interest in Sports Athletic Ability
Good Grades Chess Club Member Science Club Member
Physical Attractiveness Active Social Life
Athletic Ability
Jock Component =
Interest in Sports
VERY HIGHLY ASSOCIATED!!
Brainiac Component =
Good Grades
Chess Club Member
PCA determines the relationships between individual measures
Structures the data into independent components
Jock Component
Brainiac Component
Popular Kids Component
Reduces the number of individual measures by dropping a measure that is highly correlated with another (and therefore doesn’t make a unique contribution to the component)
Chess club membership and science club membership are very highly associated – there is no need to include both
Explores the contribution of each part to the whole
Athletic ability is a more important determinant of being a member of the jock group than is a student’s interest in sport…this will be weighted more heavily
Good grades matter more to being in the brainiac group than does membership in the chess club
Physical attractiveness and an active social life are equally important to being classified as a popular kid
Popular Kids Component =
Physical Attractiveness
Active Social Life

10. Composite 1: Timeliness and Permanency of Reunification
Returning to the world of child welfare…
Let’s now look at the individual measures for the first composite: Timeliness and Permanency of Reunification

11. Composite 1: Timeliness and Permanency of Reunification
(Component A)
Permanency
(Component B)
Timeliness and Permanency of Reunification (Composite 1)
Comprised of two components
Component A - Timeliness of Reunification
For children exiting foster care to reunification (in care for 8 days or longer), % exiting in less than 12 months of entry
For children exiting foster care (in care for 8 days or longer) to reunification, median number of months in foster care
For children entering foster care (for the first time) and remaining in foster care for 8 days or longer, % exiting to reunification in less than 12 months of entry
Component B – Permanency of Reunification
For children exiting foster care to reunification, % re-entering foster care in less than 12 months from discharge
A few important things to note about the measures and what is new:
Reunification in the CFSR measures has always included children with a discharge reason of either reunification with parents or living with relatives. This is continued for the second round of the CFSR.
All measures include children who were younger than 18. That is, the child had to be 17 for at least one day in the target year (12-month period). This was done because of the extensive variation across states in reporting children to AFCARS who were age 18 or older. Although it is known that states continue to serve children who are older than 18, the younger cut-off was used to ensure consistency across states.
Although three measures are shown under Component A and only one measure is shown under Component B – all four measures contribute to Components A and B. They are simply weighted differently (as we will talk about later).
An entry cohort measure has been added!! Entry cohort is based on kids entering in a 6-month period due to rolling annual files.
There is a median length of stay measure that provides information about kids who have been in care for longer than 12 months.
These measures now look at children in care for 8 days or longer (rather than 5 days or longer)
Trial Home Visit has been used to account for “physical reunification” – Federal law states that reunification occurs when child is returned home or to live with relatives and the State no longer has care and supervision responsibilities. No way to account for what people term “physical reunification” (SORRY BARBARA – I’M NOT SURE WHAT THIS LAST PART MEANS SO I JUST COPIED…)
Measure1
% of children exiting to reunification in less than 12 months
Measure2
Median # of months in care for children exiting to reunification
Measure3
% of first entries exiting to reunification in less than 12 months
Measure1
% of children re-entering care in less than 12 months from discharge

12. Composite 2: Timeliness of Adoption
Moving on to the second Composite…Timeliness of Adoptions

13. Composite 2: Timeliness of Adoption
Measure1
% exiting care to adoption in less than 24m of entry
Timeliness of Adoption (Composite 2)
This composite is comprised of three components:
Component A – Timeliness of Adoptions of Children Exiting Foster Care
For children exiting foster care to adoption, % exiting in less than 24 months from the time of entry
For children exiting foster care to adoption, median number of months in foster care.
Component B – Progress Toward Adoption of Children Who Have been in Foster Care for 17 Months or Longer
For children in foster care for 17 months or longer at start of reporting year, percent who exit to adoption by the end of the reporting year
For children in foster care for 17 months or longer at the start of reporting year, percent who become legally free for adoption within 6 months of the start of the reporting year.
Component C – Timeliness of Adoptions of Children Who Are Legally Free for Adoption
For children who are legally free for adoption, percent adopted in less than 12 months of becoming legally free.
And a few things to note:
Adoption continues to mean (as it did in CFSR round 1) a finalized adoption
Legally free means that there is a TPR date reported to AFCARS for all living parents
What is new?
A longitudinal assessment of timeliness of adoptions and TPR’s has been added for a cohort of children in foster care for 17 months or longer. Why 17 months or longer? This measure is intended to be consistent with ASFA (requirement that State child welfare agencies file a petition to terminate parental rights and pursue adoption for a child who has been in care for 15 of the most recent 22 months, unless an exception applies.) A child is considered to have entered foster care (for purposes of starting the clock) on the earlier of (1) the first judicial finding that the child has been subject to abuse/neglect, or (2) the date that is 60 days after the date on which the child was removed from the home. Since AFCARS does not collect information pertaining to the data of the first judicial finding, the 60-days (#2) is being used as the start date …thus 17 (continuous) months rather than 15.
The decision was made to not just look at children with a stated goal of adoption because this would ignore children for whom adoption (arguably) should be pursued because of their time in care…it would also miss those for whom adoption is being pursued, but where the State has not reported that as the goal.
Why legally free? The goal was to capture that benchmark once children become legally free.
Measure2
Median # of months in care for children exiting to adoption
Measure1
% in care 17m+ exiting to adoption
Measure2
% in care 17m+ who become free for adoption
Measure1
% of legally free adopted w/in 12m

14. Composite 3: Permanency for Children in Long Term Care
Permanency Composite 3…Permanency for children in foster care for long periods of time

15. Composite 3: Permanency for Children in Long Term Care
Permanency for Children in Long Term Care (Composite 3)
Two components:
Achieving Permanency for Children in Foster Care for Extended Periods of Time (Component A)
Of children discharged from foster care who were legally free for adoption, percent that were discharged to a permanent home
Of children in foster care for 24 months or longer (on day 1 FY), percent discharged to a permanent home by the end of the fiscal year.
Children Growing-up in Foster Care (Component B)
Of children emancipated from foster care prior to age 18 or reaching age 18 while in foster care, percent that were in foster care for 3 years or longer.
A few things to note:
24 months or longer was used as a much lower percentage of exits are observed…real drop in percentages of children achieving permanency in a 12-month period. Entry cohort – median 39.3 percent cohort – 24.5 percent median (AGAIN BARBARA…I COPIED THIS BUT WASN”T EXACTLY CLEAR HOW TO TRANSLATE IT FOR YOU…)
What is new?
This entire “composite” is new…goal is to increase the understanding of State performance with regard to outcomes experienced by children in foster care for long periods of time. The hope is that this will focus State attention on programmatic efforts to ensure that children do not grow-up in foster care (i.e., spend many of their formative years in foster care rather than a permanent home)
Measure1
% of legally free discharged to a permanent home
Measure2
% in care 24m+ discharged to a permanent home
Measure1
% emancipated or age 18 in care who have been in care for 3+ years

16. Composite 4: Placement Stability
Permanency Composite 4: Placement Stability

17. Composite 4: Placement Stability
Measure1
% in care for less than 12m with only 1 or 2 placements
Measure2
% in care for 12-24m with only 1 or 2 placements
Placement Stability (Composite 4)
No components for this composite indicator…just three measures
For children in foster care for 8 days or longer but less than 12 months, percent with 2 or fewer placement settings.
For children in foster care for 12 to 24 months, percent with 2 or fewer placement settings.
For children in foster care for 24 or more months, percent with 2 or fewer placement settings.
A few things to note:
A lot of comments were received about using 2 placement settings as a measure of stability. Many people wanted that increased for older children and for children in foster care for longer than 12 months. The 2 placement settings was not an arbitrary decision on the part of ACF…report to Congress workgroup. Many people on the workgroup had wanted it to be only one placement setting – but since a child sometimes has to go into an emergency placement setting first, 2 placements was decided to be reasonable.
What’s new?
Again, children in foster care for very short periods of time (less than 8 days) have been excluded
Expanded focus on placement stability over time…less than 12m, 12-24m, and 24m+
Measure3
% in care for 24m+ with only 1 or 2 placements

18. Timeliness & Permanency of Reunification
(Scaled) State Composite Score
(Transformed to position on national scale)
(Un-Scaled) State Composite Score
(County weighted scores summed and
divided by total # of children served)
(Weighted) County Composite Score (# of Children in Foster Care x Score)
(Un-Weighted) County Composite Score (Component A + Component B / 2)
Component A Timeliness of Reunification (Sum of Weighted Measures)
Now that we’ve gone through all of the measures, components, and composites, we’ll walk through the weighting process for computing a State’s Composite Score…
Although weights can be confusing (and it would be easier to weight everything equally) weighting allows the data to tell us what the relative contribution of each measure to each component and composite is…
We’ll use Composite 1: Timeliness & Permanency of Reunification, as an example
Start at the bottom and work-up…
Each individual measure (four measures for the reunification composite) is weighted for its respective contribution to the two components
The weighted measures are summed into Component A (Timeliness of Reunification) and Component B (Permanency of Reunification)
Component A and B are summed and divided by 2 to get an (Unweighted) County Composite Score
The County Composite Scores are weighted (weighted score = # of children in foster care x score)
All of the weighted county scores are summed and divided by the total # of children served to get an (Unscaled) State Composite Score
Finally, the State Score is transformed to a position on the national scale
Component B Permanency of Reunification (Sum of Weighted Measures)
Std Measure1 x Weight (0.462)
Std Measure2 x Weight (0.451)
Std Measure3 x Weight (0.295)
Std Measure4 x Weight (0.129)
Std Measure1 x Weight (0.085)
Std Measure2 x Weight (0.070)
Std Measure3 x Weight (-0.005)
Std Measure4 x Weight (1.025)

19. Making the Most of CFSR Scores…
Each state will be provided with a data profile that includes percentages for each measure within each composite.
The data profile also will provide the ranges, medians, and 25th and 75th percentiles of performance, so each state will know exactly how it stacks up with all others on each measure.
In California, we at CSSR will attempt to replicate each of the measures and composite scores, break them out by child welfare and probation agencies, and report/update quarterly.
The focus must be on performance on INDIVIDUAL MEASURES (with age, race, gender, etc breakouts), and an understanding of how that performance contributes to National Standard Performance.
More importantly, we must work to understand how performance on individual measures really relates to safety and permanence, what else we need to measure, and what we need to do to improve.

20. Changes to AB636 Recurrence of Abuse at 12 months
This means that a total of 11 new measures, plus 6 revised measures, must be a part of AB636 according to statute.
In order to eliminate redundancy and avoid unnecessary complexity, some current AB636 measures will be eliminated.
Recurrence of Abuse at 12 months
Recurrence when children are not removed
Time to reunification (State)
Time to adoption (State)
Placement Stability (State)
Foster Care Re-entry (State)
Least Restrictive Setting (Predominant Placement view)

21. When?
We plan to have a subset of the revised measures available in July 2007.
The July 2007 county reports will only contain measures that are going forward with minimal changes.
The rest of the measures will be added so that all will be available in time for the CFSR onsite review in February Data will be available on all new measures back to at least 2000.

22. Suggested Transitional Data Indicators July Outcomes Report

23. When?
We plan to have a subset of the revised measures available in July 2007.
The July 2007 county reports will only contain measures that are going forward with minimal changes.
The rest of the measures will be added so that all will be available in time for the CFSR onsite review in February Data will be available on all new measures back to at least 2000.

24. CSSR.BERKELEY.EDU/CWSCMSREPORTS
Barbara Needell
(pcs)
CSSR.BERKELEY.EDU/CWSCMSREPORTS
Needell, B., Webster, D., Armijo, M., Lee, S., Cuccaro-Alamin, S., Shaw, T., Dawson, W., Piccus, W., Magruder, J., Exel, M., Conley, A., Smith, J. , Dunn, A., Frerer, K., & Putnam Hornstein, E., (2007). Child Welfare Services Reports for California. Retrieved [month day, year], from University of California at Berkeley Center for Social Services Research website.
URL: <