1. 4/15/2017
CFSR 2016
Children & Families Services Review (CFSR) Presented By: Alba Quiroz Garcia, CDSS July 31, 2014
Material from the May 2014 Administration for Children and Families, CFSR Briefing for States, Seattle, WA

2. 4/15/2017
Recap, what is the CFSR?
The federal government created the CFSR in response to the 1997 Adoption and Safe Families Act. It is an assessment of a state’s performance related to child welfare.
Indicators and standards are used along with information from a sample of cases to inform the federal government about determinations of whether a state is in substantial conformity with certain safety and permanency outcomes and to assess each state’s ability to achieve Safety, Permanency, and Well-being for children and families.
All States are assessed in the areas of child protection, foster care, adoption, family connections and independent living services.
The CFSR is conducted in two parts: 1) Statewide Assessment, and 2) Onsite Review.
Conducted by the Health and Human Services Administration for Children and Families (ACF)-Children’s Bureau (CB).

3. Why a CFSR information session in 2014?
4/15/2017
Why a CFSR information session in 2014?
With Rounds 1 and 2 of the CFSR completed, Round 3 is near and modifications are occurring presently.
This session is to help inform the field about
plans for statewide data indicators, national
standards, and the case review process that will be used in the next round of the CFSR.

4. Timeframe for California’s CFSR Round 3
4/15/2017
Timeframe for California’s CFSR Round 3
California’s Round 3 CFSR is scheduled
for February through September 2016.
Some states will begin Round 3 of the CFSR as early as Fall This means that changes still pending for the redesign of the indicators need to be finalized rapidly.

5. Why CFSR modifications?
4/15/2017
Why CFSR modifications?
With regard to data indicators & standards, the CB heard feedback from the field that the process could be improved by using different methods and measures that better reflect state practices and improvements. The challenges with prior indicators & measures ranged from methodological issues, ease of interpreting composite numbers, and concerns about comparability of state data.
Also, the CB now has a greater reliance on entry cohorts, fewer & simpler indicators, and they are dropping composites in favor of single item indicators.
Release of information memorandum to states on establishing and maintaining a Continuous Quality Improvement (CQI) system.
CQI explained in IM-12-07
Issuance Date: August 27, 2012

6. 4/15/2017
CQI
California is seeking to submit a letter of intent to the CB to use our case review process as part or the next CFSR.
For the state’s case review process to be considered, the following must be met:
Use the Federal Onsite Review Instrument (ORI). This tool collects and documents outcomes to Safety, Permanency and Well-Being.
Submit a letter of intent by July 2015 to the CB. The letter must be accompanied by a plan that meets all requirements outlined in the federal register (released April 2014).
Be able to replicate a sample from a recent AFCARS period.
Use a sample period of April 1-Sept 30,
The CB is not certifying the CQI process, rather they are approving or not approving our state’s case review process. Approval status is expected as early October 2015, but no later than December 2015.
Traditional Case Review Process or submit to use our own case review process.
ORI: This tool collects and documents outcomes to Safety, Permanency and Well-Being.
The Onsite Review in the past was conducted by a federal/state team combination in three counties. By federal rule, one of the sites must be the county that is home to the state’s largest metropolitan subdivision. The other two are jointly selected by the State and the CB. This team evaluates the state’s performance by:
1. Reviewing case records;
2. Interviewing children, youth, and families engaged in services; and
3. Interviewing other stakeholders, such as the courts and community agencies, foster families, case-workers, supervisors, and service providers.
Federal/state team members interviewed youth, family members, court
In addition, 65 cases were reviewed for compliance with federal child welfare requirements. Once the onsite review was completed, states that haven’t achieved substantial conformity in all areas of the review are required to develop and implement a Program Improvement Plan (PIP) to address the areas of nonconformity. California’s latest Onsite Review was conducted February 2008 in Los Angeles, Fresno and Santa Clara Counties.

7. 4/15/2017
CQI Continued…
If the state is approved, there is no limit on how many cases the state can review during the timeframe.
However, if a state chooses or has to do a traditional CFSR, the CB will only review 65 cases total (statewide).
For Round 3, case reviews do not need to be done in pairs as was the case in the previous two CFSR Rounds.

8. Systemic Factors and Statewide Indicators
4/15/2017
Systemic Factors and Statewide Indicators
The seven (7) Systemic Factors and Statewide Indicators remain present for CFSR Round 3.
Both will be utilized to determine if the state is in substantial conformity and meeting federal targets for Safety, Permanency and Well-Being.
7 Systemic Factors and 6 Indicators

9. 4/15/2017
Systemic Factors
There will be a focus on systemic factor functioning in Round 3. In CFSR Rounds 1 and 2 the focus was on having “requirements in place.”
Seven (7) Systemic Factors:
1. Statewide Information System
2. Case Review System
3. Quality Assurance System
4. Staff and Provider Training
5. Service Array and Resources Development
6. Agency Responsiveness to the Community
7. Foster/Adoptive Parent Licensing, Recruitment & Retention
7 Systemic Factors and 6 Indicators

10. Systemic Factors Continued…
4/15/2017
Systemic Factors Continued…
The CB can make a determination about substantial conformity for 6 of the 7 systemic factors using the data provided in the Statewide Assessment, with the exception of Service Array. Service Array cannot be determined through the statewide assessment and will require stakeholder interviews (there is no way that states can get around this).
Since the CB can make determinations about substantial conformity for most, but not all systemic factor, there is a possibility of reduction in scope and number of stakeholder interviews.

11. Systemic Factors Continued…
4/15/2017
Systemic Factors Continued…
At least 95% of applicable cases must meet the target to be determined “In Substantial Conformity.”
For Round 3, there is a reduction in Systemic Factor Items:
In CFSR Round 1 and 2 there were 22, for Round 3, there are 18.
1 factor less in each:
Quality Assurance System
Service Array and Resources Development
Agency Responsiveness to eth Community
Foster/Adoptive Parent Licensing, Recruitment & Retention

12. 4/15/2017
Six (6) statewide data indicators will serve as basis for determining whether or not states meet national standards.
Two (2) for Safety
Maltreatment
Re-Report of Maltreatment
Four (4) for Permanency
Percent of the entry cohort that achieves permanency in 12 months.
Percent of those children who subsequently re-enter care.
Percent of children and youth who have been in care for 2 years or longer who achieve permanency in the next 12 months.
A placement stability indicator for children entering care
No composites.
Since indicators are not finalized, states do not know at this moment how they are performing or exactly what the methodology will be for calculations of these indicators.

13. Safety Indicators-Children are protected from abuse and neglect.
4/15/2017
Safety Indicators-Children are protected from abuse and neglect.
1. Maltreatment in Foster Care (FC)
This asks, “Of all children in care during a 12-month period, what is the rate of victimization per day of FC?”
The denominator looks at all children in care during a 12- month period and looks at the total number of days these children spent in care as of the end of the year.
The numerator looks at those same children who are in the denominator and counts the total number of substantiated or indicated reports of maltreatment (by any perpetrator) that occurred during a foster care episode.
Includes all perpetrator types and controls for amount of time spent in FC.

14. Safety Indicators-Children are protected from abuse and neglect.
4/15/2017
Safety Indicators-Children are protected from abuse and neglect.
II. Re-Report of Maltreatment
Of all children with a screened-in report of alleged maltreatment in a 12-month period (regardless of disposition type), what percent had another screened-in report within 12 months of their initial report?
Expanded timeframes: from 6 months to 12 months
Rather than limit the indicator to include victims only, the CB includes all screened-in reports of alleged maltreatment that have reached disposition.

15. Safety Indictors Continued…
4/15/2017
Safety Indictors Continued…
Rather than limit the indicator to substantiated cases only, the CB is expanding this indicator to count all children with screened-in reports of alleged maltreatment.
Why? The CB believes that looking at repeat reports, regardless of disposition type, is a viable measure of an agency's’ attempts to prevent maltreatment, based on research indicating that families with screened-in but unsubstantiated reports are at a high risk of re- report.
Using all screened-in reports rather than limiting the [indicator] to substantiated cases makes states more comparable to each other and even against their own performance over time, when considering the growing implementation of differential response (DR), where a substantial percentage of cases may not receive a formal investigation.

16. 4/15/2017
Permanency Indicators-Children have permanency and stability in their living situations.
Rather than have composites made up of multiple measures, each of these indicators will be assessed individually on a particular cohort of children in FC for the initial determinations of compliance. In some cases, for purposes of monitoring during the PIP, the CB will treat another indicator as a companion indicator (this is new).

17. I. Entry Cohort Permanency in 12 Mo.
4/15/2017
I. Entry Cohort Permanency in 12 Mo.
This first indicator asks, “Of all children who entered care in a 12-month period, what percent discharged to permanency within 12 months?”
The difference here is the attention on permanency, so this means that while the majority of discharges in a short time frame will be reunification, the CB will also count exits to living with a relative, adoption, or guardianship as a success on this indicator.
In the last CFSR, the CB had a similar measure that looked at entry cohorts, but was limited to first-time entries. The CB is now counting all entries during the year and expanded the timeframe from a 6 to12 month cohort. The past CFSRs also only counted reunifications.
By expanding the denominator to a full year, the CB can achieve more stable and reliable estimates, which is important because those children achieving permanency on this indicator will also serve as the denominator for the re-entry indicator.

18. II. Re-Entry Permanency Indicator
4/15/2017
II. Re-Entry Permanency Indicator
The second indicator looks at children in the entry cohort who discharged to permanency within 12 months, and tracks them for another 12 months after exit to see whether they re-enter foster care.
It excludes exits to adoption because they cannot be as reliably tracked.
For all exits to reunification, living with a relative, or guardianship the CB wants to be sure they were not done without adequate planning or services to support a permanent placement, resulting in a greater risk of re-entry to care.
This indicator is limited to the entry cohort, and does not include all exits to reunification that occur during the year, as it did before. The denominator now including exits to guardianship and this represents another change.
Excludes Adoptions because they can not be tracked consistently across all states.

19. III. Children in Care 2+ Yrs. Permanency 12 Mo.
4/15/2017
III. Children in Care 2+ Yrs. Permanency 12 Mo.
Looks at the cohort of children in FC on the 1st day of a year who already have been in FC for 2 years or longer.
The CB is interested in knowing what proportion of this cohort achieves permanency within 12 months of the 1st day? So the focus is on getting these children and youth to permanent homes. Again, the CB will count all discharges to reunification, living with a relative, adoption, or guardianship as a success.

20. IV. Placement Stability
4/15/2017
IV. Placement Stability
Has been refined to have one indicator instead of three.
The proposed indicator looks at the rate of placement moves in the first year of FC, for the entry cohort.
Similar to the way the CB calculates maltreatment in FC, the denominator for this indicator is the sum of all days children spent in FC, to control for various lengths of time in care. For the numerator, the CB looks at how many times children were moved from one placement to another, while they were in care.
This is actually cumulative across placement episodes in the same year. The CB will NOT count their initial placement as a move, but any change is counted. While the CFSR Round 2 indicators did not differentiate children who moved twice from those who had moved more times, this indicator counts every move to accurately capture the rate of movement, rather than the number of children affected.

21. National Standard-Methodology
4/15/2017
National Standard-Methodology
It is set at the national observed performance for each statewide indicator.
Example: For most of the indicators it is a simple formula: the number of children in the nation with the outcome divided by the number of children in the nation eligible for the outcome.
In reviewing the permanency-by-12-months indicator... Assume we have 220,000 children in the nation that achieve permanency by 12 months out of 500,000 children in the nation that were eligible. So quite simply, the national standard for this indicator is 44%. This will often be similar to the national average.
According to CB analysis, most states will need to improve on at least one indicator.
220,000/500,000

22. 4/15/2017
Data Sources
There is continued reliance on AFCARS and NCANDS data. The CB will also link the two using the child ID.
Performance on these indicators will be provided to states via data profiles, to assist in monitoring Program Improvement Plans (PIP) that may address one or more of these indicators.
AFCARS, which is the Adoption and Foster Care Analysis and Reporting System, contains case-level data on all children in foster care and receiving title IV-E funding or who are adopted with child welfare agency involvement.
It is federally mandated reporting by state child welfare agencies, and we receive files every 6 months.
The NCANDS data set, which is the National Child Abuse and Neglect Data System, contains a Child File with case-level detail on children for whom agencies received a screened-in report of alleged maltreatment. It is a voluntary dataset reported by state child welfare agencies and submitted once a year.

23. Data Sources Continued
4/15/2017
Data Sources Continued
One criticism of the measurement techniques used in prior CFSR rounds was a heavy reliance on exit cohorts and point- in-time cohorts, rather than using prospective entry cohorts. Exit cohorts and cross-sectional cohorts can distort system performance outcomes because they represent biased subsets of all children who are served by the agency in a year.
For example, looking only at children who exit foster care in a given year ignores those children who did not leave. Looking only at those in foster care at the end of the year (or at any point in time) biases the sample to include more children with long lengths of stay in foster care. Entry cohorts can avoid these problems and provide a more fair assessment of overall system performance. Thus, the CB has shifted toward a greater focus on entry cohorts.

24. Foster Care Exclusions
4/15/2017
Foster Care Exclusions
Youth who are already age 18 or older at the start of the indicator (that is, at the age of entry into care or who are already 18 at the outset of an indicator) will be excluded.
Although CA and other states do provide FC to youth age 18 and older, there is no consistent inclusion of this population across states.
Any FC episode lasting less than 8 days will be excluded from these indicators. These very short-term placements are more prevalent in some states than in others, so this is another way to eliminate a bias and make states more comparable in those outcomes.

25. Measuring States’ Performance-Methodology
4/15/2017
Measuring States’ Performance-Methodology
The goal of risk adjustment is to minimize the differences in outcomes due to factors over which the state has little control. It takes into account the characteristics of each state’s case mix, something that has been a concern for stakeholders and people in the field for many years.
For example, a state’s children may be at greater risk for a poor outcome simply because of their age or history, which might be a characteristic of the child population in that state.
By performing risk adjustment, the CB can take into account factors that differ across the states and can influence the outcomes, regardless of the quality of care a state provides.

26. Measuring States’ Performance-Risk-Adjustment Possible Variables
4/15/2017
Measuring States’ Performance-Risk-Adjustment Possible Variables
Age of child, as an example, is a risk adjustment variable that the CB is considering, others being considered by the CB are: child’s gender/sex, number of prior removals, state FC entry rate.
The CB will finalize the variables after they receive and review public comments.

27. Measuring States’ Performance Risk-Adjustment Continued…
4/15/2017
Measuring States’ Performance Risk-Adjustment Continued…
Risk Adjustment—Maltreatment in Foster Care Example
Use maltreatment in FC, hypothetically the average age of a child in FC across the country is 10 years old. In state A, the average age is 5 years old. State A can perform worse than other states simply because of the age of its children, putting them at more risk. Since we know from field analysis of NCANDS data and from the vast literature on maltreatment, that younger children are more at risk for maltreatment, regardless of the quality of care being provided….So in this case, risk adjustment would control for the age of the children in the state compared to the average age of the children across the nation.

28. Measuring States’ Performance—Confidence Intervals
4/15/2017
Measuring States’ Performance—Confidence Intervals
For example, 45%, plus or minus 2%, would represent the interval estimate. The CB can be 95% confident that a state’s overall performance in the long run lies somewhere between the lower and upper limit of the state’s confidence interval.
Example: State A, their risk-adjusted performance was 45%, but because of some error and uncertainty, the CB can be 95% confident that their true performance is somewhere between 43% and 47%.

29. Measuring States’ Performance-Categorizing States
4/15/2017
Measuring States’ Performance-Categorizing States
So the confidence interval is used to help determine where a state is relative to the national standard. The CB will categorize states into three groups:
No different than the nation,
Higher than the nation, or
Lower than the nation
States that fail to meet the national standard will be required to include that indicator in a PIP.
Regardless of which indicators are included in the state’s PIP, the CB will provide each state with a data profile that includes information on the state’s performance on all of the indicators.
The real goal of risk adjustment is to come up with an estimate of performance that can be compared more equitably across other states and against the national standard.
When CB compares a state’s performance to the national level of performance, they are not comparing the state’s observed performance to the national performance. Instead they are comparing the state’s risk adjustment performance to the national performance.
Determining the state’s risk adjusted performance requires analyzing a national data set containing child level data from every state.

30. 4/15/2017
Companion Indicators
In addition to setting targets for all statewide data indicators that may be included in a PIP, there will also be thresholds established for companion indicators.
The companion indicators are entry cohort permanency in 12 months and the re-entry indicator…because efforts to improve on the entry cohort permanency in 12 months could have the consequence of increasing the rate of re-entry, states should not get worse than the threshold for re-entry.
Each of the indicators stands alone for initial determinations of compliance, but if a state includes one of these two measures in its program improvement plan, the CB will try to prevent states from choosing strategies to improve one indicator at the expense of another.
If a state begins a PIP to address permanency in 12 mo. for their entry cohort, they must also not show a significant worsening in performance on re-entry. The same is true for the inverse.

31. Resources http://www.childsworld.ca.gov/PG1520.htm
4/15/2017
Resources