Implications of CFSR 3 for IVE Programs Richard P. Barth & Terry V. Shaw October 23, 2014 CSWE APM Tampa, Florida

Purpose of the Presentation To stimulate thinking and discussion about: What is the meaning of CFSR 3? Does it create new opportunities for SSW University-Agency Partnerships? Does it suggest changes in SW Education?

CFSR 3 Guiding Principles States and the Federal Government are unified in a commitment to safety, permanency, and well-being for the children served by the public child welfare system. The CFSRs have led to the identification of opportunities for: 1. program improvement 2. improvement in outcomes for children and families. Further improvements in the CFSR and in the related CQI processes are critical to the continued CWs advances Scientifically valid Feasible Understandable Meet All Federal Requirements

National Standards (Issues) Points of Agreement Areas for Discussion Proposal Input Aggregate administrative data on safety and permanency Recommendation to endorse CB proposal; or Compute State-specific rates and calculate a national average rate Applied Methodology Multi-level Model Risk Adjusted Recommendation to endorse CB proposal; or Endorse with additional factors; or; Do not risk adjust Output Set National Standard at national average for all measures Proposal is reasonable 4

National Standards Safety Outcome 1: Children are, first and foremost, protected from abuse and neglect. Safety Performance Measure 1: Maltreatment in Foster Care CFSR Round 2 CFSR Round 3 Differences Of all children in foster care during the reporting period, what percent were not victims of substantiated or indicated maltreatment by a foster parent or facility staff member? Of all children in foster care during a 12-month period, what is the rate of victimization per day of foster care? (most likely reported in a rate per 100,000 days) Child maltreatment while in care is calculated by directly linking data from NCANDS (numerator) with the AFCARS file (denominator) using CHILDID. All instances of maltreatment will be included regardless of type of perpetrator (including parents).

Re-Report of Maltreatment (Issues) Points of Agreement Areas for Discussion Proposal Of all the children who were victims of substantiated or indicated abuse or neglect during the first 6 months of the reporting period, what percent experienced another screened-in report with a 12-month period? It is important to include a measure of repeat maltreatment.   No support for the measure as proposed “…..getting the measure right should trump concerns over current data capacities.” Recommendations for appropriate measure(s)?   Recommendations around data availability, quality and testing proposed measures? 9

National Standards Safety Outcome 1: Children are, first and foremost, protected from abuse and neglect. Safety Performance Measure 2: Re-report of Maltreatment CFSR Round 2 CFSR Round 3 Differences Of all children who were victims of substantiated or indicated maltreatment allegation during the first 6 months of the reporting period, what percent were not victims of another substantiated or indicated maltreatment allegation within a 6-month period?   Of all children with a screened-in report of alleged maltreatment in a 12-month period, what percent had another screened-in report within 12 months of their initial report? Lengthens the timeframes being considered (from 6 months to 12 months) increases the scope to include all screened-in reports and not just Substantiated or Indicated.

National Standards Permanency Outcome 1: Children have permanency and stability in their living situations. Permanency Performance Measure 1: Permanency in 12 months for children entering foster care CFSR Round 2 CFSR Round 3 Differences Composite 1.3: Of all children entering foster care for the first time in a 6-month period, what percent discharged to reunification (or live with relative) within 12 months of entering foster care or by the time they reached 18? Of all children who enter foster care in a 12 month period, what percent discharged to permanency within 12 months of entering foster care? Expanded the exits from reunification alone, to permanency (reunification, live with relative, adoption or guardianship). Expands base time from 6 months to 12 months and includes all entries, not solely first entries.

National Standards Permanency Outcome 1: Children have permanency and stability in their living situations. Permanency Performance Measure 2: Permanency in 12 months for children in foster care for 2 years or more CFSR Round 2 CFSR Round 3 Differences Composite 3.1: Of all children in foster care on the first day of a 12-month period who had been in foster care (in that episode) for 2 or more years, what percent discharged to permanency within 12 months of the first day or by the time they reached 18? Of all children in foster care on the first day of a 12-month period who had been in foster care (in that episode) for 2 or more years, what percent discharged to permanency within 12 months of the first day? No differences.

National Standards Permanency Outcome 1: Children have permanency and stability in their living situations. Permanency Performance Measure 3: Re-entry in 12 months CFSR Round 2 CFSR Round 3 Differences Composite 1.4: Of all children discharged from foster care to reunification or live with a relative in a 12- month period, what percent re-entered foster care in less than 12 months from the date of discharge? Of all children who enter foster care in a 12- month period and discharged within 12 months to reunification, live with relative, or guardianship, what percent re-entered foster care within 12 months of their date of discharge? The measure is now based on an entry cohort instead of an exit cohort. The denominator has been expanded and now includes Guardianship as well as reunification, and live with relative.

National Standards Permanency Outcome 1: Children have permanency and stability in their living situations. Permanency Performance Measure 4: Placement stability CFSR Round 2 CFSR Round 3 Differences Composite 4.1: Of all children served in foster care during the 12- month period, what percent had two or fewer placement settings? Of all children who enter foster care in a 12- month period, what is the rate of placement moves per day of foster care? (most likely expressed as a rate per 1,000 days) The measure is now based on an entry cohort instead of all children served and in care for a period. All placement moves will be counted during a period rather than just identification of stable vs. not stable.

Risk Adjusted Analyses And, yet, we might include child sex despite the absence of any data to suggest it should be used.

Performance Goals and Thresholds (issues) Points of Agreement Areas for Discussion Proposal Input State specific data on the most recent 3 years of performance Bootstrapping is reasonable Recommendation to incorporate trend consideration Applied Method Min. & Max. Goals & Thresholds Min. = Top of bottom 5th Max. = Bottom of top 5thth Use Companion Measures Applied Method Bootstrapping to set baseline Estimate grand mean and standard deviation Recommendation to adjust calculations to 1 SD Recommendation to set at 5th and 45th State performance level Out Put Improvement Factor Grand Mean/Grand Mean + 2SD Goal Calculation Baseline * Improvement Factor Threshold Calculation Baseline – 2SD Companion Measures are reasonable Output Measurement at 3 & 5 years for some measures and 4 & 5 years for one measurement 7 Measurement timeframes are reasonable

Case Review Standards (issues) Points of Agreement Areas for Discussion Proposal Case Review determines substantial conformity for 24 items that assess case-specific practices in both in-home and foster care cases. A single standard is applied to all Case Review items. Case Review is an important component of a CQI system.   A single standard should be applied. The Panel suggested that Case Review should not be used to determine Substantial Conformity.   How would Case Review results be used to meet legislative objectives to measure State performance and improvement? What should the standard be?

Case Review Standards In prior rounds onsite case reviews were conducted during a one-week visit by external reviewers. In round 3, qualifying states may conduct their own reviews using a revised CFSR onsite review instrument. States have to meet three criteria in order to use their own case review process and be able to meet the following time lines. Must use a sample period of 4/1 to 9/30 of year prior to review year. Conduct reviews from 4/1 to 9/30 of the year of review and provide information to the Children’s Bureau Report results no later than 11/15 of the year of review to the Children’s Bureau If a state receives approval to use its own case review process in conjunction with the federal CFSR onsite review instrument they must provide to CB its case review schedules and other relevant information. This will allow the federal team to participate in the state’s case review process. At a minimum, CB participation will include observing the state’s case review process while present in the state and reviewing completed instruments. As noted above, stakeholder interviews will also still be conducted jointly by the federal and state team during the period when case reviews are conducted. Federal oversight by CB may be necessary to ensure the state conforms to our case review criteria and produces reliable results for the CFSRs. If a state does not meet the necessary criteria to use their case review process or chooses not to do their own case reviews, CB will work with the state to prepare for a more traditional week-long case review conducted jointly by the state and CB.

Case Review Standards Criterion 1 - The states operates an internal case review process at least annually that assesses statewide practice performance for the key child welfare areas using a uniform sampling process and methodology Criterion 2 - The state has a process in place for ensuring accurate and consistent case review ratings. Criterion 3 - The state uses the federal onsite review instrument and its instructions using the sample and method established above to collect data to be used for the initial determination of conformity. Criterion 1 - The states operates an internal case review process at least annually that assesses statewide practice performance for the key child welfare areas using a uniform sampling process and methodology. This means that the sampling methodology must address the following specific criteria: • A statewide schedule that selects cases randomly from the entire state universe; or a stratified schedule of counties or jurisdictions, which consists of a cross-section of state child welfare practice and includes the largest metropolitan area and significant tribal or other populations that is representative of state demographics. That same stratification will then be replicated for ongoing performance measurement. • The state uses a simple random sample design but may include additional stratification to achieve an adequate representation of key program areas. • The sample consists of a minimum of 65 cases served during the sample period with a minimum of 40 foster care cases and 25 in-home cases, inclusive of alternative response cases. Samples larger than 65 should reflect the state ratio of foster care and in-home cases as long as the minimums are met for both case types. • The sampling frame for the state foster care population consists of the listing of children served statewide or by jurisdiction strata according to the states’ AFCARS defined reportable cases for the CB defined sample period. To allow for ongoing review to occur timely, a state may use its AFCARS defined reportable cases for each day in a quarter. Criterion 2 - The state has a process in place for ensuring accurate and consistent case review ratings. This means: • The state provides consistent training of all reviewers on the case review process. • The state adheres to instructions contained in the case review instrument for rating cases. • The state has a process in place to ensure consistency of ratings across multiple sites and reviewers, and includes third party (i.e., someone who has not reviewed the case) quality assurance of cases reviewed for accuracy of ratings in accordance with the instrument and instructions. • The state ensures that individuals who had direct contact, supervision, oversight or consultation for the case being reviewed do not complete the case review or quality assurance. Criterion 3 - The state uses the federal onsite review instrument and its instructions using the sample and method established above to collect data to be used for the initial determination of conformity. • The state uses CB’s onsite review instrument and its instructions to collect information on all necessary items and implements and adheres to guidance CB issues to accompany the instrument. • The state includes case specific interviews of key informants on every case to inform the ratings, including all of the following individuals: child (if age and developmentally appropriate), parents, caregiver/foster care provider, and caseworker or supervisor, and follows a written protocol for acceptable case specific exceptions to an interview.

Sampling (issues) Areas for Discussion Points of Agreement Proposal Sample to allow for statistical inferences Combined foster care and in-home case sample Simple random sampling design Statewide with options for geographic representation Option to reduce sample size for ongoing program improvement No support for a sampling approach that could eliminate children in rural areas   No support for reducing sample size for ongoing program improvement Should sample size be increased to allow for statistical inferences for foster care and in-home cases? Is the proposed sample size feasible? Is the proposed random sampling design feasible? What are alternative sampling strategies? Panel suggestion that targeted reviews could have a role in ongoing improvement monitoring.

Sampling Random selection; or a stratified selection representative of the states demographics and includes the largest metropolitan area. Minimum of 65 cases served during the sample period (40 foster care cases; 25 in-home cases, including Alternative Response); Out of home cases must be consistent with the listings reported to AFCARS during the period. In home cases (including AR) must have been opened for at least 45 consecutive days during the sampling period . The state must be able to consistently address document case elimination.

Standards for Systemic Measures (issues) Points of Agreement Areas for Discussion Proposal The State’s CQI system would identify sources for quantifiable data for items TR1 – TR4 and FAP 2. A single standard is applied to all systemic capacity items. The prior method of assessing systemic measures was subjective. The evidence connecting the systemic factors to family outcomes is limited. What is the recommended standard and why? 90% 75% Should the standard be different for systemic capacity items assessed through case review?

Implications The measures are simpler which will help in explaining what they mean to all the various audiences - this should make it clearer to develop strategies to influence outcomes. Opportunities for SSWs to increase their role regarding development and implementation of CQI processes Opportunities to work with state partners on the importance of understanding the trajectory of children into and through the service systems Greater focus on CQI in social work education Critiquing CFSR 3 or Designing CFSR 4 could become a useful activity for a research or CWS course