Using a Summary Score Approach to Analyze Performance Measures Over Time Charlie Ferguson, Ph.D. San Jose State University School of Social Work Demonstration Session 480 at the American Evaluation Association Annual Meeting Friday, October 18, 2013, 11:00 a.m. to 11:45 a.m.

Overview of the Workshop Introduction Overview of California’s Title IV-E Child Welfare Waiver Demonstration Capped Allocation Project (CAP) Overview of the CAP Evaluation The Issues The Solution Questions

Acknowledgements Administration for Children and Families, Children’s Bureau California Department of Social Services Alameda County Department of Children and Family Services Alameda County Probation Department Los Angeles County Department of Children and Family Services Los Angeles County Probation Department

Overview of the CAP What was the CAP? A five-year Title IV-E Child Welfare Waiver Demonstration Project sponsored by the Children’s Bureau. Who participated in the CAP? California Department of Social Services Alameda County Department of Children and Family Services Alameda County Probation Department—Juvenile Probation Los Angeles County Department of Children and Family Services Los Angeles County Probation Department—Juvenile Probation

Overview of the CAP California’s Child Welfare System: Shared governance structure CDSS: monitoring, oversight, and support County DCFS: day-to-day administration California’s Probation System: County operated, adult and juveniles Portion of juvenile population in out-of-home placements Departments seek Title IV-E reimbursement for eligible youth

Overview of the CAP What was the purpose of the CAP? To assist the child welfare and probation systems in developing and implementing alternative services to foster care to bring about better outcomes for children and families. What was the strategy of the CAP? Ending the Title IV-E entitlement and capping the funding allocation for five years in exchange for eliminating eligibility restrictions, gaining fiscal flexibility, and the opportunity to generate and retain savings.

Overview of the CAP What were the goals of the CAP? The goals of the Child and Family Services Review process: (a) improve the array of services for children and families and engage families through a more individualized approach that emphasizes family involvement; (b) increase child safety without an over-reliance on out-of-home care; (c) improve permanency outcomes; and (d) improve child and family well-being.

Overview of the CAP: Theory of Change Suspension of the entitlement Limit on the allocation Lifting of the eligibility restriction Flexibility in spending Beneficial methodology to determine capped allocation Growth rate in portions of allocation Five year budget horizon Opportunity to generate savings Opportunity to reinvest savings Opportunity to retain savings “Extra” allocation resulting from the methodology used to determine the CAP Reorientation of the system to: Prevention Early Intervention Permanency Placement in the most family-like setting possible Better outcomes for children & families Better systems- level outcomes Reinvestment Savings

Overview of the Evaluation Primary Purpose: To determine whether and how changes in the funding structure for foster care affects the functioning of county child welfare systems and relevant probation systems. Secondary Purpose: To assess outcomes for dependent and delinquent children and their families before the CAP began and then during its implementation.

Overview of the Evaluation The CDSS had specific requirements for the evaluation: No random assignment and (ultimately) no comparison counties. Limited intrusion on county departments for data collection. Use publicly available data from the University of California’s CA Child Welfare Indicators Project (data aggregated to the county department level).

Overview of the Evaluation The rationales behind the CDSS requirements: A general sentiment against the use of random assignment in any situation and not feasible in this instance. Concerns about identifying appropriate comparison counties and imposition regarding data collection. Limited political will or desire to strengthen the evaluation. A desire for consistency in outcome measures across large projects and child welfare reform efforts.

Overview of the Evaluation A separate evaluation conducted in each of the four departments. Each evaluation consisted of three components: Process Study Fiscal Study Outcome Study

Overview of the Evaluation Outcome Study Interrupted Time-Series Design: Trend Analysis Base: July 1, 2002 – June 30, 2007 Cap: July 1, 2007 – June 30, 2012 Participation Rates (four rates) Child Safety (five indicators) Exits to Permanency (five indicators) Placement Stability (three indicators) Appropriateness and Restrictiveness (two indicators) Data Source—CA Child Welfare Indicators Project http://cssr.berkeley.edu/ucb_childwelfare/ReportDefault.aspx

The Issues A need to reduce the complexity: (i.e., too many line charts) Multiple variables with multiple indicators across four distinct outcome evaluations. A change in the capacity of the CA Child Welfare Indicators Project over time allowed for the use of “context” counties. A desire to increase the rigor of the graphical analyses. Moving beyond a narrative description of the trend to include multiple vantage points. A need to make the results and conclusions readily accessible. Stakeholders need “simplicity” in terms of understanding and use.

The Solution The development and use of a Summary Score in the Analysis of Indicators Analysis of Outcomes Overall Assessment

The Solution The development and use of a Summary Score for the Analysis of Indicators Analysis of Outcomes Overall Assessment

The Solution Analysis of Indicators: Baseline and CAP Slope—generated in Excel using scatter plot and linear plot with equation function Celeration—Acceleration, Deceleration, Horizontal Stability Trend—Yes (trend in the desired direction); No (trend not in desired direction) National Standard/Goal—Count of Achieved / Possible

The Solution Analysis of Indicators: Comparison between Baseline and CAP Magnitude—Increase or Decrease between the end of Base and start of CAP (% pts) Change in Trend between Base and CAP—Yes; No; Continuing Change in Trend in Desired Direction—Yes; No; Yes Continuing; No Continuing

The Solution Analysis of Indicators: Comparison between Baseline and CAP Summary Score (inclusive of Base and CAP information) 3 possible pts.; 1 pt. for each of the following if true (a) A shift in Magnitude in the desired direction (b) At/Exceeded National Standard/Goal at least once during CAP (c) Change in Trend in desired direction or continuing in desired direction

The Solution Analysis of Outcomes: Summary Score—Count & Percent for each outcome variable Child Safety (comprised of 5 indicators): n & % Exits to Permanency (comprised of 5 indicators): n & % Placement Stability (comprised of 3 indicators): n & %

The Solution Overall Assessment: Summary Score—Count & Percent for all outcome variables Trend Lines—Count & Percent for Base vs. CAP National Standard/Goal—Count & Percent for Base vs. CAP

The Solution Cons Graphical Analyses (i.e., not statistical analyses) Still somewhat subjective and open to interpretation Pros Reduced the complexity of multiple graphical analyses Relatively simple process Allowed for analyses at three levels

The Solution Questions?

San Jose State University Thanks! Contact Information: Charlie Ferguson, Ph.D. School of Social Work San Jose State University One Washington Square San Jose, California 95192-0124 charlie.ferguson@sjsu.edu 510-846-7158