CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Pro Bono Publico: Avoiding Child Welfare Data Abuse Panel on Meaningful Measurement in the Context of Litigation and Consent Decrees Daniel Webster, MSW, PhD Center for Social Services Research, University of California at Berkeley Isabel Blanco Deputy State Child Welfare Director, South Carolina Andy Barclay Fostering Court Improvement, Atlanta, GA Susan Smith, PhD Casey Family Programs Data Advocacy Group Presentation Originally Created by: Barbara Needell, MSW, PhD Melissa Correia, MSW Emily Putnam-Hornstein, MSW, PhD Bryn King, MSW AAPWA Preconference Symposium Demonstrating Sustained Well Being of Children and Youth in a Compliance Environment San Diego, CA September 9, 2012

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley The Current Placement System* (highly simplified) *adapted from Lyle, G. L., & Barker, M.A. (1998) Patterns & Spells: New approaches to conceptualizing children’s out of home placement experiences. Chicago: American Evaluation Association Annual Conference CHILD IN a bunch of stuff happens CHILD OUT the foster care system

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley counterbalanced indicators of system performance permanency through reunification, adoption, or guardianship length of stay stability of care rate of allegations/ substantiated allegations home-based services vs. out of home care positive attachments to family, friends, and neighbors use of least restrictive form of care Source: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L. (2002). Measuring Outcomes in Child Welfare. Chapel Hill: Jordan Institute for Families reentry to care Tracking Child Welfare Outcomes

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley There are three kinds of lies: Lies, Damned Lies and Statistics Abused Statistics ^

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Public Data: Putting it All Out There PROS: –greater performance accountability –community awareness and involvement, encourages public-private partnerships –ability to track improvement over time, identify areas where programmatic adjustments are needed –Region/region and region/county collaboration –transparency

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Public Data: Putting it All Out There CONS: –Potential for misuse, misinterpretation, and misrepresentation –Available to those with agendas or looking to create a sensational headline –Misunderstood data can lead to the wrong policy decisions –“Torture numbers, and they’ll confess to anything” (Gregg Easterbrook)

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 1)Compare Apples and Oranges 2)Use ‘snapshots’ of Small Samples 3)Rely on Unrepresentative Findings 4)Logically ‘flip’ Statistics 5)Falsely Assume an Association to be Causal 6)Rely on Summary Statistics Six Ways to Abuse Data (without actually lying!):

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Two doctors in San Diego, CA… Doctor #1 Doctor #2 What if Doctor #1 is a podiatrist and Doctor #2 is a cardiologist? 2/100020/1000 1) Compare Apples and Oranges Lowest mortality rate?

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Number of Crimes Period 1: 76 Period 2: 51 Period 3: 91 Period 4: 76 Crime jumped by 49%!! No change. Crime dropped by 16% Average = 73.5 Crime in San Diego, CA… 2) Data Snapshots…

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 2) Data Snapshots… A politician recently claimed that 92.3% of all the jobs lost since Obama took office were lost by women.* *

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Survey of people in San Diego, CA… 90% of respondents stated that they support using tax dollars to build a new football stadium. The implication of the above finding is that there is overwhelming support for the stadium… But what if you were then told that respondents had been sampled from a list of season football ticket holders? 3) Unrepresentative Findings…

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 4) Logical “Flipping”… Headline in U-T San Diego: 60% of violent crimes are committed by men who did not graduate from high school. “Flip” 60% of male high school drop-outs commit violent crimes?

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley A study of San Diego residents makes the following claim: Adults with short hair are, on average, more than 3 inches taller than those with long hair. Finding an association between two factors does not mean that one causes the other… Hair Length Height Gender X 5) False Causality…

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 6) Reliance on Summary Statistics The average human has one breast and one testicle.* Disaggregation is required for this analysis to be really useful. * ~Des McHale

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Disaggregation One of the most powerful ways to work with data… Disaggregation involves dismantling or separating out groups within a population to better understand the dynamics Useful for identifying critical issues that were previously undetected Aggregate Permanency Outcomes Race/Ethnicity Age County Placement Type

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley In San Diego County, entry rates are highest for infants. Entries are higher for African American children for almost all ages.

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Response to Data Maltreatment? must have the will to weather the storm(s)… continued efforts to frame the data, educate interested media, policymakers, and others –what do these findings mean? –how can these data be used to gain insight into where improvements are needed? agencies must be proactive in discussing both the “good” and the “bad” (be first, but be right). –be transparent –if not playing offense…playing defense

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Questions? Daniel Webster Needell, B., Webster, D., Armijo, M., Lee, S., Dawson, W., Magruder, J., Exel, M., Cuccaro- Alamin, S., Putnam-Hornstein, E., Williams, D., Simon, V., Hamilton, D., Lou, C., Peng, C., Moore, M., King, B., Henry, C., & Nuttbrock, A. (2012). Child Welfare Services Reports for California. Retrieved 8/22/2012, from University of California at Berkeley Center for Social Services Research website. URL: