1. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Data Abuse: Now That You’ve Found Data, What Are You Going to Do With It? Barbara Needell, MSW, PhD Larry Brown, MSW Center for Social Services Research Casey Family Programs University of California at Berkeley Presented to Children's Roundtable Summit Philadelphia, PA September 23, 2010 CSSR Slides created by Emily Putnam-Hornstein

2. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley There are three kinds of lies: Lies, Damned Lies and Statistics Abused Statistics ^

3. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Six Ways to Abuse Data 1)Rank Data 2)Compare Apples and Oranges 3)Use ‘snapshots’ of Small Samples 4)Rely on Unrepresentative Findings 5)Logically ‘flip’ Statistics 6)Falsely Assume an Association to be Causal

4. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 1) Rank Data Two streets in Anytown, PA…. It’s all relative… And SOMEONE will always be ranked last (and first) Poverty Blvd $$ Ave “Jane Doe is the poorest person living on Moneybags Avenue.” “Joe Shmoe is the wealthiest person living on Poverty Blvd.”

5. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 2) Compare Apples and Oranges Two doctors in Anytown, PA… Doctor #1Doctor #2 What if the populations served by each doctor were very different? 2/1000 mortality rate 20/1000 mortality rate Doctor of the Year?

6. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Number of Crimes Period 1: 76 Period 2: 51 Period 3: 91 Period 4: 76 3) Data snapshots… Crime jumped by 49%!! No change. Crime dropped by 16% Average = 73.5 Crime in Anytown, PA…

7. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 4) Unrepresentative findings… Survey of people in Anytown, PA… 90% of respondents stated that they support using tax dollars to build a new football stadium. The implication of the above finding is that there is overwhelming support for the stadium… But what if you were then told that respondents had been sampled from a list of season football ticket holders?

8. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 5) Logical “Flipping”… Headline in The Anytown Chronicle: 60% of violent crimes are committed by men who did not graduate from high school. “Flip” 60% of male high school drop-outs commit violent crimes?

9. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 6) False Causality… A study of Anytown residents makes the following claim: Adults with short hair are, on average, more than 3 inches taller than those with long hair. Finding an association between two factors does not mean that one causes the other… Hair LengthHeight Gender X

10. Now What Are You Going To Do With It? (Con’t) Larry Brown, MSW Consultant to: Casey Family Programs PA Roundtable - Philadelphia September 23, 2010 Larry G. Brown, MSW – Improving Outcomes for Children, Families and Communities TM

11. How Do I Use My Existing Data More Productively? A Quick Review - Stock and Flow Unpack the Data Connect the Dots / Tell the Story / Look from the Balcony / Use Multiple Indicators Compare with A Purpose (and with Caution) Data Soup: When Is It Ready to Use? It Takes Time To Change (2-4 years!) Larry G. Brown, MSW – Improving Outcomes for Children, Families and Communities TM

12. “Stock and Flow” Model Courtesy of Chapin Hall Center for Children, University of Chicago Placement Rate Case Mix Placement Setting LOS Visits Stability Discharge Planning/Prep Discharge Location Educational Status Vocational Status Medical Insurance Re-Entry Rate Employment Housing Stability Permanent Connections

13. Unpack the Data AdmissionsAge At Admission Decrease of Admissions of 20% over the last five years Numbers going down, but is who’s coming through the front door changing?

14. Unpacking Data: Truancy Example Finding: High Truancy Rate Hard to think of this as Actionable. Unpack the Data -- Where’s the variation? Where do I act? Male vs. Female (52-48%) Elementary vs. Middle vs. High School (25% - 25% - 50%) Caucasian vs. Hispanic vs. Black in School: (74% - 7% - 15%) Race and Ethnicity - Truants (32% - 17% - 47%) Where do these kids live? GIS Match Your Plan to Your Evidence: –Two pronged attack: Intervention AND Prevention/Early ID –Focus where the numbers are: Communities, Neighborhoods and Parents Larry G. Brown, MSW – Improving Outcomes for Children, Families and Communities TM

15. Connect the Dots Entry Rate Length of Stay Both powerful indicators. Together, they tell even more: HighShortFront Door LowLong Front-end Safety & Discharge Planning Entry Rate LOSFocus Larry G. Brown, MSW – Improving Outcomes for Children, Families and Communities TM

16. Pushing At What It Means; Is There a Story Behind the Numbers? Entering Care: % Chng AgeApr-06Apr- 10 0-52,1511,648-23.4% 13-214,3582,277-47.8% Total8,0564902-39.2% 0-527%34% 13-2154%46%

17. Compare with a Purpose Who Are My Peers? Why am I Comparing Myself to Them? This gives me context, but tells me little about MY system. But if there IS a good idea… …STEAL SHAMELESSLY Larry G. Brown, MSW – Improving Outcomes for Children, Families and Communities TM

18. Data Soup: CPCMS, PPI, HZA Lots of new data Lots of clean up to do So, what CAN I use this for? And where do I start? Larry G. Brown, MSW – Improving Outcomes for Children, Families and Communities TM

19. For Starters: Be Active, Informed Users Examine what each item tells you: Is it a point-in-time measure? (Cases Pending on a Date Certain) –Only tells me about that day (e.g., Age of Pending Cases) –Biased toward those that take a long time –Most often cited data; not always best Is it an entry cohort measure? (Cases Filed During a Period/Term: All starting together) –Tells us how the system works as things move through a system –Highly actionable data; highlights places where intervention may make a difference (how long for new cases to reach disposition; milestones between filing and disposition) Is it a discharge cohort measure? (Cases Terminated During a Period/Term: All ending together, but not starting together) –Tells the story from the back end. –Caution in that many influences cloud what this means –Convenient, but not always easy to untagle

20. Active, Informed Users (Con’t) Repeat process with Child Welfare (HZA) data Compare like data across systems –Entry data to entry data –Point-in-time to Point-in-time –Discharge cohorts to discharge cohorts Reach agreement over which questions these data answer: –How many cases are in my workload? (point-in-time) –How many new filings are made in a quarter? (longitudinal) –How long have these cases been with us? (point-in-time) –How long does it take for cases to reach disposition? (longitudinal) Larry G. Brown, MSW – Improving Outcomes for Children, Families and Communities TM

21. Urgency vs When Is It Good Enough? Valid and Reliable Data Needed Won’t get better until it’s used 80-20 Rule Include all stakeholders in using/seeing data Kids Can’t Wait Larry G. Brown, MSW – Improving Outcomes for Children, Families and Communities TM

22. Implementation: Lots of Moving Parts. © Fixsen & Blase, 2008 Performance Assessment Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Adaptive Technical Competency Drivers Organization Drivers Leadership Graphics by Steve Goodman,2009 Implementation Drivers

23. So, Now What? Examine the inputs, processes, outputs and outcomes. What are the most important pieces of this for your initiative? Match the data to what you need in order to manage (Less IS More -- Mies van der Rohe) Have you looked at the data beneath your indicators? Lots of things hide in a summary variable. Are you connecting the dots across data indicators? What is the explanation that describes the performance you are getting from your system? Do you understand why you look the same or different from others? Different is not necessarily bad!

24. So, Now What? (Cont) Stay patient and focused; it takes time. Leadership counts. Data champions: not the most technical, but rather the most curious. (Data phobics get to play!)

25. CSSR.BERKELEY.EDU/UCB_CHILDWELFARE Barbara Needell bneedell@berkeley.edu 510 290 6334 ______________________________________________________________ Larry Brown Associates Larry.G.Brown@gmail.com (518) 370-9999 (office) (518) 421-7271 (mobile)