A Cautionary Tale about Hazard Ratios Communicating Technical Findings to Lay Audiences Shannon Williams, PhD AEA Conference | Atlanta, GA October 26, 2016 - Story about this study and my role leading an evaluation that I ultimately don’t understand

History of Past Program Evaluations CPS look-up study conducted annually since 2001 Consistently “positive” results for CPS outcomes comparing pre and post program referrals Half of B&B parents had a prior history of CPS referral (including as the victim) Look-up showed only a small proportion of the B&B parent cohort had any CPS contact one year after their case closure

Comparing CPS Referrals Pre/Post Past Evaluations| Comparing CPS Referrals Pre/Post n = 2,512 Pre Program Post Program Substantiated Inconclusive Unfounded/UK Receiving 8+ Hours of HV Programming (2010-2014) No history

New Evaluation Design | Requirement from Major Funder Corporation for National and Community Service has funded organization nearly $1million/year for 90 B&B members, since 2000 New evaluation requirements for grantees receiving $500k or more to demonstrate higher level of evidence Programmatically not feasible to conduct RCT, quasi-experimental the highest level of evidence Uses national service to staff home visitors Corporation for National and Community Service has funded nearly $1million/year for 90 B&B members, since 2000. AmeriCorps members commit 1-4 years of service as Home Visitors and Family Resource Center Aides

Screen Shot 2016-10-26 at 7.54.16 AM.png The findings of this study provide strong evidence that Birth & Beyond home visitation supports Sacramento County parents and their families by reducing the risk of child welfare referrals after participating in the program.

Challenges | Communicating Results from New Evaluation Educate evaluation audiences about predictive studies Describe findings to lay-audiences in non- technical language Balance significance of findings, with study caveats Facilitate process with only a superficial understanding of statistical analysis

Summarizing Significant Findings All percentage differences in this table refer to hazards ratios, which give the probability difference of recidivism at t time X, provided that the individual has not already recidivated.

Client Eager to Communicate Results to Secondary Audiences Program Staff & Board of Directors Additional Funders Nurturing Parenting Program (NPP) Developers/Users County Child Welfare Agency In addition: Parents with 25-36 hours of HV were 57% less likely to have a ANY TYPE OF REFERRAL over a four-year period than non-HV parents.

Likelihood to recidivate generally increases over time. Shows likelihood to have a substantiated case at each period in time (if not previously recidivated). Remember that this is a predictive model, rather than what actually happened (see previous study)

Developed Language for Client to Communicate Evaluation

Great News Gone Awry… Study generated excitement and enthusiasm in client organization and in social service community Used evaluation findings as basis to secure additional funding Language intended to make study transparent to general public… … begun to raise questions with technical reviewers from the secondary audiences Tendency for audience to make straightforward, arithmetic calculations Language matters

Example of ‘Explanation Drift’ Parents who had previously come to the attention of child welfare services…. ORIGINAL | …and had 25-34 hours of B&B home visitation lessons, were 173% less likely to have a future substantiated allegation of child abuse than those who had not received B&B services. CORRECTED | …but did not receive B&B home visitation services, had 173% greater odds of having a future substantiated allegation of child abuse than those who received 25-34 hours of B&B services.

Lessons Learned | Transitioning from observational to predictive evaluations Make connections for client early-on regarding study limitations Simplify communication by reporting only most significant findings and numbers Budget time and resources for communicating findings beyond standard report for primary audience Use analyst to present findings