1. A Cautionary Tale about Hazard Ratios
Communicating Technical
Findings to Lay Audiences
Shannon Williams, PhD AEA Conference | Atlanta, GA
October 26, 2016
- Story about this study and my role leading an evaluation that I ultimately don’t understand

2. History of Past Program Evaluations
CPS look-up study conducted annually since 2001
Consistently “positive” results for CPS outcomes comparing pre and post program referrals
Half of B&B parents had a prior history of CPS referral (including as the victim)
Look-up showed only a small proportion of the B&B parent cohort had any CPS contact one year after their case closure

3. Comparing CPS Referrals Pre/Post
Past Evaluations|
Comparing CPS Referrals Pre/Post
n = 2,512
Pre Program
Post Program
Substantiated
Inconclusive
Unfounded/UK
Receiving 8+ Hours of HV Programming ( )
No history

4. New Evaluation Design | Requirement from Major Funder
Corporation for National and Community Service has funded organization nearly $1million/year for 90 B&B members, since 2000
New evaluation requirements for grantees receiving $500k or more to demonstrate higher level of evidence
Programmatically not feasible to conduct RCT, quasi-experimental the highest level of evidence
Uses national service to staff home visitors
Corporation for National and Community Service has funded nearly $1million/year for 90 B&B members, since 2000.
AmeriCorps members commit 1-4 years of service as Home Visitors and Family Resource Center Aides

5. Screen Shot at AM.png
The findings of this study provide strong evidence that Birth & Beyond home visitation supports Sacramento County parents and their families by reducing the risk of child welfare referrals after participating in the program.

6. Challenges | Communicating Results from New Evaluation
Educate evaluation audiences about predictive studies
Describe findings to lay-audiences in non- technical language
Balance significance of findings, with study caveats
Facilitate process with only a superficial understanding of statistical analysis

8. Summarizing Significant Findings
All percentage differences in this table refer to hazards ratios, which give the probability difference of recidivism at t time X, provided that the individual has not already recidivated.

9. Client Eager to Communicate Results to Secondary Audiences
Program Staff & Board of Directors
Additional Funders
Nurturing Parenting Program (NPP) Developers/Users
County Child Welfare Agency
In addition:
Parents with hours of HV were 57% less likely to have a ANY TYPE OF REFERRAL over a four-year period than non-HV parents.

11. Likelihood to recidivate generally increases over time.
Shows likelihood to have a substantiated case at each period in time (if not previously recidivated).
Remember that this is a predictive model, rather than what actually happened (see previous study)

12. Developed Language for Client to Communicate Evaluation

13. Great News Gone Awry…
Study generated excitement and enthusiasm in client organization and in social service community
Used evaluation findings as basis to secure additional funding
Language intended to make study transparent to general public… … begun to raise questions with technical reviewers from the secondary audiences
Tendency for audience to make straightforward, arithmetic calculations
Language matters

14. Example of ‘Explanation Drift’
Parents who had previously come to the attention of child welfare services….
ORIGINAL | …and had hours of B&B home visitation lessons, were 173% less likely to have a future substantiated allegation of child abuse than those who had not received B&B services.
CORRECTED | …but did not receive B&B home visitation services, had 173% greater odds of having a future substantiated allegation of child abuse than those who received hours of B&B services.

15. Lessons Learned | Transitioning from observational to predictive evaluations
Make connections for client early-on regarding study limitations
Simplify communication by reporting only most significant findings and numbers
Budget time and resources for communicating findings beyond standard report for primary audience
Use analyst to present findings