DC Home visiting Implementation and impact evaluation Deborah F. Perry, PhD Georgetown University Sarah Kaye, PhD Kaye Implementation & Evaluation, LLC November 27, 2012

Importance of rigorous design Federal funding is contingent on rigorous implementation and outcomes evaluation Evidence-based home visiting (HV) models Implemented with fidelity Determine impact of HV on children and families served All local agencies receiving funding for expanded HV must participate in the evaluation study Also could have to participate in the national evaluation study if DC is selected

Evaluation design Implementation Strategies Implementation Outcomes Intervention Outcomes Implementation Study Impact Study Monitoring & Feedback

Implementation Study Implementation Strategies Implementation Outcomes Intervention Outcomes Implementation Study Impact Study Monitoring & Feedback

Importance of STUDYING implementation Effective interventions cannot reach the majority of children and families with scaling up. Effective interventions, when implemented poorly, do not provide the same benefit for children and families (Durlak & DuPre, 2008). Implementing new programs and practices requires change (and change is hard). Practice can “drift” back to business-as usual without support and reinforcement (Breitenstein et al., 2010; McGrew et al., 1994). Children and families cannot benefit from services they don’t receive.

IMPLEMENTATION Questions Are Healthy Families America (HFA) services delivered with fidelity to the HFA model? Are providers operating with adherence to the 12 HFA critical elements? Are professional development activities improving staff capacity to delivery high quality HV services? How are implementation strategies related to fidelity of HFA implementation? How could implementation strategies be improved to enhance implementation outcomes?

Implementation Study Implementation Study Impact Study Implementation Strategies Implementation Outcomes Intervention Outcomes Implementation Study Impact Study Implementation Strategies Review Instrument Readiness: qualitative interviews and focus groups Fidelity: self report and independent raters checklists Adherence to HFA’s critical elements: HFA Self-Assessment Checklist Capacity: HV Capacity Assessment Monitoring & Feedback based on the National Implementation Research Network’s implementation drivers Readiness: agencies implementing HFA Fidelity and Capacity measures to be developed

Impact Study Implementation Study Impact Study Monitoring & Feedback Implementation Strategies Implementation Outcomes Intervention Outcomes Implementation Study Impact Study Monitoring & Feedback

Impact study components Random assignment of agencies awarded funding for HFA expansion Cohort 1: training and implementation right away Cohort 2: baseline data collection for 6 months, then training and implementation Outcomes aligned with federal MIECHV benchmarks and DC Early Success Framework Improvements in early learning and development  Improvements in early detection of developmental delays  Improvements in family engagement and service agencies support  Improvements in children‘s social, emotional, mental, and physical health

Outcomes What are the differences in outcomes when community agencies implement HFA versus a non-evidence-based home visitation model? Maternal depressive symptoms Maternal social support Maternal efficacy Breastfeeding initiation and duration Sleep routines (i.e., sleep location and positioning) These are examples…

Monitoring & Feedback Implementation Study Impact Study Implementation Strategies Implementation Outcomes Intervention Outcomes Implementation Study Impact Study Monitoring & Feedback

RE-AIM Evaluation Framework Construct Data Source R Reach of evidence-based home visiting programs into the target population of high risk families with young children Centralized intake data system E Effectiveness of interventions in achieving desired outcomes Impact study A Adoption of evidence-based home visiting programs by provider agencies I Implementation quality and consistency Implementation study M Maintenance of intervention effects in individuals and settings over time DOH MIS The RE-AIM evaluation framework is designed to assess the public impact of evidence-based interventions, taking into account implementation issues as well as impact on child/family outcomes. This framework will guide the selection of indicators that will be monitored and shared with stakeholders over the course of the project.

Empowerment evaluation The Community Advisory Board to the evaluation will assist in: Identification of relevant indicators, Identification of stakeholders who could benefit from timely data about the status of implementation, Dissemination of data to pertinent stakeholders. Implementation and intervention indicators will be shared with DOH, HV Council, provider agency leadership, and other relevant stakeholders through defined feedback loops.