1. DC Home visiting Implementation and impact evaluation
Deborah F. Perry, PhD
Georgetown University
Sarah Kaye, PhD
Kaye Implementation & Evaluation, LLC
November 27, 2012

2. Importance of rigorous design
Federal funding is contingent on rigorous implementation and outcomes evaluation
Evidence-based home visiting (HV) models
Implemented with fidelity
Determine impact of HV on children and families served
All local agencies receiving funding for expanded HV must participate in the evaluation study
Also could have to participate in the national evaluation study if DC is selected

3. Evaluation design Implementation Strategies Implementation Outcomes
Intervention Outcomes
Implementation Study
Impact Study
Monitoring & Feedback

4. Implementation Study Implementation Strategies Implementation Outcomes
Intervention Outcomes
Implementation Study
Impact Study
Monitoring & Feedback

5. Importance of STUDYING implementation
Effective interventions cannot reach the majority of children and families with scaling up.
Effective interventions, when implemented poorly, do not provide the same benefit for children and families (Durlak & DuPre, 2008).
Implementing new programs and practices requires change (and change is hard).
Practice can “drift” back to business-as usual without support and reinforcement (Breitenstein et al., 2010; McGrew et al., 1994).
Children and families cannot benefit from services they don’t receive.

6. IMPLEMENTATION Questions
Are Healthy Families America (HFA) services delivered with fidelity to the HFA model?
Are providers operating with adherence to the 12 HFA critical elements?
Are professional development activities improving staff capacity to delivery high quality HV services?
How are implementation strategies related to fidelity of HFA implementation?
How could implementation strategies be improved to enhance implementation outcomes?

7. Implementation Study Implementation Study Impact Study
Implementation Strategies
Implementation Outcomes
Intervention Outcomes
Implementation Study
Impact Study
Implementation Strategies Review Instrument
Readiness: qualitative interviews and focus groups
Fidelity: self report and independent raters checklists
Adherence to HFA’s critical elements: HFA Self-Assessment Checklist
Capacity: HV Capacity Assessment
Monitoring & Feedback
based on the National Implementation Research Network’s implementation drivers
Readiness: agencies implementing HFA
Fidelity and Capacity measures to be developed

8. Impact Study Implementation Study Impact Study Monitoring & Feedback
Implementation Strategies
Implementation Outcomes
Intervention Outcomes
Implementation Study
Impact Study
Monitoring & Feedback

9. Impact study components
Random assignment of agencies awarded funding for HFA expansion
Cohort 1: training and implementation right away
Cohort 2: baseline data collection for 6 months, then training and implementation
Outcomes aligned with federal MIECHV benchmarks and DC Early Success Framework
Improvements in early learning and development
 Improvements in early detection of developmental delays
 Improvements in family engagement and service agencies support
 Improvements in children‘s social, emotional, mental, and physical health

10. Outcomes
What are the differences in outcomes when community agencies implement HFA versus a non-evidence-based home visitation model?
Maternal depressive symptoms
Maternal social support
Maternal efficacy
Breastfeeding initiation and duration
Sleep routines (i.e., sleep location and positioning)
These are examples…

11. Monitoring & Feedback Implementation Study Impact Study
Implementation Strategies
Implementation Outcomes
Intervention Outcomes
Implementation Study
Impact Study
Monitoring & Feedback

12. RE-AIM Evaluation Framework
Construct
Data Source R
Reach of evidence-based home visiting programs into the target population of high risk families with young children
Centralized intake data system E
Effectiveness of interventions in achieving desired outcomes
Impact study A
Adoption of evidence-based home visiting programs by provider agencies I
Implementation quality and consistency
Implementation study M
Maintenance of intervention effects in individuals and settings over time
DOH MIS
The RE-AIM evaluation framework is designed to assess the public impact of evidence-based interventions, taking into account implementation issues as well as impact on child/family outcomes. This framework will guide the selection of indicators that will be monitored and shared with stakeholders over the course of the project.

13. Empowerment evaluation
The Community Advisory Board to the evaluation will assist in:
Identification of relevant indicators,
Identification of stakeholders who could benefit from timely data about the status of implementation,
Dissemination of data to pertinent stakeholders.
Implementation and intervention indicators will be shared with DOH, HV Council, provider agency leadership, and other relevant stakeholders through defined feedback loops.