1. Evaluating Your Home Visiting Program
Kate Lyon, MA
James Bell Associates
Tribal Home Visiting Evaluation Institute
Cohort 3 Kickoff Meeting
December 2013

2. program thriving children community family strong communities
Our objectives:
thriving children
community
program
healthy families
strong communities
family

3. Federal Guidance
The goals of evaluating home visiting in your communities are to learn how well it works in YOUR community and to add to body of evidence about effective interventions in AIAN communities
Involve community in the evaluation
Must conduct rigorous evaluations
Flexibility to focus on targeted elements related to unique goals
Build knowledge for implementing evidence based HV services in AIAN communities

4. homvee.acf.hhs.gov HomVEE
Which models show effect in each outcome domain?
Summaries across models.
Information on the studies included in review.

5. Effects Shown in Research
Which models show effect in reducing child maltreatment?

6. Study Search
What studies have been done on Parents as Teachers?
29 moderate or high quality studies reviewed.
Complete citations available.

7. Tribal HomVEE Report
Identified studies of home visiting programs implemented in tribal communities for review
Of these, none met the criteria for evidence of effectiveness
RCTs experienced too much sample attrition; QEDs could not determine baseline equivalence
Valuable information about implementation lessons learned

8. Contributing to the Knowledge Base
Evaluation allows you to determine the impact of the intervention in your community
Select an evaluation design that allows you to establish causality
Focus on outcomes or implementation supports and strategies

9. Increased adherence to prenatal care Improved birth outcomes
opulation I
ntervention C
omparison O
Increased adherence to prenatal care
utcomes
Improved birth outcomes

10. PICO Example
Do urban American Indian children ages 2-5, living below the poverty level (P) whose families receive Parents as Teachers home visitation services (I) demonstrate greater school readiness (O) compared to children whose families receive usual services (C)?
The intervention is developed/selected because it links the target population’s needs to the desired outcomes through a theory of change.

11. PICO PICO exercise during site visits to discuss community priorities
PICO to build evaluation question and frame evaluation plan

12. Implementation PlanningC P I
Needs Assessment
Model Selection
Implementation Planning
(benchmarks)
Evaluation Question O
The evaluation question is linked to and informed by the needs assessment and model selection.

13. What Is a Rigorous Evaluation?
Rigor means using an appropriate research design to answer your evaluation question(s)
Rigor also means using systematic methods-- keeping as many things as consistent and standard across all aspects of the evaluation as possible
Source of info on methodological rigor: DOHVE Evaluation Technical Assistance Brief, May 2011

14. Traditional ways of knowing
Levels of Rigor
Experimental Design
Quasi-Experimental Design
Pre-Post Design
Case Studies
Opinions, Ideas
Life Experience
Community Values
Traditional ways of knowing
Cultural Norms

15. Balancing Rigor & Feasibility
Need to balance goal of rigor with resources available for evaluation
Evaluation can be targeted to specific outcomes or components of the intervention
Link evaluation questions to benchmarks

16. Year 1 Evaluation Activities
Find the evaluator who is right for your grant
Review HomVEE findings and HV research literature
Conduct a high quality needs assessment
Select a model that will address the priority needs
Start thinking about your evaluation question
Budget for a rigorous evaluation
Evaluators can contribute to the needs assessment, inform & engage community, review literature on HV

17. Questions?

18. The Tribal Evaluation Institute is funded by the Office of Planning, research and Evaluation within the Administration for Children and Families. TEI was awarded to James Bell Associates in partnership with the University of Colorado’s Centers for American Indian and Alaska Native Health and Michigan Public Health Institute. For more information, contact the individuals on this slide.
The Tribal Home Visiting Evaluation Institute (TEI) is funded by the Office of Planning, Research and Evaluation, Administration for Children and Families, Department of Health and Human Services under contract number HHSP WC.
TEI is funded to provide technical assistance to Tribal Home Visiting grantees on rigorous evaluation, performance measurement, continuous quality improvement, data systems, and ethical dissemination and translation of evaluation findings.
TEI1 was awarded to MDRC; James Bell Associates, Inc.; Johns Hopkins Bloomberg School of Public Health, Center for American Indian Health, and University of Colorado School of Public Health, Centers for American Indian and Alaska Native Health.
For more information on TEI contact:
Nicole Denmark Kate Lyon
Federal Project Officer Project Director
Office of Planning Research and Evaluation James Bell Associates, Inc.