Evaluating Your Home Visiting Program Kate Lyon, MA James Bell Associates Tribal Home Visiting Evaluation Institute Cohort 3 Kickoff Meeting December 2013

program thriving children community family strong communities Our objectives: thriving children community program healthy families strong communities family

Federal Guidance The goals of evaluating home visiting in your communities are to learn how well it works in YOUR community and to add to body of evidence about effective interventions in AIAN communities Involve community in the evaluation Must conduct rigorous evaluations Flexibility to focus on targeted elements related to unique goals Build knowledge for implementing evidence based HV services in AIAN communities

homvee.acf.hhs.gov HomVEE Which models show effect in each outcome domain? Summaries across models. Information on the studies included in review.

Effects Shown in Research Which models show effect in reducing child maltreatment?

Study Search What studies have been done on Parents as Teachers? 29 moderate or high quality studies reviewed. Complete citations available.

Tribal HomVEE Report Identified studies of home visiting programs implemented in tribal communities for review Of these, none met the criteria for evidence of effectiveness RCTs experienced too much sample attrition; QEDs could not determine baseline equivalence Valuable information about implementation lessons learned http://homvee.acf.hhs.gov/Tribal_Report_2012.pdf

Contributing to the Knowledge Base Evaluation allows you to determine the impact of the intervention in your community Select an evaluation design that allows you to establish causality Focus on outcomes or implementation supports and strategies

Increased adherence to prenatal care Improved birth outcomes opulation I ntervention C omparison O Increased adherence to prenatal care utcomes Improved birth outcomes

PICO Example Do urban American Indian children ages 2-5, living below the poverty level (P) whose families receive Parents as Teachers home visitation services (I) demonstrate greater school readiness (O) compared to children whose families receive usual services (C)? The intervention is developed/selected because it links the target population’s needs to the desired outcomes through a theory of change.

PICO PICO exercise during site visits to discuss community priorities PICO to build evaluation question and frame evaluation plan

Implementation Planning C P I Needs Assessment Model Selection Implementation Planning (benchmarks) Evaluation Question O The evaluation question is linked to and informed by the needs assessment and model selection.

What Is a Rigorous Evaluation? Rigor means using an appropriate research design to answer your evaluation question(s) Rigor also means using systematic methods-- keeping as many things as consistent and standard across all aspects of the evaluation as possible Source of info on methodological rigor: DOHVE Evaluation Technical Assistance Brief, May 2011

Traditional ways of knowing Levels of Rigor Experimental Design Quasi-Experimental Design Pre-Post Design Case Studies Opinions, Ideas Life Experience Community Values Traditional ways of knowing Cultural Norms

Balancing Rigor & Feasibility Need to balance goal of rigor with resources available for evaluation Evaluation can be targeted to specific outcomes or components of the intervention Link evaluation questions to benchmarks

Year 1 Evaluation Activities Find the evaluator who is right for your grant Review HomVEE findings and HV research literature Conduct a high quality needs assessment Select a model that will address the priority needs Start thinking about your evaluation question Budget for a rigorous evaluation Evaluators can contribute to the needs assessment, inform & engage community, review literature on HV

Questions?

The Tribal Evaluation Institute is funded by the Office of Planning, research and Evaluation within the Administration for Children and Families. TEI was awarded to James Bell Associates in partnership with the University of Colorado’s Centers for American Indian and Alaska Native Health and Michigan Public Health Institute. For more information, contact the individuals on this slide. The Tribal Home Visiting Evaluation Institute (TEI) is funded by the Office of Planning, Research and Evaluation, Administration for Children and Families, Department of Health and Human Services under contract number HHSP23320095644WC. TEI is funded to provide technical assistance to Tribal Home Visiting grantees on rigorous evaluation, performance measurement, continuous quality improvement, data systems, and ethical dissemination and translation of evaluation findings. TEI1 was awarded to MDRC; James Bell Associates, Inc.; Johns Hopkins Bloomberg School of Public Health, Center for American Indian Health, and University of Colorado School of Public Health, Centers for American Indian and Alaska Native Health. For more information on TEI contact: Nicole Denmark Kate Lyon Federal Project Officer Project Director Office of Planning Research and Evaluation James Bell Associates, Inc. nicole.denmark@acf.hhs.gov lyon@jbassoc.com