1. Building Capacity to Conduct Scientifically and Culturally Rigorous Evaluations in Tribal Communities through the Tribal Home Visiting Evaluation Institute
The Way Forward: ACF Research with American Indians and Alaska Natives
April 2014
Erin Geary, Kate Lyon & Julie Morales
James Bell Associates, Inc.

2. Port Gamble S’Klallam Tribe
Overview
What is the Tribal Home Visiting Evaluation Institute (TEI)?
What is the Tribal Maternal, Infant, and Early Childhood Home Visiting program?
How does TEI build capacity in Tribal communities?
How does TEI’s approach align with the CB’s Roadmap for Collaborative and Effective Evaluation in Tribal Communities?
Questions
Port Gamble S’Klallam Tribe

3. TEI consists of: James Bell Associates, Inc.
Johns Hopkins Bloomberg School of Public Health, Center for American Indian Health
University of Colorado School of Public Health, Centers for American Indian and Alaska Native Mental Health
MDRC

4. TEI Federal partners Office of Planning, Research and Evaluation
Administration for Children and Families, Office of the Assistant Secretary for Early Childhood Development
Office of Child Care

5. Tribal Maternal, Infant and Early Childhood Home Visiting Program
Administered by ACF in collaboration with HRSA
Funded through ACA, MIECHV includes 3% set aside for tribal program
25 cooperative agreements awarded to Tribes, Tribal consortia, Tribal organizations and urban Indian organizations
5-year grants that begin with a needs assessment and a planning year
3 cohorts: 13 in FY 2010, 6 awarded in FY 2011, 6 awarded in FY 2012
Grantees must report to ACF on performance measures, conduct rigorous evaluation and engage in continuous quality improvement activities
Confederated Tribes of Siletz Indians

6. Tribal MIECHV program goals
Supporting the development of healthy, happy, and successful AIAN children and families
Implementing high-quality, culturally-relevant, evidence-based home visiting programs in AIAN communities
Expanding the evidence base around home visiting interventions for Native populations
Supporting and strengthening cooperation and coordination and promoting linkages among various early childhood programs, resulting in coordinated, comprehensive early childhood systems
Wavoka Monument
Yerington Paiute Tribe

7. TEI provides guidance on:
Tracking and reporting on benchmarks (i.e., performance measures)
Rigorous evaluation
Data systems
Continuous Quality Improvement
Ethical dissemination and knowledge translation
Taos Pueblo

8. Three Data Requirements
Benchmarks - Demonstrate Performance Improvement Over Time
Legislatively mandated; grantees develop their own performance measures and indicators; no client level data reported; 36 constructs
TEI helps grantees develop a benchmark plan, prepare for and conduct data collection, and analyze and report data to ACF; data systems and data management are also TA topics
Rigorous Evaluation- Answer a Focused Evaluation Question Using Rigorous Research Methods
Grantees select question using CBPR approach; use rigorous design to answer question; focus on program impact, adaptations, or implementation strategy
TEI helps grantees develop an evaluation question and design using the PICO approach, provide TA on developing IRB protocol, analysis and ethical dissemination of results
Continuous Quality Improvement – Use Data to Identify and Test Changes to Improve Program
Grantees select CQI topic like screening rates, family retention, breastfeeding initiation; use benchmark or other program data in a collaborative process to make data-driven improvements to program
TEI assists grantees in preparing for and conducting Plan-Do-Study-Act cycles

9. Key difference is how the data are used and the comparisons made
Grantees make decisions about the data to collect based on community priorities
Key difference is how the data are used and the comparisons made
Locally defined Benchmarks
Local Rigorous Evaluation
Continuous Quality Improvement
We’ve learned that we need to be really clear about explaining the distinctions between the 3. Realized that everyone uses terms like evaluation and research a little differently. To some people, when they hear evaluation, they are thinking more about something like the benchmark component. Or when we describe the rigorous evaluation component, it sounds more like research to them.
The same data can be used for multiple purposes
TEI provides TA on all 3 activities

10. TEI Collaborative Process
Federal Home Visiting Team
DOHVE (State MIECHV TA Provider)
Consultants
TEI
Federal Project Officers
Tribal Early Childhood Research Center
Model Developers
VisTA

11. Grantee Home Visiting Team Tribal Evaluation Institute
A relational technical assistance process
Federal and TA Input
Grantee Home Visiting Team
Tribal Evaluation Institute
Community Input
Federal Team
Community Advisory Board
Evaluator(s)
TEI Liaisons
Tribal Home VisTA: Programmatic TA
Tribal government or leadership
Bi-directional flow of information
Tribal Early Childhood Research Center
Elders or cultural advisors
Project Director, Coordinator and other staff

12. Grantee Home Visiting Team Tribal Evaluation Institute
A relational technical assistance process
Community Input
Federal and TA Input
Grantee Home Visiting Team
Tribal Evaluation Institute
Community Advisory Board
Federal Team
Evaluator(s)
TEI Liaisons
Tribal Home VisTA: Programmatic TA
Tribal government or leadership
Tribal Early Childhood Research Center
Elders or cultural advisors
Project Director, Coordinator and other staff

13. Grantee Home Visiting Team Tribal Evaluation Institute
Conference calls
Site visits
TA modalities
Grantee Home Visiting Team
Tribal Evaluation Institute
Grantee
meetings
Toolkits
Webinars
Feedback
on plans

14. Developing and implementing a performance monitoring system (benchmarks)
Benchmark Plan Draft
Send to TEI Liaison
TEI develops Feedback
Receive written feedback
Review Call
Benchmark
Consult with community
Grantees develop and operationalize performance measures that correspond to 36 Federally mandated benchmark constructs
Grantees select appropriate measures that correspond to community priorities and provide useful data for continuous quality improvement
Iterative process of collaborating on benchmark plan as a program team, engaging with community, and working with TEI
Bullet 1: Provide example of different ways to operationalize breastfeeding
Bullet 2: Provide example of selecting immunizations as a measure of well-child visits (or some other example)

15. Rigorous evaluation
Grantees develop an evaluation question that is important to the community and will contribute to the knowledge base
Grantees select an evaluation design that is both rigorous and acceptable to the community
Grantees are encouraged to narrow the focus of the evaluations due to resource limitations:
Measure small set of outcomes
Examine component of HV program
Focus on implementation strategy (e.g., recruitment, retention)
Evaluate enhancement or adaptation

16. Increased adherence to prenatal care Improved parent-child interaction
opulation I
ntervention C
omparison
Used to develop an evaluation question that connects Population, Intervention, Comparison and Outcomes
Facilitate PICO discussion during initial site visit to help define home visiting intervention initially
Return to PICO later for evaluation plan
Give example of a PICO question from a grantee: “Do families who receive home visiting demonstrate greater knowledge of child development than families who receive services as usual?”
We use the PICO approach because it’s a way of inserting the design into the evaluation question. This allows us to talk about how to establish causality and attribute outcomes to the intervention. PICO is a tool to help us develop capacity in our communities to understand research design and what different designs can and can’t tell us about the impact of a program. Something about how we start there, but then it’s usually a lengthy negotiated process to settle on a design that is the most rigorous possible but also acceptable to the community. O
Increased adherence to prenatal care
utcomes
Improved parent-child interaction

17. Traditional Ways of Knowing
Experimental Design
Quasi-Experimental Design
Pre-Post Design
Case Studies
Opinions, Ideas
Life Experience
Community Values
What’s the purpose of this slide?
Acknowledge value of traditional knowledge and indigenous ways of learning. Acknowledge that defining rigor is a contested process. Aim to blend these two approaches – Western Scientific and Indigenous ways of knowing. Our job is to build capacity to understand rigor from the Western Scientific perspective and make informed decisions about which design will provide the desired information and be appropriate. Negotiated process.
Traditional Ways of Knowing
Cultural Norms

18. Using data to improve programs (CQI)
Grantees receive support to implement Plan-Do-Study-Act cycles to improve implementation fidelity/quality and outcomes
PDSA is inclusive, transparent process that involves entire program team
Use existing benchmark, evaluation, or implementation data
Emphasis on discovery, learning and testing new approaches
Evaluation is often regarded as a requirement rather than as a tool for addressing local questions and priorities and providing information of local use and value. Evaluation information has often been filed away without an attempt to share the information or use it to improve program services.
It is critical to shift from evaluation-as-judgment to evaluation-as-learning and to show that evaluation can be a tool that improves programs and finds better ways to serve Tribal children and families.

19. A few hot TA topics
Identifying ethical and meaningful comparisons for rigorous evaluation
Evaluating cultural adaptations
Identifying measures of cultural knowledge, identity, and practices & community connectedness
Supporting intensive data collection efforts in Tribal communities
Preparing home visitors to collect data in the home
Addressing concerns around collecting data in small, isolated, close-knit communities.
Identifying appropriate measures of parenting
Lack of parenting measures that have been validated for Tribal communities
Need for strengths-based measures that inform practice
Use of observational measures and videotaping are often a cultural mismatch
#2 - Data are collected by home visitors as opposed to evaluators or dedicated data collectors and data collection is integrated into service provision.

20. Relationship building – what’s worked
Early site visits facilitate early relationship building
Resources available for invaluable face-to-face time
Intensive benchmark TA process builds foundation for evaluation TA
Continuity of relationships is key
Continuity helps TEI build understanding of grantees
Trust is built by adding value
Choctaw Nation of Oklahoma

21. Knowledge and skill building – what’s worked
A phased approach to developing benchmark and evaluation plans is more efficient, easier to
digest
Be strategic about what TA to provide at individual versus universal level
Peer learning opportunities
Provide concrete examples and help translate them into unique grantee settings
Modify approach based on grantee capacity and needs
Target audience for TA extends well beyond evaluators
Successive cohorts allow us room to improve
Lake County Tribal Health Consortium

22. Support for community engagement in determining evaluation priorities
TEI supports community engagement by:
Encouraging participation from a diverse group of program and evaluation team members on planning calls
Supporting changes to planning timelines to incorporate feedback/input
Emphasizing requirement for/necessity of community engagement on site visits, webinars, phone calls, plan reviews
Asking critical questions (have you presented this design to your advisory committee? How do you think your home visitors will feel about collecting these data? Etc.)
Sharing experiences across grantees/cohorts (How some communities have reacted to certain observational measures, etc.)
Better understanding the communities through face to face interaction with Tribal leadership, community partners, and families on site visits (These partners are often included in initial “PICO” exercise)
Emphasize meaningful community input in all evaluation phases
Example: Inter-tribal Council of Michigan
Evaluation should incorporate meaningful community input in all phases, including:
Determination of key questions (What do you want to know about the program, intervention, or service being provided?)
Design of evaluation plan (How is information gathered to answer the key questions? How burdensome is it to participate?)
Selection of appropriate measures (What tools, surveys, interview questions will be used?)
Interpretation of findings (What does the information that has been gathered mean? Does it answer the key questions?)
Dissemination – returning the knowledge gained to the community (How is the information shared? What publications, journals, newspapers, community forums, and so forth will disseminate the information?)
Grants to Tribal nations and organizations that require evaluation provide opportunities for Tribes to exercise their sovereignty by identifying evaluation questions, engaging in evaluation design, and establishing indigenous evaluation protocols that can provide information to inform improvements in child welfare in their communities.
Some highlights of ITCMI community engagement
Very thorough needs assessment that informed evaluation priority
Building literacy intervention (focus of evaluation)with team of evaluators (both internal and external), curriculum specialists, community partners, program staff, and advisory members
Monthly webinars with 7 communities
Frequent in-person meetings with representatives from each of the 7 communities
“De-centralized” model that leaves many staffing and implementation decisions up to the local communities with oversight, close communication, and training provided by ITCMI
TA with ITCMI-
TA liaison participates in intervention planning calls to answer evaluation questions
Internal evaluator provides initial document drafts, TA provider and other team members provide feedback and TA liaison facilitates planning calls
Site visit included stops at several of the 7 communities

23. Evaluations anchored in community and cultural context
Federal guidance allows for individualized as opposed to prescriptive approach – respects diversity
Flexibility to define evaluation questions facilitates consideration of examinations of program model fit:
Lead many grantees to examine cultural enhancements to home visiting models
Lead some grantees to develop theories of change regarding native parenting practices
Flexibility to define performance measures results in benchmark plans that reflect community context
Example: In remote Native villages where subsistence patterns of living are the traditional way of life, changes in household income have little meaning; some grantees selected adapted measures of economic security
Individualization (as opposed to prescriptive approach) address Tribe-specific history and contexts, beliefs, protocols, and program needs.
Kodiak Area Native Association

24. Evaluations that blend scientific and cultural rigor
Negotiating the balance:
Recognizing the value of indigenous ways of knowing and methods of inquiry
Application of appropriate and rigorous research methods
Accommodating both will result in evaluation findings that will be meaningful to grantee community and to other tribal communities
Example 1: Southcentral Foundation’s use of historical comparison group and propensity score matching to evaluate impact of model, plus qualitative study of cultural fit of model.
Example 2: Pueblo of San Felipe’s evaluation of a newly developed cultural parenting curriculum evaluated using an internal comparison and qualitative methods
Cultural rigor might include engagement of the entire community in the evaluation process or selected tribal experts/elders on specific cultural practices. Rigorous evaluation in Tribal communities means that sound scientific methods need to be employed but that they must also be grounded in sound cultural methods.
Ongoing negotiations of the under-standing of indigenous ways of knowing and the concepts of scientific rigor validate both Native and Western perspectives
ACF and TEI’s community-engaged approach results in evaluations that are more likely to have results that are meaningful and can be translated into practice.
Importing methodology without regard to cultural context will result in non-rigorous evaluation.

25. Impacts of capacity building
Tribal communities are better prepared to oversee and conduct evaluation and consume evaluation information
Improved services and outcomes through the integration of data into decision making
Communities are invested in and take ownership of their evaluation plans and benchmark data collection
Strengthen the capacity and authority of communities to oversee evaluation
Confederated Salish and Kootenai Tribes

26. Questions?
Aleta Meyer, PhD, OPRE, Contracting Office Representative,
Moushumi Beltangady, MSW, MPP, ACF, Program Manager,
Kate Lyon, MA, JBA, TEI Project Director & TA Liaison
Julie Morales, PhD, JBA, TEI TA Liaison,
Erin Geary, MSW, JBA, TEI TA Liaison,
This product was created by JBA, Inc., under Contract No. HHSP WC, funded by the Office of Planning Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. The content of this product does not necessarily reflect the official views of the Office of Planning Research and Evaluation.

27. For more information on TEI contact: Nicole Denmark Kate Lyon
The Tribal Home Visiting Evaluation Institute (TEI) is funded by the Office of Planning, Research and Evaluation, Administration for Children and Families, Department of Health and Human Services under contract number HHSP WC.
TEI is funded to provide technical assistance to Tribal Home Visiting grantees on rigorous evaluation, performance measurement, continuous quality improvement, data systems, and ethical dissemination and translation of evaluation findings.
TEI1 was awarded to MDRC; James Bell Associates, Inc.; Johns Hopkins Bloomberg School of Public Health, Center for American Indian Health, and University of Colorado School of Public Health, Centers for American Indian and Alaska Native Health.
For more information on TEI contact:
Nicole Denmark Kate Lyon
Federal Project Officer Project Director
Office of Planning Research and Evaluation James Bell Associates, Inc.
The Tribal Evaluation Institute is funded by the Office of Planning, research and Evaluation within the Administration for Children and Families. TEI was awarded to James Bell Associates in partnership with the University of Colorado’s Centers for American Indian and Alaska Native Health and Michigan Public Health Institute. For more information, contact the individuals on this slide.