1. Tribal Home Visiting Evaluation Institute
Accessing and Collecting Data for Effective Tribal Services: Tribal MIECHV Grantees’ Experiences with Performance Measurement Data ACF Tribal Grantee Meeting 2014
Erin Geary, MSW
Tribal Home Visiting Evaluation Institute
James Bell Associates

2. A few questions… What data needs are you noticing in your communities?
How are you using data currently?
Grant reports?
Program improvement?
Showcasing your program?
Collaborating with other programs?
What strategies/technical assistance has been helpful in accessing and collecting data?

3. Port Gamble S’Klallam Tribe
Overview
Description of the Tribal Home Visiting Evaluation Institute
Overview of the Tribal Maternal, Infant, and Early Childhood Home Visiting program
What can we learn from Tribal MIECHV Grantees about accessing and collecting data?
What has worked
What has been challenging
How are grantees using data
How are grantees engaging their
communities
Implications for other programs that serve
Tribal communities
Port Gamble S’Klallam Tribe

4. Tribal Home Visiting Evaluation Institute (TEI)
Provides Technical Assistance on:
Tracking and reporting on benchmarks (i.e., performance measures)
Rigorous evaluation
Data systems
Continuous Quality Improvement
Ethical dissemination and knowledge translation
Taos Pueblo

5. TEI Consists of: James Bell Associates, Inc.
Johns Hopkins Bloomberg School of Public Health, Center for American Indian Health
University of Colorado School of Public Health, Centers for American Indian and Alaska Native Mental Health
MDRC

6. TEI Federal Partners Office of Planning, Research and Evaluation
Administration for Children and Families, Office of the Assistant Secretary for Early Childhood Development
Office of Child Care

7. Tribal Maternal, Infant and Early Childhood Home Visiting Program
Administered by ACF in cooperation with HRSA
Funded through Affordable Care Act (ACA), MIECHV includes 3% set aside for tribal program
25 cooperative agreements awarded to Tribes, Tribal consortia, Tribal organizations and urban Indian organizations
5-year grants that begin with a needs assessment and a planning year
3 cohorts: 13 in FY 2010, 6 awarded in FY 2011, 6 awarded in FY 2012
Grantees must report to ACF on performance measures, conduct rigorous evaluation and engage in continuous quality improvement activities

8. Tribal MIECHV Grantees
Cohort One (FY 2010)
Choctaw Nation
Fairbanks Native Association
Kodiak Island Native Association
Lake County Tribal Health Consortium
Native American Professional Parent Resources
Native American Community Health Center
Northern Arapaho Tribe
Port Gamble S’klallam Tribe
San Felipe Pueblo
Southcentral Foundation
South Puget Intertribal Planning Agency
White Earth Band of Chippewa
Yerington Paiute Tribe

9. Tribal MIECHV Grantees
Cohort Two (FY 2011)
Eastern Band of Cherokee
Native American Health Center
Riverside San Bernadino County Indian Health
The Confederated Salish and Kootenai Tribes
Taos Pueblo
United Indians of all Tribes
Cohort Three (FY 2012)
Cherokee Nation
Choctaw Nation
Confederated Tribes of Siletz Indians
Intertribal Council of Michigan
Red Cliff Band of Ojibwe
Yellowhawk Tribal Health Clinic

10. Tribal MIECHV Program Goals
Supporting the development of healthy, happy, and successful AIAN children and families
Implementing high-quality, culturally-relevant, evidence-based home visiting programs in AIAN communities
Expanding the evidence base around home visiting interventions for Native populations
Supporting and strengthening cooperation and coordination and promoting linkages among various early childhood programs, resulting in coordinated, comprehensive early childhood systems

11. Context
“Evaluation activities have generally been imposed on Native communities by funding agencies that view evaluation from the dominant cultural paradigm. These approaches often failed to recognize the sovereignty of Tribes and to take advantage of long traditions of successful evaluation strategies that draw on indigenous practice.”
-A Roadmap for Collaborative and Effective Evaluation in Tribal Communities (2013)
Our TA doesn’t exist in a vacuum but needs to recognize past experiences with research/evaluation

12. Grantee Home Visiting Team Tribal Evaluation Institute
A relational technical assistance process
Federal and TA Input
Grantee Home Visiting Team
Tribal Evaluation Institute
Community Input
Federal Team
Community Advisory Board
Evaluator(s)
TEI Liaisons
Tribal Home VisTA: Programmatic TA
Tribal government or leadership
Tribal Early Childhood Research Center
Elders or cultural advisors
Project Director, Coordinator and other staff

13. Grantee Home Visiting Team Tribal Evaluation Institute
A relational technical assistance process
Community Input
Federal and TA Input
Grantee Home Visiting Team
Tribal Evaluation Institute
Community Advisory Board
Federal Team
Evaluator(s)
TEI Liaisons
Tribal Home VisTA: Programmatic TA
Tribal government or leadership
Tribal Early Childhood Research Center
Elders or cultural advisors
Project Director, Coordinator and other staff

14. Grantee Home Visiting Team Tribal Evaluation Institute
Conference calls
Site visits
TA modalities
Grantee Home Visiting Team
Tribal Evaluation Institute
Grantee
meetings
Toolkits
Webinars
Feedback
on plans

15. Tribal MIECHV Data Collection

16. Data Requirements: Needs Assessment
Needs Assessment – Understand Community Health and Well-being
Key part of planning year
Grantees gather quantitative and qualitative data on key community health and well-being indicators
Needs Assessment findings inform home visiting model selection
Grantees use the process as a community engagement effort

17. Data Requirements: Performance Benchmarks
Benchmarks - Demonstrate Performance Improvement Over Time
Legislatively mandated
Grantees develop their own performance measures and indicators
No client level data reported
37 constructs
TEI helps grantees develop a benchmark plan, prepare for and conduct data collection, and report data to ACF; data systems and data management are also TA topics

18. Data Requirements: Rigorous Evaluation
Rigorous Evaluation- Answer a Focused Evaluation Question Using Rigorous Methods
Grantees select question using CBPR approach
Use rigorous design to answer question
Focus on program impact, adaptations, or implementation strategy
TEI helps grantees develop an evaluation question and design using the PICO approach, provide TA on developing IRB protocol, analysis and ethical dissemination of results

19. Data Requirements: Continuous Quality Improvement
Continuous Quality Improvement – Use Data to Identify and Test Changes to Improve Program
Grantees select CQI topic like screening rates, family retention, breastfeeding initiation
Use benchmark or other program data in a collaborative process to make data-driven improvements to program
TEI assist grantees in preparing for and conducting P-D-S-A cycles

20. Key difference is how the data are used and the comparisons made
Grantees make decisions about the data to collect based on community priorities
Key difference is how the data are used and the comparisons made
Locally defined Benchmarks
Local Rigorous Evaluation
Continuous Quality Improvement
We’ve learned that we need to be really clear about explaining the distinctions between the 3. Realized that everyone uses terms like evaluation and research a little differently. To some people, when they hear evaluation, they are thinking more about something like the benchmark component. Or when we describe the rigorous evaluation component, it sounds more like research to them.
The same data can be used for multiple purposes
TEI provides TA on all 3 activities

21. How are Grantees Accessing Data?
Families (self-report)
Demographic information
Service use (i.e. when was your last prenatal exam?)
Screenings (depression, substance abuse, development, domestic violence)
Partnerships (administrative records)
Head Start
Child Care
Health Clinics
Child Welfare Programs
State Partners
Home Visitors (documentation and observation)
Visits completed/missed
Observational assessment of parents/children
Process information (referrals, information provided, etc.)

22. Developing and implementing a performance monitoring system (benchmarks)
Benchmark Plan Draft
Send to TEI Liaison
TEI develops Feedback
Receive written feedback
Review Call
Benchmark
Consult with community
Grantees develop and operationalize performance measures that correspond to 37 Federally mandated benchmark constructs
Grantees select appropriate measures that correspond to community priorities and provide useful data for continuous quality improvement
Iterative process of collaborating on benchmark plan as a program team, engaging with community, and working with TEI
Bullet 1: Provide example of different ways to operationalize breastfeeding
Bullet 2: Provide example of selecting immunizations as a measure of well-child visits (or some other example)

23. Rigorous evaluation
Grantees develop an evaluation question that is important to the community and will contribute to the knowledge base
Grantees select an evaluation design that is both rigorous and acceptable to the community
Grantees are encouraged to narrow the focus of the evaluations:
Measure small set of outcomes
Examine component of HV program
Focus on implementation strategy (e.g., recruitment, retention)
Evaluate enhancement or adaptation
Cultural rigor might include engagement of the entire community in the evaluation process. Rigorous evaluation in Tribal communities means that sound scientific methods need to be employed but that they must also be grounded in sound cultural methods.
Aleta asked us to make the point that our community-engaged approach results in evaluations that are more likely to have results that are meaningful and can be translated into practice.
questions.
Importing methodology without regard to cultural context will result in non-rigorous evaluation.
Ongoing negotiations of the under-standing of indigenous ways of knowing and the concepts of scientific rigor validate both Native and Western perspectives

24. Evaluations anchored in community and cultural context
Federal guidance allows for individualized as opposed to prescriptive approach – respects tribal sovereignty and diversity of communities
Flexibility to define evaluation question has lead many grantees to examine cultural enhancements to home visiting models
Flexibility to define performance measures results in benchmark plans that reflect community context
Individualization (as opposed to prescriptive approach) address Tribe-specific history and contexts, beliefs, protocols, and program needs.

25. Evaluation Approach
All knowledge will be generated through local evaluations
No cross-site evaluation
Evaluation questions are developed by grantees in consultation with their community to reflect local interests & priorities
Evaluation questions are informed by findings of needs assessment and connected to implementation decisions
Tribal ownership of evaluation process, data and dissemination is respected
IRB and Tribal approval is required
Iterative, connected, and circular process for program planning, development, implementation, and evaluation

26. Evaluation Approach cont.
Evaluations can be limited in size and scope
A focused question is answered with rigorous design and methods
Flexibility to focus evaluation on a component of home visiting
Evaluations will inform grantees, communities, and the field about what works in implementing home visiting in Tribal communities
Intensive technical assistance is provided to increase Tribal capacity and empowerment to conduct different types of evaluation
Don’t have large budgets, so we’re encouraging grantees to look at a subset of outcomes or examine a component of the home visiting program, such as just the prenatal component or a set of modules. Or maybe a cultural enhancement or implementation strategy, like training and supervision of HV or family recruitment and retention strategies.

27. Lessons from Tribal MIECHV Grantees
Evaluation is often regarded as a requirement rather than as a tool for addressing local questions and priorities and providing information of local use and value. Evaluation information has often been filed away without an attempt to share the information or use it to improve program services.
It is critical to shift from evaluation-as-judgment to evaluation-as-learning and to show that evaluation can be a tool that improves programs and finds better ways to serve Tribal children and families.

28. Grantees have used requirements to advance community-wide discussions about data needs
“Conversations took place with Head Start and Early Head Start to agree upon a way in which the programs could collaborate so that families were not receiving multiple screenings at the same time points.”
Grantees examined and improved program processes in their benchmark planning
“The benchmark process required us to get concrete about program processes and outcomes.”

29. Grantees tailored performance measures to community needs
“The benchmark data we do collect will be more accurate because our manner of collection fits our process and our program. This is an important way in which the individuality of each tribe and program
was honored.”
The planning process facilitated individual skill and organizational capacity building
“The process allowed team [members] to expand their personal skills as well as overall organizational capacity in research and development of the appropriate constructs for a benchmark plan... As a result this has added to their ability to serve the community and to also carry out future work.”

30. Grantees had to be thoughtful about balancing service provision and data collection needs
“Selection of data collection tools to be utilized for each benchmark construct was determined and based on the feasibility of collecting meaningful data without undue burden to the client. For example, we stayed away from lengthy or unwieldy tools even if they were validated and considered reliable.”
White Earth Nation

31. Grantees have needed to assess measures for utility and cultural relevance
“Choosing an appropriate tool was the most challenging part of developing the plan… Having a cohort one year ahead of us in the process helped us have examples of grantees using various surveys to talk to. We also read many articles about assessments and utilized the compendium of measures to find the appropriate measures.”

32. Building a performance measurement system required time, resources, and considerable technical assistance
“At times, it appeared that a better process might have been to have an electronic template, with various [drop] down choices under each benchmark area that already met appropriate language requirements and standards from which the local team could make a selection. On the one hand this would have been quicker but individuals involved might not have learned as much about all areas.”

33. Grantees utilized community advisory groups
Grantees engaged their community by drawing on information from the community needs assessment
“We repeatedly referred to needs assessment results… Community desire to focus on family assets and strengths, as well as parent involvement/engagement was a key factor in the choice to use [certain] measures.”
Grantees utilized community advisory groups
“We went through each question with the council, discussing order of questions, purpose, wording, rephrasing, and what data the council wanted us to collect.”

34. Grantees engaged community members in the review of potential measures
“The team refined the list of potential measures, and the evaluator put together a plan describing how the measures could be administered. This plan was presented to key representatives from the Task Force as part of a benchmark working group meeting. The list was further refined based on feedback, and then the plan was presented to the entire Task Force. After we received approval from the Task Force, it was submitted for approval to Federal partners.”

35. Grantees plan to use benchmark data for program monitoring and improvement
“We will meet monthly to discuss how the program is doing as reflected by the benchmark plan and create opportunities to reflect and make changes as needed...”
Grantees are beginning to translate data for use with Tribal government, partner organizations, and community groups
“This information will be valuable to the entire tribal community when looking at health and parenting issues. Eventually, the team plans to share the information with other tribal departments and the tribal leadership for future planning and program development.”

36. Takeaways for other tribal programs
Strong partnerships facilitate the ability to have strong data
Accessing and collecting helpful data relies on an understanding of:
program processes
community needs
current infrastructure
grant requirements

37. Takeaways for other tribal programs
Accessing and collecting data can be challenging
Context of mistrust
Burdensome for staff and families
Requires (sometimes costly) technology and expertise
Having access to useful, accurate data can be beneficial
Program improvement
Improved collaboration
Sustainability

38. Questions?

39. For more information on TEI contact: Nicole Denmark Kate Lyon
The Tribal Home Visiting Evaluation Institute (TEI) is funded by the Office of Planning, Research and Evaluation, Administration for Children and Families, Department of Health and Human Services under contract number HHSP WC.
TEI is funded to provide technical assistance to Tribal Home Visiting grantees on rigorous evaluation, performance measurement, continuous quality improvement, data systems, and ethical dissemination and translation of evaluation findings.
TEI1 was awarded to MDRC; James Bell Associates, Inc.; Johns Hopkins Bloomberg School of Public Health, Center for American Indian Health, and University of Colorado School of Public Health, Centers for American Indian and Alaska Native Health.
For more information on TEI contact:
Nicole Denmark Kate Lyon
Federal Project Officer Project Director
Office of Planning Research and Evaluation James Bell Associates, Inc.
The Tribal Evaluation Institute is funded by the Office of Planning, research and Evaluation within the Administration for Children and Families. TEI was awarded to James Bell Associates in partnership with the University of Colorado’s Centers for American Indian and Alaska Native Health and Michigan Public Health Institute. For more information, contact the individuals on this slide.