The PICO Model: Involving the Community in the Evaluation Process May 2012 Kate Lyon, MA, James Bell Associates, Inc. Aleta Meyer, PhD, Office of Planning Research and Evaluation, Administration for Children and Families Julie Morales, PhD, James Bell Associates, Inc.

PICO P – The target population you plan to serve I – The intervention or program to be evaluated C – The comparison you will make to understand how well the program works in your community O – The intended outcomes you want to see achieved A framework for developing a well-built evaluation question PICO developed by Mark Testa (UNC Chapel Hill) and used extensively with the Children’s Bureau’s Permanency Innovations Initiatives to reduce long-term foster care. (find material from slides) Adapted from: Permanency Innovations Initiative Evaluation Team (2011) Logic Model and Theory of Change. Presented at The Children’s Bureau’s PII Kickoff Meeting, Washington DC, November 2010

Tribal Maternal, Infant and Early Childhood Home Visiting Program Administration for Children and Families, Office of Child Care in partnership with HRSA 5 year grants to Tribes, Tribal Organizations and Urban Indian Organizations 19 current grantees, up to 5 more Funded to implement and identify evidence-based HV programs

Evaluation Component Rigorous evaluation Lack of evidence around the effectiveness of home visiting programs in Tribal communities Community-based participatory evaluation

Tribal HV Evaluation Institute Technical assistance on rigorous evaluation, tracking and measuring benchmarks, continuous quality improvement, data systems Intensive individualized TA and universal TA Site visits with Federal team Using PICO framework Context in which TEI does PICO – during site visits to our grantees, we facilitate discussions around evaluation using PICO. The goal is to engage the entire group in thinking about what the evaluation question should be. We facilitate the discussion, but all of the content emerges from the grantee. It’s a way of allowing a participatory process in the development of the evaluation question, yet still working within a prescribed framework, as is so often the case.

The PICO Discussion Brings together program leadership and staff, stakeholders, and evaluators Collaborative discussion about community needs, program development, and evaluation PICO builds rigor into the discussion Allows participatory process within a prescribed framework

Population Who is your target population for home visiting? What are their prioritized needs? Needs assessment informs target population. From broad: what is your community like? To specific: who will you serve?

Intervention Highlights the linkages between the needs of the target population, the program, and the benefits you hope to achieve. What is the theory of change for the program(s) you have selected? What implementation supports need to be in place to have a successful program?

Comparison Walk through different types of comparisons. What is the alternative course of action that your comparison group will experience? What is the strongest contrast that is feasible? The strongest possible contrast will help to attribute any observed differences between the groups to your program.

Outcomes What are the short- and long-term outcomes you hope to achieve? Is there evidence that the program will impact those outcomes? Which outcomes are most critical for the community?

PICO Example Do urban American Indian children ages 0-5, living below the poverty level (P) whose families receive Parents as Teachers home visitation services (I) demonstrate greater school readiness (O) compared to children whose families receive usual services (C)?

PICO Example Do women who are pregnant with their first child (P) who receive Nurse Family Partnership home visiting services (I) experience better birth outcomes (O) compared to pregnant women who gave birth at the clinic before NFP was implemented (C)?

PICO Example Do home visitors (P) who receive enhanced reflective supervision (I) demonstrate greater confidence and competence in addressing difficult family issues (substance use and domestic violence) (O) as compared to home visitors who receive regular supervision (C).

For more information: Kate Lyon, TEI, James Bell Associates lyon@jbassoc.com Aleta Meyer, ACF, OPRE Aleta.Meyer@acf.hhs.gov Julie Morales, TEI, James Bell Associates morales@jbassoc.com Anne Bergan, ACF, OPRE Anne.Bergan@acf.hhs.gov

The Tribal Evaluation Institute is funded by the Office of Planning, research and Evaluation within the Administration for Children and Families. TEI was awarded to James Bell Associates in partnership with the University of Colorado’s Centers for American Indian and Alaska Native Health and Michigan Public Health Institute. For more information, contact the individuals on this slide. The Tribal Home Visiting Evaluation Institute (TEI) is funded by the Office of Planning, Research and Evaluation, Administration for Children and Families, Department of Health and Human Services under contract number HHSP23320095644WC. TEI is funded to provide technical assistance to Tribal Home Visiting grantees on rigorous evaluation, performance measurement, continuous quality improvement, data systems, and ethical dissemination and translation of evaluation findings. TEI1 was awarded to MDRC; James Bell Associates, Inc.; Johns Hopkins Bloomberg School of Public Health, Center for American Indian Health, and University of Colorado School of Public Health, Centers for American Indian and Alaska Native Health. For more information on TEI contact: Nicole Denmark Kate Lyon Federal Project Officer Project Director Office of Planning Research and Evaluation James Bell Associates, Inc. nicole.denmark@acf.hhs.gov lyon@jbassoc.com