Demonstrating the Value of Stakeholder Input for Evaluation Design, Implementation and Dissemination Yvonne Abel, MS Abt Associates Inc. AEA Annual Conference, Anaheim, CA November 4, 2011 Evaluation of the New York City Health Bucks Farmer’s Market Incentive Program

Abt Associates | pg 2 Collaborators Abt Associates Inc.  Lauren Olsho, PhD  Deborah Klein Walker, EdD  Yvonne Abel, MS  Cheryl Austin, MPH, RD, LDN  Cristina A. Booker, MPH  Jacey Greece, DSc, MPH  Erin Lee, MPH  Jessica Levin  Leah Staub-DeLong Centers for Disease Control  Jan Jernigan, PhD  Gayle Payne, PhD  Sonia Ann Kim, PhD  Holly Wethington, PhD, MS  Raegan Tuff, MPH NYC Department of Health and Mental Hygiene  Sabrina S. Baronberg, MPH  Alyson Abrami, RD, MPH  Kasey Holloway, MPH

Abt Associates | pg 3 Presentation Overview  Health Bucks Program and Evaluation Overview  Stakeholder Input for Evaluation Design  Stakeholder Input for Evaluation Implementation  Stakeholder Input for Evaluation Dissemination  Questions?

Abt Associates | pg 4 NYC Health Bucks Program  Began as pilot program in South Bronx in 2005  Distributes coupons for purchase of fresh fruits and vegetables at participating farmers’ markets  Administered by NYC Department of Health and Mental Hygiene through District Public Health Offices (DPHOs), Farmers’ Market Federation of New York (FMFNY)  Distribution: –SNAP incentive –Community-based organizations

Abt Associates | pg 5 Health Bucks Evaluation  CDC-sponsored evaluation, guided by CDC Framework for Evaluation (1999) Utility → Feasibility → Propriety → Accuracy  Mixed methods public health approach  Process evaluation to inform replication efforts.  Outcome evaluation to assess impact  Collaborative approach to ensure relevance of evaluation design and foster stakeholder engagement

Abt Associates | pg 6 Stakeholders  CDC Division of Nutrition, Physical Activity and Obesity (DNPAO)  New York City Department of Health and Mental Hygiene (NYC DOHMH)  Farmers’ Market Federation of New York (FMFNY)  District Public Health Offices  Farmers’ Market Managers/Coordinators  Consumers and Farmers/Vendors  Researchers and Evaluators

Abt Associates | pg 7 Stakeholder Influence Across Multiple Phases Description of Program (Logic Model) Inform Instrument Design and Data Collection Plan Identification of and Access to Key Stakeholders Design Assistance with Data Collection Outreach to Key Stakeholders Continuous Input and Feedback Implementation Engage other researchers and evaluators Review of draft and final deliverables/reports Inform design of toolkit Dissemination

Abt Associates | pg 8 Stakeholder Influence During Design Program Logic Model

Abt Associates | pg 9 Stakeholder Input During Implementation Example of Consumer Focus Groups (FGs) Health Bucks staff, District Public Health Office staff, State staff, Market managers, vendors, survey staff, consumers

Abt Associates | pg 10 Stakeholder Input During Implementation Informed FG Logistics and Recruitment  Recruitment materials were prepared in English and Spanish;  Health Bucks, farmers’ market, and survey staff assisted with recruitment;  $35 incentive payments were provided in cash for consumer participants;  Groups were conducted in libraries and an asthma center in each DPHO – and scheduled for mid-morning and mid-afternoon

Abt Associates | pg 11 Stakeholder Input During Dissemination Products and Deliverables  Share Evaluation Findings and Seek Feedback on Reports  Input on Tool Kit –Address all phases of evaluation, using CDC Framework for Program Evaluation –Build in User-friendly Components Sections of Tool Kit mirror framework Step-by-Step Guide Links to resource materials, including all instruments used for Health Bucks evaluation Evaluation Tips Checklists

Abt Associates | pg 12 Stakeholder Input During Dissemination Tool Kit and CDC Framework Step 1: Engage Stakeholders Step 2: Describe the Program Step 3: Focus Evaluation Design Step 4: Gather Credible Evidence Step 5: Justify Conclusions Step 6: Share Lessons Learned

Abt Associates | pg 13 Stakeholder Input During Dissemination Examples of User-Friendly Components in Tool Kit —Evaluation Tip— Close the feedback loop—make sure to share your evaluation results with all respondent groups and stakeholders. Process Evaluation Questions What are the differences across District Public Health Offices, changes over time, barriers and facilitators to…  Program funding?  Program implementation (promotion/outreach; recruitment of markets and community based organizations; distribution methods and redemption; reimbursements to farmers)?  Roles of stakeholder organizations (and adherence to intended roles)?  Program monitoring and evaluation to track operations and inform program improvement? Health Bucks Evaluation Data Collection Techniques Key Informant Interviews Surveys Focus Groups Observations

Abt Associates | pg 14 For More Information  Yvonne Abel, Process Evaluation Lead:  Lauren Olsho, Abt Project Director:  Final Evaluation Report – Available Spring 2012  Health Bucks Evaluation Toolkit – Available late 2011 in the Health Bucks section of the Center of Excellence for Training and Research Translation website: