Yvonne Abel, Abt Associates Inc. November 2010, San Antonio, TX Enhancing the Quality of Evaluation Through Collaboration Among Funders, Programs, and Evaluators: The Example of the New York City Health Bucks Program Evaluation AEA 2010 Annual Meeting

Yvonne Abel, Cheryl Austin, Cristina Booker, Jacey Greece, Erin Lee, Jessica Levin, Leah Staub-DeLong, Lauren Olsho (Task Manager), Debbie Walker (Project Leader) The Intersection of Quality and Collaboration: From an Evaluator’s Perspective AEA 2010 Annual Meeting

November AEA Annual Conference 3 Presentation Overview I. Evaluation Overview II. Collaboration with New York City Department of Health and Mental Hygiene (NYC DOHMH) III. Collaboration with CDC Division of Nutrition, Physical Activity and Obesity (DNPAO) IV. Lessons Learned (to date) V. Next Steps

November AEA Annual Conference 4 I. Health Bucks Evaluation Overview  CDC-sponsored evaluation, guided by CDC Framework for Evaluation (e.g., utility, feasibility, propriety, and accuracy)  Process and outcome evaluation  Mixed-methods public health approach (through two key project phases)  Collaborative approach

November AEA Annual Conference 5 CDC Program Evaluation Framework & Health Bucks Evaluation Design and Implementation Form Core Evaluation Team Engage Key Stakeholders Site Visit Key Informant Interviews Document Review Finalize Research Questions Develop Logic Model Develop Study Protocols Engage Stakeholders Describe the Program Focus Evaluation Design Gather Credible Evidence Findings & Conclusions Ensure Use, Share Lessons Learned Clearance of Study Protocols Data Collection (qualitative & quantitative) Secondary Sources Data Analysis Interim Reports Final Report Toolkit Site Visit Key Informant Interviews Document Review Utility → Feasibility → Propriety → Accuracy

November AEA Annual Conference 6 II. Collaboration with New York City Department of Health and Mental Hygiene (NYC DOHMH) Influence Across Multiple Phases:  Proposal Phase  Formative Phase  Implementation  …and Findings (data collection and analysis still in process)

November AEA Annual Conference 7 II. Collaboration with New York City Department of Health and Mental Hygiene (NYC DOHMH), cont Collaboration during (Proposal and) Formative Phase:  Description of Health Bucks Program  Inform planning and evaluation efforts  Identification of and access to key stakeholders

November AEA Annual Conference 8 II. Preliminary NYC Health Bucks Program Description and Logic Model: A Tool for Planning and Evaluation of Ongoing Activities

November AEA Annual Conference 9 II. Inform Planning and Evaluation: Preliminary Research Questions

November AEA Annual Conference 10 II. Inform Planning and Evaluation: Preliminary Data Collection Plan

November AEA Annual Conference 11 II. Identification of and Access to Key Stakeholders… One example

November AEA Annual Conference 12 II. Collaboration with New York City Department of Health and Mental Hygiene (NYC DOHMH), cont Collaboration during Implementation:  Assistance with data collection  Access to program data from current and past seasons  Inform market and Health Bucks staff  Serve as point-of-contact  Outreach to key stakeholders  Talk to “experts” and listen  Get the timing right  Ongoing input and feedback  Updates on market operations and program changes  Guidance on real-time adjustments

November AEA Annual Conference 13 II. Collaboration During Implementation: Data Collection ActivityType of Respondents/Data SourceTimeframe Qualitative Data Collection Focus Groups Farmers’ market and Health Bucks consumers Farmers’ market vendors September 2010 November or March 2010 Interviews Key informants November 2009 – November 2010 Site Visits and Observation Farmers’ market observation November 2009 – November 2010 Quantitative Data Collection Surveys Farmers’ market consumers Neighborhood residents (RDD survey) Farmers’ market vendors Farmers’ market managers Local community organizations July – November 2010 October – November 2010 September – November 2010 October – November 2010 Secondary Data Analysis Data Analysis NYC Community Health Survey Dataset 2002 – 2009

November AEA Annual Conference 14 II. Collaboration During Implementation: Data Collection, cont Example of Consumer Focus Group Implementation:  Stakeholder assistance  Focus Group logistics  Guidance for recruitment of participants

November AEA Annual Conference 15 II. Collaboration During Implementation: Example of Consumer Focus Groups Stakeholder Assistance… Health Bucks staff, District Public Health Office staff, State staff, Market managers, vendors, survey staff, consumers

November AEA Annual Conference 16 II. Collaboration During Implementation: Example of Consumer Focus Groups, cont …informed Logistics and Recruitment  Recruitment materials were prepared in English and Spanish;  Health Bucks, farmers’ market, and survey staff assisted with recruitment;  $35 incentive payments were provided in cash for consumer participants;  Groups were conducted in libraries and an asthma center in each DPHO – and scheduled for mid- morning and mid-afternoon

November AEA Annual Conference 17 III. Collaboration with CDC Division of Nutrition, Physical Activity and Obesity (DNPAO) Essential during Design and Formative Phase:  Collaboration on instrument design  Consideration of CDC guidelines for OMB Continuous input & feedback throughout the project (to address challenges, adapt to changes, be time sensitive)

November AEA Annual Conference 18 IV. Lessons Learned Key Lessons Learned  Engage stakeholders at all levels  Practice transparency and gain permission to secure cooperation  Do not underestimate time and level of effort for OMB and IRB requirements  Be knowledgeable and aware of program operations  Build flexibility into data collection

November AEA Annual Conference 19 V. Next Steps Program Enhancements:  Feedback to NYC DOHMH to inform programming and evaluation efforts Future Research and Policy:  Findings to inform CDC’s efforts (as well as other researchers and policy community) Broader Dissemination:  “Tool kit” for other states/programs interested in similar efforts

November AEA Annual Conference 20 Thank You (on behalf of the entire Health Bucks evaluation team) Abt Associates Inc.