Western Region Evaluation Framework Update Andy Riesenberg, MSPH Team Lead, Food Security and Obesity Prevention Supplemental Nutrition Assistance Program Western Region

Feb 2015

Introduction

Community Example: Linda Vista Neighborhood, San Diego Feb 2015

What sets SNAP-Ed apart? Focus on low-income population Comprehensive strategies (nutrition education, marketing, environmental changes) Community and stakeholder engagement Evidence-based prevention strategies Feb 2015

Evaluation Framework – At a Glance Feb 2015

The outcomes are presented in the short‐term; medium‐term; long‐term; and impacts. Shortterm outcomes are early markers of program success. While there is no set time‐parameter for short‐term activities, generally these are the immediate results that can be measured during or after program implementation. Medium‐term and long‐term outcomes build upon previous accomplishments. For some outcomes in the outer spheres of the SEM, it will be difficult, if not impossible, to tease out the relative contributions of SNAP‐Ed. For these measures, we might consider the collective impact of partnerships. At the individual level, the distinction between medium-term and long-term is that medium-term represents intermediate markers of progression toward meeting the DGA and Physical Activity Guidelines for American (PAGA) recommendations. Medium-term outcomes represent changes in actions or behaviors as measured by pre- and post-surveys before and after individual and group based education and health promotion activities. The medium-term indicators are actionable for on-going program evaluation. Long-Term indicators at the individual level are more in alignment with the DGA and PAGA. The long-term indicators are designed for use by States that either conduct surveys among SNAP-Ed participants, or States that conduct annual or biennial population-level surveillance of low-income audiences using state-run surveillance systems, such as the Behavioral Risk Factor Surveillance System (BRFSS). The reference points for cups or servings of foods and beverages in long-term indicators align with population benchmarks used in surveillance surveys. This approach facilitates comparisons between the SNAP-Ed population and the general population when measuring State or national trends in nutrition and physical activity. Feb 2015

Evaluation Questions Individual‐level To what extent does SNAP‐Ed programming improve participants’ diet, physical activity, and health? Environmental‐level To what extent does SNAP-Ed programming facilitate access and create appeal for improved dietary and physical activity choices in the settings where nutrition education is provided? Sectors of Influence To what extent is the SNAP-Ed grant program integrated into comprehensive strategies that collectively impact lifelong healthy eating and active living in low-income communities? Social and Cultural Norms and Values To what extent do community-level obesity prevention strategies impact the public’s priorities, lifestyle choices, and values for healthy living? Feb 2015

Healthy People in Healthy Places in Healthy Communities Individual: People Environmental: Places Sectors of Influence: Communities Unit of Analysis Youth and Adults Settings and Networks Low-income Communities Reach Participants Organizations and Places Populations and Jurisdictions What Gets Measured? Behavioral changes in nutrition, physical activity Adoption, Implementation, Effectiveness, Maintenance Collective Impact/ Nutrition in all policies How to Measure? Surveys Focus Groups/ Interviews Direct Observation Assessments/ Environmental Scans Document Review Network Analysis Data warehouses CDC/DHHS/RWJF Indicators Health registries Feb 2015

Example Feb 2015

FY ‘15 Priority (Preferred) Indicators Feb 2015

FY 15 Priority Indicators – Basic Individual Level Environmental Level Sectors of Influence MT1: MyPlate Behaviors ST4: Identification of Opportunities MT9: Agriculture MT2: Shopping Behaviors ST6: Partnerships MT3: Physical Activity Behaviors LT2: Fruits and Vegetables LT4: Dairy LT6: Food Security Feb 2015

FY 15 Priority Indicators – Enhanced (States > $2 mil) Environmental Level Sectors of Influence MT4: Nutrition Supports Adopted MT8: Local Government MT5: Physical Activity Supports Adopted MT12: Health Care MT6: Marketing and Messaging Feb 2015

Measuring Individual Changes Feb 2015

Number of SNAP-Ed participants with a matched pre- and post-test. MT2 Shopping Behaviors  Logic Model Component   Medium-Term Outcome – Changes in individual and family behaviors that reflect smarter shopping and food resource management strategies, enabling participants to stretch their food resource dollars. What to measure: Number, or %, of SNAP-Ed participants who increased targeted shopping behaviors during the period assessed. For each participant, compare responses to survey administered before the first lesson and after the last lesson. Numerator: Number of SNAP-Ed participants with a matched pre- and post-test who performed the behavior at post-test for the first time or with greater intensity or frequency. Denominator: Number of SNAP-Ed participants with a matched pre- and post-test. Examples of behaviors Read nutrition facts or nutrition ingredients lists Compare prices before buying foods Identify foods on sale or use coupons Shop with a list How to Measure Visually-Enhanced Food Behavior Checklist Do you use this food label when shopping? Plan, Shop, Save, and Cook Survey How often do you shop with a grocery list? How often do you compare unit prices before buying food? Cooking Matters for Adults Survey How often do you compare prices before you buy food? How often do you use the “nutrition facts” on food labels?  Feb 2015

Example Feb 2015

Measuring Policy, Systems, and Environmental Changes Feb 2015

RE-AIM Reach, effectiveness, adoption, implementation, maintenance Reach, adoption, effectiveness, implementation, maintenance Feb 2015

REACH (MT4-5) Reach: Number of SNAP-Ed eligibles that benefitted from the change(s) during the period assessed (e.g., number of persons < 185% of Federal Poverty Level reached by the change in SNAP-Ed eligible settings). Delineates at the environmental level where services are provided. Short-term Opportunities, partnerships, community champions Medium-term Description of changes adopted Long-term Is SNAP-Ed working towards a comprehensive approach? WRO does not expect all the be covered, but at least two. Feb 2015

ADOPTION (MT4-5) Adoption: Aggregate number of SNAP-Ed settings where at least one organizational or environmental change is made in writing or practice to improve or strengthen access or appeal for healthy eating or physical activity during the period assessed Feb 2015

Interpreting Reach and Adoption MT4: Nutrition Supports Adopted  Logic Model Component   Medium-Term Outcome – Reach and adoption of nutrition policies or environmental changes What to measure: The documentation of change(s) adopted in the low-income setting and associated reach Numerator: Number of settings where at least one change is made in writing or practice to expand access or improve appeal for healthy eating Denominator: Number of settings in ST4, by category How to Measure ADOPTION Documentation (direct observation, photograph) or interviews with key informants to confirm the uptake of the policy or environmental change in the low-income setting. REACH Means-tested setting The number of SNAP-Ed eligibles, based on qualification for free or reduced price meals or federal poverty level, who benefit from the change(s). For example, if the school cafeteria adopts changes in meal service, and there are 750 students who qualify for free or reduced price meals, the reach is 750. Census tract, or census designated place The number of SNAP-Ed eligibles, based on census data and direct observation, who benefit from the change(s). For example, if there are 6,000 (out of 10,000) individuals within 185 percent of FPL in a given census tract where a new farmers market opens, and on average in the past month, there have been 350 customers at the market per shopping day, multiply 350 x .60, which equals 210. Feb 2015

Example Feb 2015

IMPLEMENTATION (LT9) Implementation: Aggregate number of SNAP-Ed settings that report a multi-component initiative with one or more organizational or environmental changes adopted AND at least one of the following: 1) evidence-based education, 2) marketing, 3) parent/community involvement, 4) staff training on continuous program and policy implementation. Feb 2015

EFFECTIVENESS (LT9) Effectiveness: Number of settings with improved food or physical activity environment assessment scores using a reliable and consistent environmental assessment tool. Feb 2015

MAINTENANCE (I3) Maintenance: Average percentage increase, or number of institutional or community resources invested in nutrition and physical activity supports or standards at SNAP-Ed settings in terms of staff (number of full time equivalents), cash, or in-kind supports Feb 2015

Channels: Eat, Live, Learn, Work, Play, Shop Eat: Restaurants, mobile vending/food trucks, congregate meal sites (or, other places where people primarily go to “eat”) Live: Public housing, shelters, places of worship, community organizations, residential treatment centers, adult or senior services (or, other community or neighborhood settings where people “live” or live nearby) Learn: Child care, head start, early care and education, adult education, schools, after-school, Cooperative Extension offices (or, other places where people go to “learn”) Work: Worksites with low-wage workers5, job training programs, TANF worksites (or, other places where people go to “work”) Play: Parks and recreation, YMCA, county fairs, Boys and Girls clubs, bicycling and walking paths (or, other places where people go to “play”) Shop: Farmers markets, grocery stores, food retailers, food pantries (or other places where people “shop” for food) Feb 2015

Feb 2015 Number of Sites by Category of Channel Eat Live Learn Work NEOPB PSE Strategy Eat Live Learn Work Play Shop Early Childcare   55 schools/ childcare centers School Wellness Policies 29 schools 23 district offices School Wellness Policy- Water Stations 1 city 23 schools Farm to School 9 district offices Joint Use Agreements 1 school Healthy Retail 70 stores Restaurant and Mobile Vending 2 vendors Structured Physical Activity 1 public housing 5 churches 2 schools Community/School Gardens 1 rehab center 4 apt complexes 18 churches 12 community sites 4 community centers 25 schools 1 park Worksite 52 worksites Safe Routes to Schools 2 neighborhoods 1 community center 14 schools 3 parks/ public land Farmers' Markets 16 markets 4 produce stands 1 business facility 1 public land Healthy Food and Beverage Standards 5 emergency food 28 churches 2 community centers 4 community-based organizations 3 WIC programs 2 cities 1 store Feb 2015

Next Steps Selecting priority indicators for FY 16 Developing instructions and reporting book for priority indicators Developing interpretive guide for 51 indicators Ongoing collaboration with ASNNA Eval Sub-committee, Regional Centers of Excellence, NIFA, CDC, & HRSA for alignment of indicators Feb 2015

Evaluation References CDC. State indicator report on fruits and vegetables. http://www.cdc.gov/nutrition/downloads/State‐Indicator‐ Report‐Fruits‐Vegetables‐2013.pdf CDC. State indicator report on physical activity. http://www.cdc.gov/physicalactivity/downloads/pa_state_indicator_report_2014.pdf CDC. Recommended community strategies and measurements to prevent obesity in the United States. (COCOMO) http://www.cdc.gov/mmwr/preview/mmwrhtml/rr5807a1.htm. Leeman et al. An evaluation framework for obesity prevention policy interventions. Prev Chronic Dis 2012;9:110322. Available at: http://dx.doi.org/10.5888/pcd9.110322. NIFA. Community Nutrition Education Logic Model. http://www.nifa.usda.gov/nea/food/fsne/logicmodeloverview.pdframework. Center for Training and Research Translation Web site. Available at: http://www.centertrt.org Altarum Institute and RTI International. Cates, S., Blitstein, J., Hersey, J., Kosa, K., Flicker, L., Morgan, K., and Bell, L. Addressing the Challenges of Conducting Effective Supplemental Nutrition Assistance Program Education (SNAP-Ed) Evaluations: A Step-by-Step Guide. Prepared by Altarum Institute and RTI International for the U.S. Department of Agriculture, Food and Nutrition Service, March 2014. Available at: http://snap.nal.usda.gov/ RE-AIM Framework. http://www.re-aim.hnfe.vt.edu/ Healthy People 2020. http://www.healthypeople.gov/2020. Feb 2015

Thank you! andrew.riesenberg@fns.usda.gov (415)645-1927 Feb 2015