MPR Impact Reporting Survey questions, aggregation, and moving forward Sandra Biller May 10, 2018 SNAP-Ed Regional Conference

Benefits of Aggregation Show region-wide impacts Better illustrate program value to stakeholders Assess impacts across states/agencies Highlight programs of excellence Monitor performance Quality improvement Track progress towards outcomes at a regional level

Background State/regional impact reports Can aggregate reach Cannot easily aggregate impacts Problems with aggregation Known variations in measurement of impacts Lack of knowledge about specifics What tools are being used? What questions are being asked? How are surveys conducted? (pre/post, retrospective, etc.) Can we collect this information to get a baseline?

Objective & Methods Objective Assess similarities and differences in survey strategies Priority indicators MT1-3 Adults and youth Goal: inform potential aggregation solutions used in future fiscal years Methods Collect surveys, questions, or data points used from Implementing or State Agencies All 10 states within MPR contacted Data compiled and examined for themes, with a focus on: Fruit Consumption Vegetable Consumption Food Resource Management Physical Activity

Results: Overview 80% of states responded (n=8) 84% of agencies included (n=11) Data is still incomplete 257 Questions 119 Adult 138 Youth All states track both adults and youth 90% of questions are used by 1-2 states

Adult Fruit Consumption Example Fruit Frequency Questions Source # of States Do you eat more than one kind of fruit each day? EFNEP Adult Behavior Change Checklist 4 Eat different kind of fruits Kansas 1 How often do you typically eat fruit? Cooking Matters Adults, Parents, Families Survey During the past week, not counting juice, how often did you eat fruit? Create Farm Fresh Food During the past week, how often did you eat more than one type of fruit? Food $ense (6-lesson follow-up)

Adult Vegetable Consumption Example Vegetable Frequency Questions Source # of States Do you eat more than one kind of vegetable each day? EFNEP Adult Behavior Change Checklist 4 Eat different kinds of vegetables Kansas 1 During the past week, how often did you eat more than one type of vegetable? Food $ense (6-lesson follow-up) I eat more than 1 kind of vegetable per day because of the parent wellness workshops. Culture of Wellness in Preschools (post)

Adult Fruit & Vegetable Highlights Two main strategies Measure frequency of intake Measure volume of intake More variation for vegetable consumption Several states use EFNEP questions Wording of questions is similar Examples: How often do you typically eat fruit? During the past week, not counting juice, how often did you eat fruit? During the past week, how often did you eat more than one type of fruit?

Adult Food Resource Management Example Food Resource Management Questions by Category Source # of States MT2b - How often do you use the Nutrition Facts label to make food choices? EFNEP Adult Behavior Change Checklist, Cooking Matters Adults, Parents, Families Survey 7 MT2g - How often do you run out of food before the end of the month? EFNEP Adult Behavior Change Checklist 5 MT2h - How often do you compare prices before you buy food? 6 MT2j - How often do you shop with a grocery list?

Adult Food Resource Management Highlights 100% of responding states track two indicators MT2b – Read Nutrition Facts labels MT2j – Shop with a list EFNEP questions used by a large majority Wording of questions is very similar: Examples: How often do you shop with a grocery list? How often do you use a grocery list when grocery shopping?

Adult Physical Activity Example Physical Activity Questions Source # of States Do you participate in at least 30 minutes of moderate physical activity daily? EFNEP Adult Behavior Change Checklist 3 Are you active on 4 or more days a week? 2 I participate in physical activities for at least 30 minutes per day Kansas 1 Choose to be physically active for at least 30 minutes, 5 times a week Food $ense (6-month) How often are you physically active for at least 30 minutes on 4 or more days of the week? UNL-NEP No. of days physically active at least 60 mins/day Culture of Wellness in Preschools

Adult Physical Activity Highlights 88% of responding states track increase in time spent physically active Some states specify moderate physical activity – would bring this to 100% Less agreement on specific questions Wording of questions is similar Examples: Do you participate in at least 30 minutes of moderate physical activity daily? I participate in physical activities for at least 30 minutes per day Choose to be physically active for at least 30 minutes, 5 times a week

Youth Significant variation in evaluation: Grades/ages targeted Questions asked Methodology Pre/Post Post-only Parent/staff surveys Little overlap between states When overlap did occur, an age- specific EFNEP survey was used Due to these challenges, was unable to pull out many common themes.

Challenges and Limitations Variations in methodology, particularly with youth Some disagreement on what questions count for what indicators Example: Youth FRM ST vs. MT In some cases, questions asked are not similar Examples: Do you eat more than one kind of vegetable each day? How often do you typically eat green salad? More common for youth New and upcoming changes in surveys (EFNEP) Tight time frame for collecting data Did not collect question response options (i.e., what type of Likert scale was used?)

Commonalities(Adult) All states track MT1-3 Significant overlap in outcome measures Within outcome measures, similar wording of questions Large variability in specific questions/survey tools

How should we aggregate our data? Align outcome measures Example: MT1c. Ate more than one kind of fruit throughout the day or week Ask the same questions with the same response options Example: Do you eat more than one kind of fruit each day? Use the same evaluation tool Benefits and challenges to data aggregation SNAP-Ed adoption of the Interpretive Guide – a first step in the process More discussion needed to decide on a solution

Many Thanks All states who so generously let me use their data for this project! MPR Evaluation workgroup Wyoming SNAP-Ed Team