1. Evaluation and Implementation 21 October 2015 PUBH 535

2. Phase 5: Evaluation 21 October 2015 PUBH 535

3. Sources Social Marketing for Nutrition and Physical Activity Web Course: Phase 5: Evaluation www.cdc.gov/nccdphp/dnpa/socialmarketing/ training www.cdc.gov/nccdphp/dnpa/socialmarketing/ training

4. Objectives List the six steps for evaluation Explain how to gather credible evidence and what to gather it on Discuss options for evaluating the 4 P's Discuss use of the evaluation results and the importance of sharing results

5. Introduction Develop a plan for evaluating the intervention This will overlap with phase 4. May want to create this eval. while completing phase 4 Good idea to appoint an evaluator (professor) Someone who knows research methods

6. Evaluation Purpose Gain insight- basically formative research Improve the program- process evaluation Assess program efforts- outcome evaluation

7. Process Evaluation – Monitor whether your intervention was implemented – Monitor if your intervention was implemented as intended – Determine how and why certain things happen as they did

8. Outcome and Impact Evaluation What outcomes to evaluate from program? May access proximal outcomes For example: amount of fruits and vegetables eaten Or amount of physical activity performed by target audience

9. Steps for Program Evaluation Engage Stakeholders Describe the Problem Focus the Program Gather Credible Evidence Justify Conclusions Ensure Use and Share Lessons Learned

10. Step One: Engage Stakeholders People with a vested interest in program and future use of its evaluation – Implementers: those involved in program ops. – Partners: those who actively support the program – Participants: those served or affected by program – Decision Makers: those in a position to make changes in the program Who can help with eval. Plan

11. Step 2: Describe the Program Work from phases 1-4 of the SM process Compile the details from your logic model – Influential factors -Long-Term Outcomes – Inputs -Program’s Goal – Activities – Outputs – Initial Outcomes – Intermediate Outcomes

12. Step 3: Focus the Intervention Look at logic model- see what you can eval Did this happen? Did this relationship occur? Only answer necessary questions What do you want to know at the end of eval? Do you need to measure impact to get additional funding Focus on what you can change

13. Identify Evaluation Questions Eval questions should be broad They are used for program implementation Examples of Questions: – Was target audience exposed to the intervention? – Did exposed audience members change behavior? – Did behavior change occur the way we expected?

14. Criteria for Evaluation Amount of resources available Amount of resources needed to get an answer Willingness to change as a result of an answer What do you already know? Process or outcome eval?

15. Step 4: Gather Credible Evidence You need to identify: Who will answer questions Method for answering Data source Evaluation design Logistics How will you collect the info This should be used as a marker for how you did and data for the next phase

16. Indicators You will now use indicators for data collection Indicator is a description of a characteristic or change you expect you see that shows progress you are making toward achieving a specific outcome, what you measure: – Minutes of physical activity – Amount of fruits and vegetables consumed – Number of volunteers

17. Performance Indicator Develop a performance indicator, or goal for each indicator This sets the bar for what success looks like Example: – 15% increase in minutes of physical activity – 2 cups of fruit per day – 25 total volunteers – See table in CDC material "Kids walk to school"

18. Monitoring the Program Monitoring is a type of process evaluation that keeps track of intervention being implemented Can get audience feedback to make adjustments in intervention Be prepared to make changes mid-course

19. Monitoring the Program Program Implementation: going the way you planned? Effectiveness of strategy and activities Other possible changes: Any unintended consequences from program? Feedback from participants and implementers Program's reach: Reaching intended audience?

20. Monitoring the Program Go ahead and fix what needs to be fixed Monitor the audience Make sure you are addressing real issues Also get feedback from non-participators in target audience – Can provide insight to why they did not participate – See the example in the CDC guide about VERB program

21. Evaluating Product and Price Determine if the product and related benefits were acceptable to the audience How well did you minimize costs? – Did the audience understand what you were asking them to do? – Did they attempt the behavior? – Did they adopt it? – Were benefits enough to persuade overcoming costs?

22. Monitor Competition Did anything happen that would increase audience's desire to participate in competing behaviors? – Did price of unhealthy items drop in vending machines? – Were legislators persuaded to adopt a policy opposing the one you were advocating? – Or did program have boomerang effect?

23. Evaluating Place Successful at putting materials and messages in places you identified? Were places the best ones to reach the target audience? Did target have easy/ convenient access to all tangible products/ services you offered?

24. Evaluating Promotion Keep exposure in mind How many people where exposed to your program? Did you reach the right people? Look at all the ways you promoted program – Did word of mouth work? – How did participants hear about program?

25. Measuring Behavior Change Look back to behavioral objectives in Phase 3 Did you met your behavioral objectives? May want to measure any precursors to change – Intention, knowledge, skills, self-efficacy – Was your logic model accurate?

26. Measuring Policy/ Environmental Change Measure how well implementation and enforcement occurred Do changes make a difference? Did city council approve funds? Were sidewalks (for example) installed? Do you have enough resources? Are people using sidewalks? Are people walking more? Intended outcomes

27. Step 5: Justify Conclusions Analyze data and interpret the results Compare results to performance indicators How successful were you? Did you find any alternative explanations for your results? Unexpected results?

28. Step 6: Ensure Use and Share Lessons Learned Prepare tangible products of evaluation (recommendations and reports) Share with stakeholders and other audiences Maybe want to publish reports in peer reviewed journals or online Use results to improve the program

29. Learn from Results Look at data to see why things did not go the way you wanted them May want to conduct further qualitative research to help understand results Share this with your planning committee This can help for future plans Learn from mistakes and successes!

30. Summary Now have a Evaluation plan Understand the six steps Understand how to learn from mistakes, adjust your plan, and share your results

31. Phase 6: Implementation 21 October 2015 PUBH 535

32. Objectives Discuss the process for implementing an intervention Develop a program budget and timeline Discuss the importance of planning for sustainability

33. Introduction- Time for Action Implement intervention and monitoring/evaluation plan Be sure to monitor the implementation process and make adjustments as needed Try to observe the program as it is going on Observe any issues and try to correct them Keep notes on what you are learning/ observing

34. Develop a Program Budget Should have a budget before now but consider rest of spending Need to predict costs for each activity and funding sources Create budget with committee and partners Think about and solicit other funding sources Reassess partners May have to scale back activities

35. Develop a Program Timeline Should also have a timeline by now Keep track of what needs to be done for each activity Keep it flexible and allow for mid mods Remember all tasks and activities involved Start and completion dates Major deadlines and who is responsible for meeting deadlines Any internal or external review processes that have to take place before you can move forward.

36. Implementing in Phases May need to implement program in steps and phases so you can concentrate resources at a time Do not have to do everything at once Good side is that you can do a process evaluation without overwhelming yourself You can make adjustments as you go along Remember: you can do a pilot program

37. Sustainability of Program Think ahead, what if something happens to you? Can someone take over or can you find new funding? Can the program continue after the plan duration? Long term implementation of your plan?

38. Sustainability of Behavior How easy will it be for target audience to continue their new behaviors after the program is over? This can be addressed by looking at environmental and policy changes for permanent/ continued support. Remember the social ecological model? – Can help address behavior sustainability

39. Summary Process for implementing an intervention Develop a program budget and timeline Importance of planning for sustainability Always keep an audience focus and have realistic expectations Keep feedback for future use Always learn, always improve, and have fun!