3. Session 4 Agenda 1. Strategic Prevention Framework Sustainability Step 4: Implementation Step 5: Evaluation 2. Bringing It All Together 2

4. Session 4 Learning Objectives Recognize the keys to sustainability and how they are integrated into the SPF Describe high quality implementation and why it’s important Explain the purpose of evaluation, and considerations when reporting the results Identify ways to apply the training material to your work 3

5. Sustainability 4

6. Information Sheet 4.1: Keys to Sustainability 5 Sustainability is the process of an effective system achieving and maintaining desired long-term results.

7. 6

8. Keys to Sustainability  Community Support  Organizational Capacity  Effectiveness 7

9. ACTIVITY – Sustainability in the SPF  Form small groups (3, 6, 9, or 12 groups total).  Each group is assigned one of the three keys to sustainability: o Build community support o Enhance organizational capacity o Ensure effectiveness  As a group, determine the following: o What does your “key” have to do with sustainability? o Give an example of that key based on the discussion yesterday 8

10. Key: Build Community Support Encourage Maintain Develop Connect Celebrate 9

11. Key: Enhance Organizational Capacity Enhanced organizational capacity Formal Linkages Access to resources Strong Administrative Support 10

12. Key: Ensure Effectiveness Interventions that are aligned Interventions that are a good fit and evidence-based High quality implementation Evaluation plan that provides ongoing data 11

13. Step 4: Implementation 12

14. What’s Wrong with this Slogan? “JUST DO IT” – Nike “JUST DO IT” – Nike 13

15. What Does Implementation Involve? IMPLEMENTATION Mobilize Support & Build Capacity Carry Out Evidence-based Interventions Monitor, Evaluate, & Adjust Information Sheet 4.2: Key Tasks of Implementation 14

16. Mobilize Support and Build Capacity Introduce intervention to stakeholders Provide training Increase community awareness 15

17. Implementing Evidence-Based Interventions with Fidelity Fidelity involves adhering to the components of a program and the procedures for implementing it Components Procedures PROGRAM 16

18. Adapting Evidence-Based Interventions Information Sheet 4.3: Guidelines for Adaptation 17 ADAPTATION FIDELITY

19. CASE STUDY ACTIVITY – Implementation, Part 1  Get in your case study groups.  On Worksheet 4.4: Case Study Activity – Implementation, Part 1, write the risk factor and intervention you selected from Session 3, Worksheet 3.13: Case Study Activity – Determining Fit.  Brainstorm a list of action steps to take in the first 3-6 months of implementing this intervention. Write the risk factor, intervention, and action steps on Worksheet 4.4: Case Study Activity – Implementation, Part 1.  Assign someone to report out. Worksheet 4.4 Case Study Activity – Implementation, Part 1 18

20. Sample Action Plan TaskWho?Timeline Designate Date and location Coordinator & Coalition Chair By 5/1 Coalition planningCoalition MembersBy 6/15 Coordinate with the community Designated coalition members with key community stakeholders By 7/15 Intervention: Community Clean Up 19

21. CASE STUDY ACTIVITY – Implementation, Part 2  In your case study groups, brainstorm potential challenges to implementing the intervention you selected (refer to Part 1). Write them down on Worksheet 4.4: Case Study Activity – Implementation, Part 2  Each group will exchange their list of challenges with another group (including the information about the risk factor, intervention and action steps from Part 1)  Brainstorm potential solutions to the challenges identified by the other group for their risk factor and intervention.  Assign someone to report out one challenge/solution Worksheet 4.4 Case Study Activity – Implementation, Part 2 20

22. Implementation of Evidence-Based Interventions Step 3: Create an action plan Step 4: Mobilize support Step 4: Carry out intervention Step 4: Monitor, evaluate and adjust Step 5: Process and outcome evaluation 21

23. Step 5: Evaluation 22

24. Evaluating Outcomes Process Evaluation Outcome Evaluation Interventions Short-term and Long-term Outcomes Information Sheet 4.5: Overview of Evaluation Did anything change? Was the intervention completed as intended? 23

25. ACTIVITY – Process or Outcome? Worksheet 4.6: Evaluation Questions  Work individually on Worksheet 4.6: Activity – Evaluation Questions  Refer to Information Sheet 4.5: Overview of Evaluation, if needed.  Review answers with the large group. 24

26. Logic Model: Short- and Long-term Outcomes Risk and Protective Factors Interventions Behaviors and Related Problems Long-term Outcomes Short-term Outcomes 25

27. Example: High Blood Pressure Risk and Protective Factors Interventions Behaviors and Related Problems Long-term Outcomes High blood pressure Overweight Exercise (Long-term) Lower blood pressure to 120/80 (Long-term) Lower blood pressure to 120/80 Short-term Outcomes (Short-term) Reduce weight by 10 pounds (Short-term) Reduce weight by 10 pounds Reduce size of food portions 26

28. Example: Underage Drinking Risk and Protective Factors Interventions Behaviors and Related Problems Long-term Outcomes Underage drinking Retail access Compliance checks Merchant education (Long-term) Reduce underage current use (Long-term) Reduce underage current use Short-term Outcomes (Short-term) Reduce the number of sales to minors (Short-term) Reduce the number of sales to minors 27

29. CASE STUDY ACTIVITY – Determining Outcomes Risk and Protective Factors Interventions Behaviors and Related Problems Long-term Outcomes Underage Drinking (Long-term) Reduce Underage Current Use to 30% (2017) (Long-term) Reduce Underage Current Use to 30% (2017) Short-term Outcomes  Get in case groups. Use the risk factor and intervention in Worksheet 4.4: Case Study – Implementation.  Determine a short-term outcome for that risk factor.  Write your risk factor, intervention, and short-term outcome on three large post-it notes (one color per risk factor).  Each group will post their risk factor, intervention, and short-term outcomes in the appropriate column of the logic model in the front of the room. 28

30. Example of Successful Outcomes Projected Outcomes Actual Outcomes (from the Evaluation) Short-term Outcomes Reduce prevalence of high school youth who obtain alcohol from social sources From 38% of high school youth in 2012 to 30% in 2014 Prevalence of social access 2012 – 38% 2013 – 35% 2014 – 30% 2015 – 29% 2016 – 28% 2017 – 28% Long-term Outcomes Reduce current alcohol use among high school youth (past 30 days) From 33% in 2012 to 30% in 2017 Current use (past 30 days) 2012 – 33% 2013 – 33% 2014 – 33% 2015 – 31% 2016 – 30% 2017 – 28% 29

31. Projected Outcomes Actual Outcomes (from the Evaluation) Short-term Outcomes Reduce prevalence of high school youth who obtain alcohol from social sources From 38% of high school youth in 2012 to 30% in 2014 Prevalence of social access 2012 – 38% 2013 – 35% 2014 – 30% 2015 – 29% 2016 – 28% 2017 – 28% Long-term Outcomes Reduce current alcohol use among high school youth (past 30 days) From 33% in 2012 to 30% in 2017 Current use (past 30 days) 2012 – 33% 2013 – 33% 2014 – 33% 2015 – 33% 2016 – 33% 2017 – 33% Making Mid-Course Corrections 30

32. ? Why Evaluation is Important Monitor Implementation Improve Implementation Make Future Decisions 31

33. Reporting Evaluation Results Brief stakeholders throughout Plan how to disseminate results Select formats for reporting results Help stakeholders understand the data Information Sheet 4.7: Reporting Evaluation Results 32

34. CASE STUDY ACTIVITY – Reporting Your Results Worksheet 4.8: Case Study Activity – Reporting Your Results  Get in your case study groups.  Identify one stakeholder group related to the previous logic model activity.  Answer the following questions and write your answers on Worksheet 4.8: Case Study Activity – Reporting Your Results. o What is the primary areas of concern or interest for this stakeholder group, in terms of the evaluation results? o What is the most appropriate format for report the evaluation results to that stakeholder group, and why?  Share with the large group. 33

35. In Conclusion… 1.My feelings about evaluation are… 2.One thing I learned about evaluation is… 3.A question I still have about evaluation is… 4.One thing that I am going to do at work, given what I have just learned about evaluation, is… 34

36. Strategic Prevention Framework Information Sheet 4.9: SAMHSA’s Strategic Prevention Framework At-A-Glance Step 1: Assessment Assess Behaviors and Related Problems Prioritize Problems (criteria: largest, most severe, getting worse, high in comparision) Assess Risk and Protective Factors Step 2: Capacity Assess Capacity: Resources and Readiness Build Capacity: Increase Resources and Improve Readiness Step 3: Planning Prioritize Risk and Protective Factors (criteria: importance, changeability) Select Interventions (criteria: effectiveness, conceptual fit, practical fit) Develop a Comprehensiv e Plan that Aligns with the Logic Model Step 4: Implementa- tion Build Capacity and Mobilize Support Carry Out Interventions Balance Fidelity with Necessary Adaptations Monitor, Evaluate, and Adjust Step 5: Evaluation Conduct Process Evaluation Conduct Outcome Evaluation Recommend Improvments and Make Mid-Course Corrections Report Evaluation Results 35

37. Bringing It All Together 36

38. Review: Curriculum  Setting the Foundation: From Theory to Practice  Strategic Prevention Framework:  Steps 1 – 5  Cultural Competence  Sustainability 37

39. Review: Your Expectations  Did you learn what you came here to learn? How so?  Did you experience what you came here to experience? How so?  What did you learn that you didn’t expect to learn?  In what ways was the training valuable to you? 38

40. Assess Your Learning Experience You have the tools… Worksheet 4.10: Assess Your Learning Experience 39

41. Thank You & Congratulations!! Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it is the only thing that ever has. - Margaret Mead 40

42. Final comments 41 captus.samhsa.gov/

43. References 1 Substance Abuse and Mental Health Services Administration, Center for the Application of Prevention Technologies. (n.d.). Keys to sustainability [slide presentation]. Rockville, MD: Author. Unpublished. 2 Substance Abuse and Mental Health Services Administration, Center for the Application of Prevention Technologies. (n.d.). Keys to sustainability [slide presentation]. Rockville, MD: Author. Unpublished. 3 Substance Abuse and Mental Health Services Administration, Center for the Application of Prevention Technologies. (n.d.). Keys to sustainability [slide presentation]. Rockville, MD: Author. Unpublished. 4 Substance Abuse and Mental Health Services Administration, Center for the Application of Prevention Technologies. (n.d.). Keys to sustainability [slide presentation]. Rockville, MD: Author. Unpublished. 5 Substance Abuse and Mental Health Services Administration, Center for the Application of Prevention Technologies. (n.d.). Keys to sustainability [slide presentation]. Rockville, MD: Author. Unpublished. 6 Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Prevention. (n.d.). Evaluation for the unevaluated: Program evaluation 101 [online course]. Retrieved from http://pathwayscourses.samhsa.gov/eval101/eval101_1_pg7.htm 42