1. AMCHP-CDC Skills Building Suite: Evidence-based Practice: Moving along the Continuum from Selection to Sustainability Friday, September 13, 2013 Starr Banks, MPH Kathi Wilson, PhD, MPH, CHES

2. Quick Overview How to Use Web Technology Mute your line by using the mute function on your phone or by using *6 to mute/un-mute Asking a Question – You can type your questions into the chat box (shown right) – Raise your hand. Using the icon at the top of your screen (example shown right) Lastly active participation will make sure today’s webinar a success! 2

3. Evaluation and Sustainability of Your Evidence-Based Programs Starr Banks Kathi Wilson Community Guide Branch, Centers for Disease Control and Prevention Division of Epidemiology, Analysis, and Library Services (proposed)

4. Disclaimer Any views or opinions expressed by the speaker do not necessarily represent the views of the CDC, HHS, or any other entity of the United States government. Furthermore, the use of any product names, trade names, images, or commercial sources is for identification purposes only, and does not imply endorsement or government sanction by the U.S. Department of Health and Human Services. 4

5. Acknowledgements This training is brought to you by 5

6. Objectives Name your most important outcomes. Define different types of evaluation. Match your evaluation methods with your objectives. Identify components of sustainability. Assess your program’s sustainability. Recognize the importance of incorporating a sustainability plan for your evidence-based approach. 6

7. Community Assessment Implementation & Evaluation Establishing Goals & Objectives; Planning for Evaluation Finding Evidence Selection Adaptation 7 Where do We Stand? Sustainability

8. EVALUATION PLANNING

9. Why collect evaluation data? Communicate findings to staff, partners and funders Contribute to the evidence base. Create practice-based evidence When findings indicate that objectives were not met, explore and address the reasons When findings indicate objectives were met, celebrate!

10. Causes of Program Success or Failure How intervention is designed How the intervention is implemented SoundFaulty SoundProgram success Intervention problem Faulty Implementation problem Program failure

11. What outcomes are most important?  In designing an evaluation plan, find balance between Desired outcomes Feasible outcomes

12. Outcomes Are the changes that result from your program Are linked to either activities or objectives Can be Immediate Short-term Long term

13. Poll Question #1 13

14. General Evaluation Procedure PlanningDevelop the questions, consult with the program stakeholders or resources, make a timeline. Data CollectionPilot testing. How will the questions be asked? Who will ask them? Data AnalysisWho will analyze the data and how? ReportingWho will report and how? Who will receive the data and when? How will it affect the program? ApplicationHow could your results be applied in other places?

15. Objectives Contribution to Evaluation Objective Type Desired Result (Examples) What to Evaluate Process Intervention delivered as planned Target audience engaged Intervention delivery Attendance or awareness Satisfaction Learner (Immediate) Increased knowledge Attitude change Changes in knowledge Changes in attitudes -Behavior change -Policy or system change (Short-term) Increased activity Increase number of places to buy fresh food in the community Behavior changes Policy changes Cause of changes Overall (Long-term) Improved QOL Reduce morbidity, mortality Health status changes Cause of changes

16. Poll Question #2

17. Write SMART objectives Specific: Boy and girls between the ages of 10-13 Measureable : Decrease obesity rate in adolescents from 35% to 5% Achievable: Make sure a 30% decrease is possible in the time fame given Realistic: Adolescents in 3 middle schools in Oconee County, GA Time-framed: In 2 years

18. Activities Achieve Objectives Activities are the specific things that you do to accomplish your objectives. Link activities to objectives Describe actions Specify timing and intended population BOTTOM LINE: Each activity leads in an obvious way to an objective.

19. Evaluation Planning: Evaluation Phases and Types  Process evaluation Are all activities being implemented as planned? Are you reaching the intended audience?  Short-term outcome evaluation Does behavior change? Is a policy documented? Is it enforced?  Long-term outcome evaluation Does health improve? Do morbidity and mortality decrease?

20. Process Evaluation  Can find problems early  Includes an assessment of the staff, budget and how well the program is doing overall  Includes measurement of adoption, reach, participation, exposure, implementation, satisfaction, barriers and facilitators, and maintenance  Data collection methods may include: databases registration sheets surveys interviews Windsor et al., Evaluation of health promotion, health education, and disease prevention programs, 2003.

21. Short-term Outcome Evaluation  Assesses short-term effects on awareness, knowledge, attitudes, behavior, setting or environment  Can assess adoption or enforcement of a policy (including in schools, worksites, faith organizations, etc.)  Measures progress in meeting your objectives  Data collection methods may include: surveys interviews focus group observation program implementation logs

22. Poll Question #3

23. Evaluation Methods & Tools  Tracking logs: Keep an ongoing record of activities or outcomes  Existing data: Use data that has already been collected for another purpose.  Surveys or interviews: mailed, handed, or emailed to clients, patients, providers either one time or both before and after an activity.  Focus groups: Open ended questions asked of a small group of similar people to explore questions more deeply.  Observation: site visits to document extent to which activities are implemented as planned. 23

24. Long-term Outcome Evaluation  Assesses if the long-term goals were met  Measures sustained program or strategy changes   Measures longer term changes in rates of illness, death, or health status of your population  Use common data collection methods

25. Activity #1

26. Sharing Evaluation Results  Evaluation findings need to reach all stakeholders In language that everyone understands In time to make decisions  Program maintenance v. adaptation based on evaluation findings  Broader dissemination and application

27. Sample 1: Evaluation Worksheet

28. Sample 1: Evaluation Worksheet Instructions  Program Goal and Objectives column Write a major objective of the program.  Overall Intervention column Write at least one activity that will help you achieve your objective.  Evaluation Activities column Look for possible evaluation activities. They should be based on the goals and objectives as well as planned intervention activities.  Evaluation Results column Brainstorm about what you hope to learn from the evaluation.  Evidence of Activities and Quality column Brainstorm about what kinds of materials will be needed. For example, a focus group would require a question guide and consent forms. You might need instruction materials for the participants. Are they available?  Evidence of Results column Think about what kind of results would show that an objective has been met.

29. Sample 2- Smoke-free Policy Evaluation Methods Type of Evaluation Evaluation QuestionData Data Collection Method Process Evaluation What materials were created for smoke-free policy formation? How many organization were targeted? Number of materials or organizations Interview with staff Survey of organizations Process Evaluation Did policy staff implement all process steps Checklist of conduct of each policy process step Policy implementation checklist Process Evaluation What are barriers or facilitators to implementation or enforcement? Barriers, FacilitatorsOrganizational survey Key informant interviews Short-term Outcome Evaluation How many organizations adopted the policy? Number of organizations Survey of participants (after the program) about healthy eating Review of church policy Outcome Evaluation What are the smoking rates after the smoke-free rule adoption? Number of employees who smoke Annual employee survey

30. Evaluation Resource List  CDC Framework for Program Evaluation  http://www.cdc.gov/eval/framework/index.htmhttp://www.cdc.gov/eval/framework/index.htm  American Evaluation Association http://www.eval.org/http://www.eval.org/  CDC Demonstrating your Program’s Worth http://www.cdc.gov/ncipc/pub-res/demonstr.htm http://www.cdc.gov/ncipc/pub-res/demonstr.htm  Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) http://cancercontrol.cancer.gov/is/reaim/ http://cancercontrol.cancer.gov/is/reaim/  The Evaluation Center at Western Michigan University www.wmich.edu/evalctr www.wmich.edu/evalctr  University of Wisconsin - Extension: Program Development and Evaluation http://www.uwex.edu/ces/pdande/index.htmlhttp://www.uwex.edu/ces/pdande/index.html  W.K. Kellogg Foundation Evaluation Handbook http://www.wkkf.orghttp://www.wkkf.org

31. Evaluation References  Glasgow, R.E., Vogt, T.M., & Boles, S.M. (1999) Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89, 1322-7.  Audrey, S., Holliday, J., Parry-Langdon, N., & Campbell, R. (2006). Meeting the challenges of implementing process evaluation within randomized controlled trials: The example of ASSIST (A Stop Smoking in Schools Trial). Health Education Research, 21, 366-77.  Beresford, S.A., Shannon, J., McLerran, D., & Thompson, B. (2000). Seattle 5-a-Day work-site project: Process evaluation. Health Education & Behavior, 27, 213-22.  Grembowski, D. (2001). The practice of health program evaluation. Sage: Thousand Oaks, CA.

32. SUSTAINING YOUR INTERVENTION

33. Community Assessment Implementation & Evaluation Establishing Goals & Objectives; Planning for Evaluation Finding Evidence Selection Adaptation 33 Where do We Stand? Sustainability

34. Planning for Sustainability Sustainability is… … the existence of structures and processes which allow a program to leverage resources to most effectively implement evidence-based policies and activities. Sustainability capacity is… …the ability to maintain programming and its benefits over time. Sustainability requires… … knowledge of factors critical to program success. Luke D, Calhoun A, Herbers S. Building & Managing Sustainability in Public Health Programs. CDC Roundtable Presentation. Feb. 23, 2012

35. Why is planning for sustainability important?  WITHOUT planning for sustainability Waste $ and other resources Damage trust between program and community Limit ability to succeed  WITH planning for sustainability Lasting outcomes Healthier communities 35

36. Sustainability Planning Resources http://ctpr.wustl.edu/sustainability/

37. Sustainability Tool Definitions  Program – refers to the set of formal organized activities that you want to sustain over time.  Organization – encompasses all the parent organizations or agencies in which the program is housed.  Community – refers to the stakeholders who may benefit from or who may guide the program. 37

38. Program Sustainability Assessment Tool  Purpose – Assess program’s current capacity for sustainability across a number of factors Identify strengths and challenges to sustainability Guide action to improve sustainability as part of program planning 38

39. Program Sustainability Domains  Funding Stability – making long term plan on stable funding  Political support – internal and external political environments  Partnerships – connection between program and community  Organizational Capacity – resources needed  Program Adaptation – ability to adapt and improve  Program Evaluation – monitoring of processes and outcomes  Communications – strategic distribution of successes  Public Heath Impacts – program’s effect  Strategic Planning – process that defines program directions

40. Sustainability Activities - Examples  Seek additional funding  Maintain or secure a program champion  Continue partnerships developed or build new ones to share costs/implementation  Referrals for further service, if needed  Make intervention a part of organizational services  Surveillance of the impact of the PSE (e.g., tracking of smoking rates, BMI of children)

41. Poll Question #4 41

42. Homework Challenge  Complete Sustainability Plan - Center TRT 42

43. Questions & Discussion 43

44. Center for Surveillance, Epidemiology, and Laboratory Services (proposed) Division of Epidemiology, Analysis, and Library Services (proposed) Thank you! Kathi Wilson Starr Banks Community Guide Branch, Centers for Disease Control and Prevention Kathi Wilson kwilson@cdc.govkwilson@cdc.gov Starr Banks smbanks@cdc.govsmbanks@cdc.gov