1. Erik Augustson, PhD, National Cancer Institute Susan Zbikowski, PhD, Alere Wellbeing Evaluation

2. Learning Objectives What is evaluation? What are the benefits of evaluation? The importance knowing goals of the program (from different stakeholders perspectives) and how these goals will inform the evaluation. The components of a process evaluation. The components of an outcome evaluation.

3. Body of evidence- Research Consistent body of literature –Curry et al., 1995; Zhu et al., 1996 –Glibert et al., 2006; Tzelepis et al., 2011 Consistent support across multiple populations –Demographics –Comorbid medical and psychiatric Consistent support within international contexts –Largely within developed countries

4. Body of evidence- Research Quit rates 1.5-2.0 times higher than self-help alone Quit rates 1.1-1.6 times higher than medication alone Abstinence rates 10-20% - Meta-analysis ~13% Dose response with increasing # of calls delivered **Further research on general effectiveness is not high priority**

5. The difference between research and evaluation Research Done to test hypotheses Investigator controlled May required controlled conditions or a control group Focused on internal and external validity Isolate changes and control circumstances Evaluation Done to improve practice Stakeholder controlled Focused on utility, feasibility, and accuracy Incorporate changes and account for circumstances

6. “ What is Evaluation? ” Program evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future program development.

7. What is Evaluation? Practical and feasible Conduct within confines of resources, time, and political context Serve a useful purpose Conduct in an ethical manner to produce accurate findings Use results to make decisions and improvements

8. Why Evaluate Quitlines? To monitor progress toward program’s goals To demonstrate that the quitline is effective To determine whether the components of the quitline are producing desired effects To permit comparisons among groups, particularly among those with disproportionately high tobacco use and adverse health effects Justify the need for further funding and support To learn how to improve programs To ensure that only effective programs are maintained and that resources are not wasted on ineffective programs

9. Evaluation Planning- Steps STANDARDS Utility Feasibility Propriety Accuracy Engage Stakeholders Describe the Program Focus the Evaluation Design Gather Credible Evidence Justify Conclusions Ensure use and Share lessons learned

10. Goals and Objectives of the Evaluation A goal is the overall mission or purpose of the program –Promoting quitting among youth and adults Objectives are the results to be achieved by the program –Good objectives should be specific and measurable (SMART) Specific- specific goal or action to take place Measurable- Quantifies amount of change to be achieved Achievable and Ambitious- realistic but challenging Relevant- how it relates to the program goals Time-bound- time by which the objective is to be achieved

11. Stakeholders Identify key stakeholders: Stakeholders are people and organizations who are invested in the program and interested in the results Primary intended users of the findings (funding agency, tax payers, partners) Those served or affected by the program (tobacco users, advocacy groups, community members) Those involved in operations (program staff)

12. Stakeholders Know the goals of the program from different stakeholders perspectives and how these goals will inform the evaluation Considering the needs of stakeholders will increase the likelihood that the findings will be accepted and used

13. Typical Evaluations Process How well a program has been implemented Reach Operations Utilization Satisfaction Outcomes Assess the impact of the program on short, intermediate, and long- term objectives Change in beliefs Change in behavior Change in rates of smoking Change in morbidity/mortality

14. Exercise- Review Data Tables and Chart What about this data is useful? Does this data suggest a problem/concern? Can you think of possible contributions and causes How might you assess this problem/concern further?

15. Does this data suggest a problem or concern Can you think of possible contributions and causes? Decreased response time Decreased satisfaction Increased Calls

16. Possible Contributors and Causes How might you assess this problem/concern further? Operations __________ Services _________ Context _________ Decreased Satisfaction

17. Thank You