Paul Hunting, MPH Health Education Specialist Centers for Disease Control and Prevention Office on Smoking and Health TM Session #: EVAL 111 National Conference on Tobacco or Health Minneapolis, MN October 25, :30 – 5:00 pm Use of Outcome Indicators for Planning and Evaluating the NTCP The findings and conclusions in this presentation are those of the author and do not necessarily represent the views of the Centers for Disease Control and Prevention.

Presentation Outline  Project approach  Methodological approach  Findings  Recommendations to states  Possible focus for further research. TM

Key Outcome Indicators Document  Key Outcome Indicators for Evaluating Comprehensive Tobacco Control Programs This guide provides information on 120 key outcome indicators for evaluation of statewide, comprehensive tobacco prevention and control programs. Indicators are organized by evidence-based logic models.

Project Approach Looking at information provided by states in the NTCP Chronicle, we tried to answer the following questions: – How many states are working towards each goal? – How many states use which indicators? – Which indicators are states using most often? TM

Methodological Approach  Reporting period for the state information analyzed is June 2003 – June Select Indicator use frequency re-assessed for plan years.  Indicator data were extracted, coded and transferred into SAS/SPSS, and coding manuals prepared.  Basic frequency tables and summaries were prepared.  Draft report prepared. NOTE: All analyses were performed by Nevbahar Ertas, ORISE Fellow, TM

Methodological Approach: Caveats  Information in the Chronicle may not fully represent a state’s comprehensive plan or the related activities of partners or foundations.  States may have subsequently modified the representation of this information within the Chronicle and this has not been tracked. TM

1. How many states are working towards each goal?  States have quickly adapted to using indicators for the Cessation, Initiation and Secondhand Smoke goals.  Preliminary analyses reveal some issues with logic flow. – Example: states may report on short and long-term indicators, but do not show an associated intermediate indicator. TM

1. How many states are working towards each goal? To demonstrate within the SHS Goal Area:  43 states indicate work in Box 4- Creating policies (short-term)  51 states indicate work in Box 7- Reduced exposure (long-term) but only  30 states indicate work in Box 6– Compliance (intermediate)- which is the required causal pathway TM

Eliminating nonsmokers' exposure to secondhand smoke Decreased tobacco- related disparity Reduced tobacco- related morbidity and mortality Rate of consumption and cessation Reduced exposure to Secondhand smoke among adults and young people Increased knowledge, improve attitudes, and increase support for the creation and active enforcement of smoke free policies Creation of smoke free policies Enforcement of smoke free public policies Compliance with smoke free policies Short-termLong-termIntermediate TM

1. How many states are working towards each goal?  The causal pathway showing a state’s intended programmatic outcomes may therefore not be fully represented within the context of their CDC action plan contained in the Chronicle.  Note that short, intermediate and long term classification indicates an evidence-based, casual sequence, rather than time-sequencing. TM

2. How many states use which indicators?  All states (n=51) report key indicators for at least one goal.  Most states (n=44) are using key indicators across all goals.  On average, states report 16 key indicators across all goals, with an average of 5 key indicators for each goal. TM

2. How many states use which indicators? Initiation Secondhand Smoke Cessation Total (For all Goals) AvailUsedAvailUsedAvailUsedAvailUsed Short Inter Long Total Number of indicators used compared to available TM

2. How many states use which indicators?  38% of all indicators reported in the Chronicle are “created” by states (i.e., custom indicators).  Many custom indicators reflect only slight changes to key indicators. – For example, some states apply short-term indicators to intermediate term objectives. TM

2. How many states use which indicators? Use of custom indicators similar to key indicators Initiation Secondhand Smoke CessationTotal 68/166 81/124 99/152248/442 41%65% 56% TM

3. Which indicators are used most often? Indicator # of states # of states Type Youth tobacco use prevalence rate 4145Init LT Smoking prevalence3540Cess LT Quit attempt rate for one day or longer by adults 3140Cess IT Prop. of adults reporting exposure to secondhand smoke in the workplace 2833SHS LT Proportion of youth who report never having tried a cigarette 2736Init LT TM

3. Which indicators are used most often? Indicator # of states # of states Type Number of calls to telephone quitlines 2434Cess ST Quit attempt rate for 1 day or longer by youth 2322Cess IT Proportion that thinks that secondhand smoke is harmful 2018SHS ST Level of support for creating smoke-free policies in public and workplaces 2021SHS ST Proportion of smokers with the intention to quit 2024Cess ST TM

Summary of Findings  While state use of key indicators is increasing, critical gaps exist in some states’ causal pathways.  Many indicators created by states are similar or identical to existing key indicators.  There appears to be confusion over causal sequence of presumed outcomes and temporal outcome achievement. – For example, a state may move an indicator to a different term length to reflect the states assumption that it will take longer for them to achieve the outcome. TM

Recommendations to States  Reflect a complete evidence-based causal pathway for each goal in your state plan.  Use OSH key indicators in the proper causal sequence rather than customizing key indicators to fit a time-based sequence.  Consider working with neighboring or similar states to collaborate on unique or region-specific trainings and to share experiences and lessons learned.  Work with your Project Officer to “clean up” your plan prior to the next Continuation Application. TM

Possible Focus for Further Research  Examine the completeness and strength of state linkages between short, intermediate and long term indicators.  Track state movement towards evidence-based causal pathways using OSH Key Indicators.  Examine which key indicators are not being used and why.  Examine the use of, and data associated with, certain indicators over time as a method of tracking progress made by the NTCP in addressing the four OSH Goal Areas.

Paul Hunting, MPH Centers for Disease Control and Prevention Office on Smoking and Health 4770 Buford Hwy., NE, MS K-50 (mailing) 3005 Chamblee-Tucker Rd., 2nd floor (delivery) Atlanta, GA Ph: Fx: TM State Use of Indicators: Initial Chronicle Findings Use of Outcome Indicators for Planning and Evaluating the NTCP