1. APHA Annual Meeting, Washington D.C., November 5, 2007
Improving Evaluation Data Quality by Addressing Nuances in Commonly-Used Tobacco Prevention and Cessation Indicators
Molly Aldridge, MPH1, Laura McCormick, Dr PH2, Carol Ripley-Moffitt, M.Div.1, Kathryn Kramer, PhD1, and Adam Goldstein, MD1
1School of Medicine, University of North Carolina at Chapel Hill, Department of Family Medicine; 2 NC Health and Wellness Trust Fund
APHA Annual Meeting, Washington D.C., November 5, 2007

2. Background State tobacco control programs Data collection systems
Data on reliability
Over/underestimates
Comparisons across states

3. NC Statewide Tobacco Control Program
North Carolina Health and Wellness Trust Fund
Teen Tobacco Use Prevention and Cessation Initiative
46 grantees
Four major goal areas
Initiation
Secondhand smoke exposure reduction
Cessation
Eliminating tobacco-related health disparities
Independently evaluated by UNC
The Health and Wellness Trust Fund (HWTF) began funding the Teen Tobacco Use Prevention and Cessation Initiative in as the first statewide program in North Carolina to reduce and prevent tobacco use among youth. The Initiative now includes support for 46 local coalitions and statewide organizations that work in several areas of tobacco control including tobacco prevention education, youth empowerment, tobacco-free schools policy, reduction of youth exposure to secondhand smoke, reduction of health disparities, and youth smoking cessation. The University of North Carolina School of Medicine independently evaluates the initiative.

4. Evaluation
Evaluation of outputs
Uses indicators based on logic models based on best practices
OUTCOMES
RESOURCES
ACTIVI
OUTPUTS
Short-Term
Intermediate
Long-term
Funding from HWTF
Community/ School and Priority Population grantees
Special Project
Grantees
(e.g., SAVE)
HWTF Media Vendor
NC Tobacco Quitline
Local and statewide media campaign
Promote access to Quitline and other cessation services among youth and health providers
Activities focus on youth empowerment, community mobilization, and targeting youth populations with tobacco-related disparities
Completed activities to disseminate information about youth cessation services
Increased calls and fax referrals to Quitline
Completed activities to promote adoption of cessation programs in schools
Increased youth access to cessation services
Increased successful quit attempts among youth
Increased cessation among youth
Reduced tobacco-use prevalence and consumption among youth
Reduced tobacco-related morbidity and mortality among youth
Decreased tobacco-related disparities among youth
Technical Assistance Providers
Community Partners
ACTIVITIES
A brief discussion of our evaluation to put things into perspective. A large part of our evaluation strategy uses indicators that are based on logice models that we created for the initiative and of course, those logic models are based on best practices as laid out by the CDC. Here’s the logic model for the cessation goal area of the Teen Initiative.

5. Evaluation – data collection
Grantees provide monthly reports on indicator and non-indicator activities using the indicator Progress Tracking System (iPTS)
28 indicators across 4 goal areas

6. Data flow
Grantee
UNC SOM TPEP Reviewer 1
UNC SOM TPEP Reviewer 2
HWTF

7. Example of a cleaned indicator change

9. Evaluation – Data Cleaning
Initially, cleaned data based on brief operational definitions and internal decision rules
Noticed a high level of error (2,218 errors)
Goal: Improve data quality
Create detailed definitions addressing as many nuances as possible
Create a codebook
In general, the cleaning was based on some brief definitions for each indicator and a lot of decision rules that were made in-house and not always communicated specifically to grantees. So, over time, we noticed a high level of error which ultimately resulted in a lot of time and resources spent cleaning data. So, we decided that our goal was to reduce the error rates by creating very detailed indicator definitions that would address as many nuances of each indicator as we could possible imagine. And then to document those decision rules and definitions in a codebook that was disseminated to all grantees.

10. Addressing Nuances
Based on experience reading monthly reports from community-level grantees
Must address
What qualifies as an indicator unit change
How to quantify the indicator unit change
So let’s talk a little more about what I mean by nuances of the indicators. During the cleaning process, we noted several measures whose definitions became unclear when applied to the local activities of the grantees. For example, we could have asked for the number of meetings with elected officials to discuss a certain policy. But one of the grantees works in an American Indian tribe, there they only meet with tribal leaders. These are not technically elected officials, but do we count this or not? There were dozens of these types of decision rules to make. And it became clear that we needed to address not only what types of activities or outcomes should count as an indicator change, but also how each item should be quantified. Hopefully, that will make more sense when we take a look at the next slide together.

11. Example 1 Refining the count Refining the definition Indicator
Do all media messages count the same?
Do posters made by a youth group count the same as a radio spot?
Should a flier distributed in multiple locations be counted once or once for each place where it was distributed?
If mass s are counted, should they be reported as one unit for the or one unit for each to which it was sent?
If one media piece contains messages addressing two focus areas within the initiative, should it be counted as one unit or two?
Indicator
What counts as a media message?
Should promotional items be counted?
Should fliers or student-made posters be counted?
Should mass s be counted?
# of youth cessation media/ promotional messages published or aired

12. Example 2 Refining the definition Refining the count Indicator
What counts as a meeting?
Should a telephone call be counted?
Should an be counted?
Should the definition be limited to include only face-to-face meetings?
Is there a minimum time required to be counted as a meeting or would a “stop in the hall” count?
If one meeting is held with representatives from three different high schools, should it be counted as one unit or three?
If a meeting spans more than one day (e.g. a two-day retreat), should it be counted as one unit for the whole event or one unit per day?
# of presentations or meetings promoting Quitline, NOT, 5A, counseling for youth or Spit Tobacco cessation

13. So, we tried to nail down all the answers to those questions for each indicator and document them in a codebook as shown here. The codebook is available to the grantees via and it is also incorporated into their reporting system.

14. Reflections Data quality improved
Recognized potential benefit of discussions with other states
Reduce time and resources spent by each state individually
Allow for national evaluation
After one full year of using these more detailed definitions and the codebook, we noticed a 67% decrease in error rates. Now, this may seem intuitive. Of course, if people have more instruction, they will be able to come closer to that which they are being expected to do. But, the implications of this for statewide tobacco control programs is great.

15. Recommendation
Greater standardization of a minimum data set across statewide programs
Based on best practices and broad experience of community-level grantees
Widespread dissemination

16. Questions?

17. Molly L. Aldridge, Research Associate UNC School of Medicine
Tobacco Prevention & Evaluation Program
(919)
For more info about TPEP and this evaluation: