Strategies for Evaluating Cessation Programs – Improving Response Rates Julie Rainey Marcy Huggins Professional Data Analysts, Inc. Minneapolis PROFESSIONAL DATA ANALYSTS, INC.

Follow-up with participants in tobacco cessation programs Response to our telephone surveys was declining

Literature Review confirms response rates declining overall U of Michigan’s Survey of Consumer attitudes [Curtin, Presser, and Singer 2005] : % responding % responding Declines of 1.5 percentage points per year BRFSS Since 2002, 70% of states showed declines of a median 2.2 percentage points [Link, et al. 2006]

Contributing Factors Survey fatigue Increased use of telephone surveys past 30 years Call screening Voic capability is ubiquitous Caller ID up 10% to nearly 50% from 1995 to 2000 and being used to screen calls [Tuckel, O’Neill 2002] Cell phones Cell phone users: 1995 = 35 million, 2005 = 200 million Cell phone-only households: 12.8% by end of 2006 [Blumberg and Luke, 2007]

Response rates affect the precision of quit rate estimates Why is this a concern for evaluation of tobacco cessation programs?

Two ways to calculate a quit rate: “Completer rate” = # quitters / # completed surveys Includes only those who respond to the survey Studies suggest that people who respond to follow-up surveys are more likely to have quit. “Intent-to-treat rate” = # quitters / # in the survey sample Includes the entire survey sample, or all those whom the program intended to treat. Rate assumes that all non- respondents are still smoking.

Calculating a response rate Response rate = # complete / # attempted Follow-up survey sample: % Response Rate80% Response Rate Non- respondent Complete Non- respondent Complete

Effect of response on quit rate Survey shows: 100 people quit at time of follow-up “Completer rate” 100 / 250 = 40% “Intent-to-treat rate” 100 / 500 = 20% Wide gap

Effect of response on quit rate Survey shows: 100 people quit at time of follow-up “Completer rate” 100 / 400 = 25% “Intent-to-treat rate” 100 / 500 = 20% Narrower gap

Are we underestimating quit rates? A recent study suggests ITT rates overestimate the proportion of non-respondents who are still smoking, at least when there is a high response rate (70%). (Tomson, Bjornstrom, Gilljam and Helgason 2005)

Response rate conclusions Whether you report a “completer” or “ITT” rate, achieving a high response rate improves the precision of the quit rate estimate Reporting the response rate and how it is calculated improves the ability to compare results with other cessation programs

What is a reasonable expectation for response rates? To find out, we conducted an informal survey of tobacco quitlines in the U.S 10 states completed full survey Content Recent follow-up of tobacco users Survey methods Response rates and how they were calculated

Quitline Survey Findings: Great variety: Resources allocated to evaluation Kinds of follow-up being done Methods: # states Follow-up timeframe 6 or 7 months 7 Obtain consent at intake7 Follow-up survey by telephone10 Advance letters attempts to reach participants attempts Survey length 6 Varied

Quitline Survey Findings Range of response rates from PDA surveys 49.5% to 79.0% (n=8 surveys, conducted )

Strategies to Improve Response Rates

Refusals Never Reached Lost to follow-up Main reasons for non-response

Problem: Refusals Participant refusal Household-level refusal Hang-up Terminated mid-interview

Problem: Refusals Refusals are not increasing over time - training can help reduce occurrence PDA interview staff: 4% - 7% refusals Call center vendor: 7% - 10% refusals

Overcoming Refusals Dedicated, trained interviewers Finely honed survey introduction Short (10-15 seconds) Include the name of the program ASAP and the words “quit tobacco” or “stop smoking” Shortest possible preamble to first question Adaptable introduction Allow for conversion of soft refusals

Problem: Never Reached

No answer Busy Voic Never Reached

Solutions: Never Reached 97%90%80% 15 attempts to reach respondents Follow-up timeframe 6 or 7 months

Solutions: Never Reached Rotate attempts across days and time slots Advance or pre-notification letter Letters increase response 6 percentage points [Dillman, Clark and Sinclair, 1995] Letters are cost effective [CDC: BRFSS] Identify envelope and letter with sponsoring organization [Brunner and Carroll, 1969] Timing: mail 3-4 days prior

Disconnected phone / non-working number Wrong numbers Participant has moved Fax number Lost to Follow-up

Problem: Lost to Follow-up Worksite programTelephone program

Problem: Respondents lost to follow-up Solutions: Consider collecting more / alternate contact information at intake Note: simple internet search for updated phone numbers is relatively ineffective, while search services create HIPAA issues

Consider other survey modes Mail: Expect response rates from 58% to 92%, average 74% [Dillman, 2000] PDA’s recent mail survey response rate 84.7% (2006) Mixed-mode example: Recent evaluation of stop-smoking website Online survey of website registrants Non-respondents followed up by telephone Overall response rate 78% (online 42%, phone 36%)

PROFESSIONAL DATA ANALYSTS, INC.