1. Strategies for Evaluating Cessation Programs – Improving Response Rates Julie Rainey Marcy Huggins Professional Data Analysts, Inc. Minneapolis PROFESSIONAL DATA ANALYSTS, INC.

2. Follow-up with participants in tobacco cessation programs Response to our telephone surveys was declining

3. Literature Review confirms response rates declining overall U of Michigan’s Survey of Consumer attitudes [Curtin, Presser, and Singer 2005] : 1979 72% responding 2003 48% responding Declines of 1.5 percentage points per year 1996-2003 BRFSS Since 2002, 70% of states showed declines of a median 2.2 percentage points [Link, et al. 2006]

4. Contributing Factors Survey fatigue Increased use of telephone surveys past 30 years Call screening Voicemail capability is ubiquitous Caller ID up 10% to nearly 50% from 1995 to 2000 and being used to screen calls [Tuckel, O’Neill 2002] Cell phones Cell phone users: 1995 = 35 million, 2005 = 200 million Cell phone-only households: 12.8% by end of 2006 [Blumberg and Luke, 2007]

5. Response rates affect the precision of quit rate estimates Why is this a concern for evaluation of tobacco cessation programs?

6. Two ways to calculate a quit rate: “Completer rate” = # quitters / # completed surveys Includes only those who respond to the survey Studies suggest that people who respond to follow-up surveys are more likely to have quit. “Intent-to-treat rate” = # quitters / # in the survey sample Includes the entire survey sample, or all those whom the program intended to treat. Rate assumes that all non- respondents are still smoking.

7. Calculating a response rate Response rate = # complete / # attempted Follow-up survey sample: 500 50% Response Rate80% Response Rate Non- respondent Complete Non- respondent Complete

8. Effect of response on quit rate Survey shows: 100 people quit at time of follow-up “Completer rate” 100 / 250 = 40% “Intent-to-treat rate” 100 / 500 = 20% Wide gap

9. Effect of response on quit rate Survey shows: 100 people quit at time of follow-up “Completer rate” 100 / 400 = 25% “Intent-to-treat rate” 100 / 500 = 20% Narrower gap

10. Are we underestimating quit rates? A recent study suggests ITT rates overestimate the proportion of non-respondents who are still smoking, at least when there is a high response rate (70%). (Tomson, Bjornstrom, Gilljam and Helgason 2005)

11. Response rate conclusions Whether you report a “completer” or “ITT” rate, achieving a high response rate improves the precision of the quit rate estimate Reporting the response rate and how it is calculated improves the ability to compare results with other cessation programs

12. What is a reasonable expectation for response rates? To find out, we conducted an informal survey of tobacco quitlines in the U.S 10 states completed full survey Content Recent follow-up of tobacco users Survey methods Response rates and how they were calculated

13. Quitline Survey Findings: Great variety: Resources allocated to evaluation Kinds of follow-up being done Methods: # states Follow-up timeframe 6 or 7 months 7 Obtain consent at intake7 Follow-up survey by telephone10 Advance letters2 6-10 attempts to reach participants 3 11-15 attempts Survey length 6 Varied

14. Quitline Survey Findings Range of response rates from PDA surveys 49.5% to 79.0% (n=8 surveys, conducted 2004-2007)

15. Strategies to Improve Response Rates

16. Refusals Never Reached Lost to follow-up Main reasons for non-response

17. Problem: Refusals Participant refusal Household-level refusal Hang-up Terminated mid-interview

18. Problem: Refusals Refusals are not increasing over time - training can help reduce occurrence PDA interview staff: 4% - 7% refusals Call center vendor: 7% - 10% refusals

19. Overcoming Refusals Dedicated, trained interviewers Finely honed survey introduction Short (10-15 seconds) Include the name of the program ASAP and the words “quit tobacco” or “stop smoking” Shortest possible preamble to first question Adaptable introduction Allow for conversion of soft refusals

20. Problem: Never Reached

21. No answer Busy Voicemail Never Reached

22. Solutions: Never Reached 97%90%80% 15 attempts to reach respondents Follow-up timeframe 6 or 7 months

23. Solutions: Never Reached Rotate attempts across days and time slots Advance or pre-notification letter Letters increase response 6 percentage points [Dillman, Clark and Sinclair, 1995] Letters are cost effective [CDC: BRFSS] Identify envelope and letter with sponsoring organization [Brunner and Carroll, 1969] Timing: mail 3-4 days prior

24. Disconnected phone / non-working number Wrong numbers Participant has moved Fax number Lost to Follow-up

25. Problem: Lost to Follow-up Worksite programTelephone program

26. Problem: Respondents lost to follow-up Solutions: Consider collecting more / alternate contact information at intake Note: simple internet search for updated phone numbers is relatively ineffective, while search services create HIPAA issues

27. Consider other survey modes Mail: Expect response rates from 58% to 92%, average 74% [Dillman, 2000] PDA’s recent mail survey response rate 84.7% (2006) Mixed-mode example: Recent evaluation of stop-smoking website Online survey of website registrants Non-respondents followed up by telephone Overall response rate 78% (online 42%, phone 36%)

28. PROFESSIONAL DATA ANALYSTS, INC.