1. Evaluating Internet Based Cessation Programs II: Minnesota’s QUITPLAN.COM sm Intensive Follow-up Study (IFUS) Sharrilyn Evered, Ph.D., Jessie Saul, Ph.D., Annette Kavanaugh, MS, Michael Luxenberg, Ph.D., Nathan Cobb, MD, Randi Lachter, MPH, Lawrence An, MD, Barbara Schillo, Ph.D., and Ann Wendling, MD MPH

2. Evaluation Team Minnesota Partnership for Action Against Tobacco (MPAAT)Minnesota Partnership for Action Against Tobacco (MPAAT) Jessie Saul, PhD, Barbara A. Schillo, PhD, Ann Wendling, MD, Randi Lachter, MPHJessie Saul, PhD, Barbara A. Schillo, PhD, Ann Wendling, MD, Randi Lachter, MPH Professional Data Analysts, Inc (PDA)Professional Data Analysts, Inc (PDA) Sharrilyn Evered, PhD, Michael Luxenberg, PhD, Annette Kavanaugh, MSSharrilyn Evered, PhD, Michael Luxenberg, PhD, Annette Kavanaugh, MS QuitNet.comQuitNet.com Nathan Cobb, MD; Pat Milner, Dave Atkins, JDNathan Cobb, MD; Pat Milner, Dave Atkins, JD ConsultantsConsultants Lawrence An, MD, University of MinnesotaLawrence An, MD, University of Minnesota

3. Background MPAAT launched quitplan.com (fall ‘03)MPAAT launched quitplan.com (fall ‘03) Online six-month follow-up survey obtained low response rate (~10%)Online six-month follow-up survey obtained low response rate (~10%) Quitters are more likely to respond, producing an inflated cessation rateQuitters are more likely to respond, producing an inflated cessation rate Intention-to-treat assumes all non- respondents still smoking - underestimates cessation rateIntention-to-treat assumes all non- respondents still smoking - underestimates cessation rate

4. Low Response Rates Produce Big Gaps between Completer and ITT Standard Follow-Up Survey

5. Evaluation Context Contracted with PDA to design an evaluation that would produce a valid cessation rate (i.e., high response rate)Contracted with PDA to design an evaluation that would produce a valid cessation rate (i.e., high response rate) Targets: 70% consent rate and 70% response rateTargets: 70% consent rate and 70% response rate Barriers: distrust, reconnecting 6 months post-registrationBarriers: distrust, reconnecting 6 months post-registration Adopted “cafeteria” approachAdopted “cafeteria” approach

6. Evaluation Design Mixed mode survey with incentiveMixed mode survey with incentive Obtained consent at registrationObtained consent at registration Mailed “pre-notification” letter, requesting updated contact informationMailed “pre-notification” letter, requesting updated contact information Emailed live link to take survey (two reminders: 3 & 7 days)Emailed live link to take survey (two reminders: 3 & 7 days) After 12 days, phoned non-respondents (up to 25 attempts)After 12 days, phoned non-respondents (up to 25 attempts) Respondents mailed $10 thank you checkRespondents mailed $10 thank you check

7. Online (N=283) 52.8% Phone (N=252) 47.1% Responded (N=535) 78.1% Did Not Respond (N=150) 21.9% Consented (N=685) 60.1% Declined (N=454) 39.9% Invited (N=1,139)* Pre-IFUS (8/1/03 - 2/1/04) (N=3,743) Post-IFUS (4/14/04 - 8/28/04) (N=1,319) IFUS (2/2/04 - 4/13/04) (N=1,294) * Excludes 9 people who did not permit us to use their responses for research purposes and 7 people who were ineligible for the study (under 18 or non-MN resident)

8. High Response Rate Closes the Gap between Completer and ITT IFUS

9. Design Tips Contact info update card: $253 in postage; 19 people updated their info; 18 completed the survey (3.4%)Contact info update card: $253 in postage; 19 people updated their info; 18 completed the survey (3.4%) Calling 25 times: 90% in 10, 99% in 15Calling 25 times: 90% in 10, 99% in 15 SurveyMonkey.com: low cost, controlSurveyMonkey.com: low cost, control

10. Key to Our Success Getting commitment to participate in a scientific study at the outsetGetting commitment to participate in a scientific study at the outset Provide an incentiveProvide an incentive Pre-notification letterPre-notification letter Highly trained and invested interviewersHighly trained and invested interviewers Careful wording of messages is crucialCareful wording of messages is crucial

11. How Well do the IFUS Respondents Generalize to Other quitplan.com Registrants? Two Sets of Comparisons to Assess Bias

12. Groups Compared for Response Bias: “IFUS-era” Online (N=283) 52.8% Phone (N=252) 47.1% Responded (N=535) 78.1% Did Not Respond (N=150) 21.9% Consented (N=685) 60.1% Declined (N=454) 39.9% Invited (N=1,139)

13. First Comparison: “IFUS-era” Logistic regression to predict response status (respondent vs. non-respondent)Logistic regression to predict response status (respondent vs. non-respondent) sex, education level, race/ethnicitysex, education level, race/ethnicity smoking intensity, time to first cigarette, stage of readiness, quit attempt past year, use of quit aids, where heard about websitesmoking intensity, time to first cigarette, stage of readiness, quit attempt past year, use of quit aids, where heard about website website features visited, post- registration loginwebsite features visited, post- registration login

14. Statistically Significant Relationships: “IFUS-Era” The following groups were more likely to be an IFUS respondent:The following groups were more likely to be an IFUS respondent: womenwomen those with some education post HSthose with some education post HS used the nicotine lozenge in the year prior to registrationused the nicotine lozenge in the year prior to registration used more of the site’s features (i.e, set a quit date, posted to a forum)used more of the site’s features (i.e, set a quit date, posted to a forum) logged in after registrationlogged in after registration

15. Online (N=283) 52.8% Phone (N=252) 47.1% Responded (N=535) 78.1% Did Not Respond (N=150) 21.9% Consented (N=685) 60.1% Declined (N=454) 39.9% Invited (N=1,139)* Pre-IFUS (7/31/03 - 2/1/04) (N=3,743) Post-IFUS (4/14/04 - 8/28/04) (N=1,319) IFUS (2/2/04 - 4/13/04) (N=1,294) Groups Compared for Response Bias: “Pre/Post IFUS”

16. Significant Relationships: “Pre/Post IFUS” Overrepresented in IFUS womenwomen more educatedmore educated used lozenge past yearused lozenge past year heard about the site on TV or from healthcare providerheard about the site on TV or from healthcare provider Logged in after registrationLogged in after registration Underrepresented tried to quit at least once in the past year heard about the site from newspaper or magazine

17. Summary of Response Bias Analysis Bivariate analyses revealed seasonal variation can account for some relationshipsBivariate analyses revealed seasonal variation can account for some relationships Differences between IFUS respondents and others (not attributable to seasonal variation): education, gender, site usageDifferences between IFUS respondents and others (not attributable to seasonal variation): education, gender, site usage

18. Great Response Rate and a Representative Sample! But Did We Sacrifice Internal Validity?

19. Online (N=283) 52.8% Phone (N=252) 47.1% Responded (N=535) 78.1% Did Not Respond (N=150) 21.9% Consented (N=685) 60.1% Declined (N=454) 39.9% Invited (N=1,139) Pre-IFUS (8/1/03 - 2/1/04) (N=3,743) Post-IFUS (4/14/04 - 8/28/04) (N=1,319) IFUS (2/2/04 - 4/13/04) (N=1,294) Groups Compared for Mode Bias: “Online vs. Phone”

20. Analysis Compared respondents on:Compared respondents on: missing datamissing data sex, education level, race/ethnicitysex, education level, race/ethnicity smoking intensity, time to first cigarette, stage of readiness, quit attempt past year, use of quit aids, where heard about websitesmoking intensity, time to first cigarette, stage of readiness, quit attempt past year, use of quit aids, where heard about website website features used, post-registration loginwebsite features used, post-registration login cessation and satisfaction outcomescessation and satisfaction outcomes

21. Results: Demographics, Clinical Characteristics, and Site Use Percent of cases with complete data on every variable did not vary by modePercent of cases with complete data on every variable did not vary by mode These groups were overrepresented among online respondentsThese groups were overrepresented among online respondents womenwomen those in action or maintenance stage at registrationthose in action or maintenance stage at registration those who used the site morethose who used the site more

22. Results: Cessation Outcomes

23. Results: Attitude Items

24. Mode Bias: Summary Cessation outcomes do not vary by modeCessation outcomes do not vary by mode Some attitude items vary, but not in a consistent directionSome attitude items vary, but not in a consistent direction Mode confounded with an early response, so results are difficult to interpretMode confounded with an early response, so results are difficult to interpret

25. Conclusions Mode bias suggests attitude questions may be less reliable with this designMode bias suggests attitude questions may be less reliable with this design Mixed mode design with incentive can produce a high response rateMixed mode design with incentive can produce a high response rate High response rate generates a more representative sample and closes the gap between completer and ITT ratesHigh response rate generates a more representative sample and closes the gap between completer and ITT rates Web-based cessation programs can be meaningfully evaluatedWeb-based cessation programs can be meaningfully evaluated

26. For more information… About the design, contact me at:About the design, contact me at:sharrilyn_r_evered@bluecrossmn.com651-662-9383 About the results, contact Jessie Saul at:About the results, contact Jessie Saul at:jsaul@mpaat.org952-767-1415