1. 1 Protocol & preliminary results from a transnational cognitive interviewing study: Ensuring valid cross-cultural comparisons INSTITUTO NACIONAL DE SALUD PÚBLICA MÉXICO James F. Thrasher, Anne C. K. Quah, Ron Borland, Rahmat Awang, Maizurah Omar, Buppha Sirirassamee, Marcelo Boado, Ashlee Watts, Ana Dorantes Alonso TTURC Planning Meeting, Portland, OR, USA March 2, 2008

2. 2 Purpose of the study Develop a protocol for assessing and reducing systematic response error in multi- lingual and multi-cultural surveysDevelop a protocol for assessing and reducing systematic response error in multi- lingual and multi-cultural surveys Focus on existing ITC survey questions:Focus on existing ITC survey questions: –Assess equivalence of comprehension and response error across 6 countries, 4 languages; –Recommend alterations to questions to increase equivalence in later studies. Increase data quality  more confidence in results from comparative data analysesIncrease data quality  more confidence in results from comparative data analyses

3. 3 Questionnaire protocol 1.Ask participant survey question. 2.Passive recording of behaviors that indicate comprehension problems (i.e., behavioral coding*). 3.Ask follow-up questions that aim to assess response error across the steps involved in cognitive processing of survey questions (i.e., cognitive interviewing). *Johnson, TP et al (2006). Cultural variability in the comprehension of health survey questions. Annals of Epidemiology, 16, 661-668.

4. 4 Study Steps Identify candidate questions Determine likely biases Develop probes to assess biases Develop translator guide Translate protocol Train interviewers Pilot protocol Review pilot data Adjust protocol Implement protocol Enter data Translate responses to English Review quantitative data Identify themes in qualitative data Determine evidence for systematic bias Suggest changes

5. 5 Sample Characteristics* USAAustraliaUruguayMexicoMalaysiaThailand Male 65%50% 65%100%80% Age 36 40383139 <HS 5%30% 20%10%85% EduHS 65%30%40%55%45%10% Uni 30%40%30%25%30%5% Daily smoker 85%90%80%85%90%85% Average cigs/day 2315141513 *n=20 for each country

6. 6 Nicotine in cigarettes is the chemical that causes most of the cancer: False

7. 7 Behavioral codes indicate some systematic bias Did the participant… USAUSTURGYMEXMALTHAIp …need you to repeat the question? 110011 1.00 …have difficulty with response options? 530030 0.008 …ask for clarification or qualify answer? 410100 0.047 Total105 0141

8. 8 Can you tell me what you know about nicotine? ThemeUSAAUSTURGYMEXMALTHAI Addictive 80%60%20% 15%35% Harmful 15% 30% 35%30% Chemical-drug 0%30%0%15%20% Stimulant 5% 0%10%0% Discoloration 10%5%10%0%5% Don’t Know 0%15%10%15%5%20% Haven’t heard of nicotine 0% 20% 5%25%

9. 9 Conclusions & Recommendation Behavioral codes  systematic bias –Awkward wording? Acquiescence bias? –“don’t know” or haven’t heard of nicotine Order effect after health outcome questions? Recommendation: Clarify question intent: –Assess knowledge? Why does this matter? Could first ask if they have heard of nicotine –ITC-SEA W3: Cigarettes contain nicotine

10. 10 Tobacco is addictive: “agree” or “strongly agree”

11. 11 Behavioral Codes indicate no problem with “Tobacco is addictive” Did the participant… USAAUSTURGYMEXMALTHAI p …need you to repeat the question? 000001 1.00 …have difficulty with response options? 110011 0.87 …ask for clarification or qualify answer? 100000 1.00 Total 210012

12. 12 Other things that are addictive CategoryUSAAUSTURGMEXMALTHAI Alcohol 70% 50%15% Other drinks 5% 25% 10%0% Illegal drugs 80%70% 55%60%80% Legal drugs 35% 0%10%25%0% Inhalants 0% 5%20%15% Food 40%20%25%5%0% Sex 15%10%0%20%5%0% Gambling 5%15%10%5%0% Other 35%15% 0%20%5% DK 5%0%5%25%15%0%

13. 13 What does it mean to say that something is addictive? ThemesUSAAUSTURGMEXMALTHAI Control-General 50% 55% 45%30% Physiological 40%25%5%15%0% Control-Psych 10% 0% 30%0% Control-Quit 0%10%5%10% 15% Freq-Quantity 5%50%40%15%20%30% Danger 5%0% 35% Pleasure 5%0%5%0%5% It's got a grip of you and you can't let go. It's a must have, you've gotta have it. 39 year old Australian male The body gets tolerance for it and the body won't function normally with out it. 27 year old US female If he doesn’t get that thing, he cannot do other things…His mind is always imagining that thing…if he doesn’t get it, he cannot sit still. 26 year old Malaysian male

14. 14 Conclusions & Recommendations Behavioral coding  no clear bias Connotations of and associations with addiction appear to differ across countries Recommendation: Ask question(s) that focus specifically on: –Control over smoking (e.g., Smokers cannot go long without smoking, even when they don’t want to smoke) –Control over quitting (e.g., Completely stopping smoking is extremely difficult)

15. 15 Lights are less harmful than regular cigarettes: “Disagree”/”Strongly disagree”

16. 16 Behavioral codes indicate no systematic bias Did the participant… USAAUSTURGYMEXMALTHAI p …need you to repeat the question? 210123.377 …have difficulty with response options? 110123.343 …ask for clarification or qualify answer? 010021.279 Total31 0267

17. 17 Other differences beween light and regular cigarettes? themesUSAAUSTURGYMEXMALTHAI Nicotine & Tar 65%60%5% 15% 10% Other Chemicals 30% 15%5%15% Filter & Tobacco Qty 35%5% 0%5%10% Taste & Flavor 20%15%50%30%50%60% Other sensory effects 10%5%0%5%35%50% other (danger, packaging) 10%15%20%0%15%0%

18. 18 Conclusions & Recommendations Behavioral coding  no systematic bias Perceptions of light vs. regular appear to differ across countries –Focus on product characteristics in western countries –Focus on sensory effects of product use in non- western countries Recommendation Be aware of the possibility of the salience of these characteristics when developing new questions about product risks and effects.

19. 19 Noticed warning labels “often” or “very often” in last month

20. 20 Behavioral codes indicate some evidence of systematic bias Did the participant…USAAUSTURGYMEXMALTHAIp …need you to repeat the question? 003111.377 …have difficulty with response options? 310110.377 …ask for clarification or qualify answer? 440100 0.011 Total 31 0267

21. 21 Follow up probes indicate incorrect reference period and issues with the term “notice” USAAUSTURGYMEXMALTHAIP Correct reference period identified (last month) 15%25%15%10%5%.437 Can you notice without looking closely? 55%80%70%65%35%75%.049 Can you notice if you know what label says? 75% 55%90%60%80% 0.146

22. 22 Conclusions & Recommendations Behavioral coding  indicates systematic bias (*problems with US & Australia) Consistently incorrect identification of reference period for the question Some indications of systematic differences in meanings of “notice” Recommendation: –Ask about last time when “noticed” –Review terms used for translation of “notice”

23. 23 In last 6 months, noticed ads on… MeasureUSAAUSTURGMEXMALTHAIALLp* TV15%5%25%35%30%5%19% 0.054 Dif to remember25%10%50%15%60%40%33%0.003 Radio15%0%25%20%10%0%12% 0.041 Dif to remember30%5%45%25%65%50%37%<.001 Papers/Mags75%0%40%50%30%10%34% <.001 Dif to remember25%10%30%15%45%40%28%0.095 Billboards60%0%60%95%35%10%43% <.001 Dif to remember20%25%45%35%50%40%36%0.332 In stores95%45%70%90%70%30%67% <.001 Dif to remember15%35%30%20%25%35%27%0.660 Correct period (last 6 months) 0%20%30%10%5%15%5% 0.067

24. 24 Conclusions & Recommendations Behavioral coding  no systematic bias Across channels, participants in all countries found it difficult to remember whether they had seen an ad –Some indication of systematic bias in difficulty of remembering exposure for TV and radio. Consistently incorrect identification of reference period for noticing Recommendation: –Ask about last time “saw”, with options for “never” and for “more than 6 months ago”

25. 25 General Conclusions Cognitive interviewing and behavioral coding techniques can be implemented across linguistic and cultural settings Overcoming challenges of assessing equivalence across languages demands collaboration from multilingual partners. Comprehension of some questions appeared generally comparable whereas other questions appear to generate response error and need to be adjusted.

26. 26 General Recommendations Any changes to harmonize item comprehension should be verified through another round of cognitive interviews. Interviewers should be better trained to clarify ambiguous or repetitive responses.

27. 27 General Recommendations Prepare translation notes (i.e., definitions of the concept being measured & question intent) for new questions. Choose source language questions that use simple, translatable language –“Notice” vs. “saw” To avoid augmenting “translation loss”, questions should originate from one source. –NOT: English  Spanish  Malay Equivalence of comprehension may demand flexibility in “source” language item

28. 28 General Recommendations Systematic documentation of the evolution, translation, and meanings of ITC survey questions, key terms, and phrasing across countries.  Translator’s notes  May include possible translation equivalents in the target languages  Provides record of original survey item wording  Allows future researchers to understand intentions behind survey items and any changes to them

29. International Tobacco Control Policy Evaluation Project http: / / www.itcproject.org http: / / www.roswelltturc.org ITC Project Research Support Core support provided by the U.S. National Cancer Institute to the Roswell Park TTURC (P50 CA111236) Additional major funding provided by the Canadian Institutes of Health Research

30. 30 Study Steps Identify candidate questions Determine likely biases Develop probes to assess biases Develop translator guide Translate protocol Train interviewers Pilot protocol Review pilot data Adjust protocol Implement protocol Enter data Translate responses to English Review quantitative data Identify themes in qualitative data Determine evidence for systematic bias Suggest changes