1. 1 Using Qualitative Methods to Develop Culturally Adequate Surveys Anna Nápoles-Springer, Ph.D. University of California San Francisco Center for Aging in Diverse Communities

2. 2 Overview u Integrating quantitative and qualitative methods to develop surveys u Focus groups u Cognitive interview methods u Using results of cognitive interviews to make decisions about item revisions/deletions

3. 3 Integrating Quantitative and Qualitative Methods u In diverse populations, qualitative work is necessary in addition to more traditional quantitative studies u Prior to quantitative: to develop concepts, items appropriate to culture u After quantitative: –to help identify reasons for items not performing well quantitatively –to explore possible explanations for unexpected results

4. 4 What are Qualitative Methods? u Data consist of words, not numbers u Richly descriptive, open-ended u Focus on inductive analytic approaches u Many types: ethnography, participant- observation, direct observation, interviews, focus groups

5. 5 When are Qualitative Methods Useful? u To understand the meaning of participants’ events, situations, and actions u To understand contextual influences on participants’ actions u To identify unanticipated phenomena and influences (e.g., exploratory studies to design questionnaires and identify variables for study) u To understand the processes underlying observed relationships between variables

6. 6 Combining Qualitative and Quantitative Methods u Qualitative Methods to Develop Quantitative Tools u Qualitative Methods to Explain Quantitative Results Qualitative Results Quantitative Results Qualitative

7. 7 Combining Qualitative and Quantitative Methods cont. u Quantitative Methods to Expand on Qualitative Study u Quantitative and Qualitative Methods Equal QualitativeResults Quantitative Qualitative Results Quantitative (Tasjakkori A. Sage Publications, Inc. 1998)

8. 8 Focus Groups can Identify Right Questions to Ask in Expanded Surveys u 1996 US Census Bureau –little difference between rates of work force participation of low-income families of children with special needs and those with no special needs u Might conclude child care systems adequately meeting needs of special needs families-able to work u Focus groups-enormous strain, scarce programs, working nights, losing multiple jobs (Sunhua L. 2002)

9. 9 From Focus Groups to Intervention The Subjective Culture of Smoking u Purposeful sampling stratified by comparison groups (by ethnicity) u Examine similarities and differences in attitudes, beliefs and behaviors (Triandis) u Antecedents and consequences of smoking

10. 10 Focus Groups to Identify Themes Themes Extracted from Focus Groups u Family concerns - 2nd hand smoke u Heightened health issues u Importance of appearance and interpersonal relations - simpatía  Habitual use less important u Addiction less of a concern

11. 11 Integrate Results to Develop Intervention: Programa Latino Para Dejar de Fumar u Guia Para dejar de Fumar--self-help guide u Electronic media in Spanish u Emphasis on family and collective orientation of culture u Immediate effects of smoking u Health effects in different contexts

12. 12 Develop Structured Questionnaire Scales developed based on themes/lit/theory u 17 antecedents to smoking (habitual activities, social activities, emotional states) u 15 reasons to quit (family, appearance, health) u 3 reasons to keep smoking (weight, nervousness, concentration)

13. 13 Triangulation of Focus Group Themes: Latinos’ Reasons to Quit OR95% CI Criticized by family1.93(1.26, 2.98) Burn clothes1.57(1.02, 2.42) Children's' health1.67(1.08, 2.57) Bad breath2.07(1.40, 3.06) Family pressure1.69(1.10, 2.60) Good example1.83(1.21, 2.76) Not to gain weight0.38(0.24, 0.59)

14. 14 Integrating Qualitative and Quantitative Methods to Develop Surveys u Goal: develop survey items that mean the same thing across groups u Quantitative methods (descriptive statistics, reliability, validity, missing data) identify ethnic differences in response patterns u Qualitative methods uncover cognitive processes respondents use to answer questions; disclose reasons for ethnic difference in response patterns u Iterative quantitative and qualitative research methods

15. 15 Integrating Qualitative and Quantitative Methods to Develop Surveys u Example: –Item: In the past 12 months, how many times have you been to see a doctor? –Test-retest reliability was.56-.65 for Chinese and Vietnamese compared to.90 in Whites –Interviewer debriefing revealed problems with interpretation of what constituted a visit to a doctor among Chinese and Vietnamese Pasick RJ, et al. 2001 Public Health Rep;116:223-43

16. 16 Open-ended Interviews to Develop Structured Questionnaire Example: Constructing attitudinal scales u Use open-ended interviews (individual or focus groups) to obtain statements on topic of interest to identify relevant concepts u Organize statements into “item pool” u Pretest closed-ended items u Modify items based on pretest and administer in structured interview

17. 17 Goals of Purposeful Sampling u Achieve representative, typical settings, individuals, or activities u Adequately capture heterogeneity, sample for broadest range of variation rather than typical individuals u Set up contrasts to examine differences between settings or individuals (Maxwell JA. Sage Publications, Inc., 1996)

18. 18 Sampling in Qualitative Research u Quantitative research-2 types of sampling: –Probability –Convenience u Qualitative research-3 rd type of sampling: –Purposeful »Deliberately select settings, persons or events to best answer research questions

19. 19 Qualitative Methods to Develop Surveys u Focus Groups –Open-ended guided group discussion with probing of responses u Cognitive interviews –Open-ended probes to assess how items are interpreted and adequacy of response choices –Respondent is expert u Expert panels provide input on relevant concepts u Interviewer debriefing

20. 20 Focus Groups u Group discussion led by experienced moderator usually 1.5 - 2 hours u Purposeful sampling of 6-10 homogenous participants per group u Use open-ended questions with follow-up probes for clarification, usually covering 10-12 topics u Participants stimulate comments of others u Audio-record and transcribe discussion

21. 21 Focus Groups-Logistics u Moderator skills: listening, communication, negotiation, cultural similarity to participants u Costs of group: $600 - $1000 per group (incentives, audio-taping, transcription, translation, food) u Convenient and hospitable community setting u In-person recruitment with telephone and mail follow-up

22. 22 Example: The Interpersonal Processes of Care u Examined similarities and differences in attitudes, beliefs and behaviors u Sampling stratified by comparison groups: Whites, African Americans and Latinos u Probed on aspects of communication, decision making and interpersonal style of physician u Used information to design structured questionnaire

23. 23 Identify Themes: How Interpersonal Processes of Care Might Affect Health Outcomes u Poor communication - doctors... –Use unclear language, jargon –don’t listen carefully –don’t make sure that the patient understands u Decision-making did not involve patients u Cultural insensitivity –patients’ beliefs, preferences not taken into account

24. 24 Identify Domains of Communication u General clarity u Elicitation of and responsiveness to patient problems and concerns u Explanations and information –about condition, prognosis –about processes of care –about what to do for themselves

25. 25 Define Domains: Definition of General Clarity in Communication u Communicates clearly even when patient does not speak English well –Uses interpreters if needed u Easy to understand u Speaks clearly and slowly enough u Explains things well u Makes sure patients understand

26. 26 Develop Items Based on Definition: General Clarity Over the past 12 months, how often did doctors… use words that were hard to understand speak too fast explain things well asked if you understood what they told you

27. 27 Develop Item Pool and Pretest u Develop 5 – 6 items per domain or content area of interest u Will end up with 3 – 4 that work u Use pre-testing to identify best candidate items for structured survey u Types of pretests –General debriefing on comprehension, flow –Cognitive interviews

28. 28 Cognitive Interviews u Derived from social and cognitive psychology to explore processes respondents use to answer survey questions u Diagnostic tool for pretesting survey questions

29. 29 Cognitive Interviews Examine 4 Steps in Answering Questions u Interpret and understand the question –as intended by the researchers u Retrieve the information –various schemas used to access memory u Judgment formation - formulate an answer –calculate or judge the correct information u Edit response - decide what to report –is answer embarrassing, socially undesirable?

30. 30 Purpose of Cognitive Interviews To learn.. u.. if respondents understand words and phrases as intended (meaning) u.. about the process of answering the questions u.. whether items are unacceptable u.. about the usefulness of response choices –whether response choices are adequate –how they use the response choices

31. 31 Two Types of Cognitive Interviews u Think aloud interviews –Respondent asked to think aloud as they answer question u Probe interviews –Interviewer asks specific questions to elicit how respondent answered question u Think alouds - greater respondent burden

32. 32 Steps Involved in Cognitive Interview Pretesting u Decide on a final item pool u Develop interview “probes” for a subset of items u Translate interview and probe questions u Recruit sample for cognitive interviews u Conduct cognitive interviews u Analyze results u Revise items based on results

33. 33 Reducing Item Pool u Reduce items from item pool (subjective process among research team) u Criteria: –maintain breadth of concept, multiple items/concept –reduce redundancy (but OK to test alternate versions of items) –eliminate items that are unclear, complex, require high verbal skills, lack face validity, or will not translate well

34. 34 Writing Probe Questions u From reduced set of items, select potentially problematic items for pretesting u Write open-ended probe questions –worded to reveal if suspected problem with item is present

35. 35 Types of Cognitive Probes u General –Tell me what you were thinking when you answered that question –How easy or difficult was it to answer that question? Why? u Explore meaning of word or phrase –I asked you how often doctors take a genuine interest in you. What does the phrase “genuine interest” mean to you?

36. 36 Types of Cognitive Probes (cont.) u Retrieval –How did you remember that? u Judgment –Why did you pick that number for your answer? u Response –Do you think that most people answer this question honestly? Collins D. Quality of Life Research 2003. 12:229-38.

37. 37 Types of Cognitive Probes (cont.) u Redundancy –How is the phrase “give you advice about your diet and exercise” different from the phrase “talk to you about your diet and exercise”? u Acceptability –When I asked you how often you felt discriminated against by doctors because of your race or ethnicity, you answered (read answer given). Were you offended by this question?

38. 38 Types of Cognitive Probes (cont.) u Cultural appropriateness –I asked you how often doctors asked you about your health beliefs? What does the term ‘health beliefs’ mean to you?

39. 39 Sampling for Cognitive Interviews u Aim is not representative samples u Include respondents from major segments of population to be sampled for main survey u Approximately 5-30 interviews/group

40. 40 Recruiting for Cognitive Interviews u Explain how their help fits into larger study, process of creating questions u Explain their role clearly: –“help us learn how to ask better questions” –“help us make questions clearer for others” –“help us to identify problems with questions” u Pay subjects – interview is demanding u If survey is long, pretest different sections on different subjects

41. 41 Conducting Cognitive Interviews u Individual face-to-face, in-depth interviews u Standard administration of closed-ended items u Administer probe questions at the end u Typically 1 to 1.5 hr interview u Each interview audiotaped and transcribed

42. 42 Analysis of Cognitive Interviews u Data preparation –Data are transcripts –Translate transcripts if necessary back to English u Two approaches –Behavioral coding –Use qualitative data analysis software to perform content analysis

43. 43 Behavioral Coding u Review transcripts to identify problems with items –respondent and interviewer “problem” behaviors u Assign “problem behavior” codes to each item using pre-determined categories u Summarize results for each item: –proportion of interviews with each problematic behavior for each item »e.g., 7/48 respondents requested clarification of item 10

44. 44 Examples of Behavioral Codes Interviewer behavior u Hard to read - interviewer experiences difficulty reading question Respondent behavior u Repeat question - respondent asks to have the question repeated

45. 45 Analysis of Cognitive Interviews (cont.) u Using qualitative analysis software, review all dialogue that ensued during standard administration of closed-ended items and open- ended probes –can reveal source of problems –can help in deciding whether to keep, modify or drop items u Allows you to examine dialogue for each item –within groups –across groups/languages

46. 46 Example of Results: Interpreting the Question Original item: How satisfied are you with the amount of stress or worries in your life? Probe: Did you answer this question in terms of stress, worries, or both? u AA - tended to answer in terms of worries u No term in Spanish for “stress” u Item revised –How satisfied are you with the amount of worries in your life? Warnecke RB, etal. JNCI Monographs No. 20, 1996; 29-38.

47. 47 Example of Results: Unclear Phrase Original item: Have you had any medical tests or procedures in past year? –26% of respondents asked for clarification Probe: What did you include as medical tests or procedures? u Medical test or procedures unclear (e.g., asked if it included dental or cosmetic procedures) u Item revised to include examples: –Have you had any medical tests or procedures, such as blood tests, x-rays, or cancer screening tests?

48. 48 Example of Results: Information Retrieval Original item: How satisfied are you with your ability to travel on vacations? Probe: What do you think we meant by vacation? u Neither concept of “vacation” nor “travel for pleasure” had relevance to lifestyle of AA and MA; travel for family reasons u Dropped item Warnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38.

49. 49 Example of Results: Response Sets Original scale: “Very unimportant to Very Important” on 0-100 scale (bipolar) u Problem: No direct Spanish translation for “unimportant” –bilinguals understood translation “sin importancia” as equivalent to unimportant –Spanish monolinguals did not understand it as the negative pole of the scale u Solution: Changed English to unipolar scale “not at all important” to “very important” to correspond to best Spanish Warnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38

50. 50 Example of Results: Redundancy of Items Original items: –How often did doctors explain what was causing your health problem? –How often did doctors explain your diagnosis? Probe: What do the words health problem and diagnosis mean to you? u Respondents viewed them as the same u Some respondents did not know meaning of diagnosis u Dropped item with word diagnosis

51. 51 Summary u Greatest problems are with question interpretation –usually due to need to write at lower level of verbal comprehension u Sometimes English concepts not meaningful in other languages or are irrelevant for certain groups u Need to pretest response sets –Ethnic groups may use them differently

52. 52 Advantages of Cognitive Interviews u Complement other survey field test methods (e.g. where problems are identified by missing or truncated answers) u Identifies where responses might be affected by cultural or group experiences u Suggests ways to revise items, response sets u Improves validity of questions

53. 53 Disadvantages of Cognitive Interviews u Flags problems, but significance of the problem remains matter of subjective judgment –at which point need to revise or drop items u Based on small number of respondents u Time and labor intensive

54. 54 Conclusions:Integrating Quantitative and Qualitative Methods u Focus groups, cognitive interviews complement and are necessary component of research in diverse groups u To identify relevant themes, constructs, items u In context of developing theory (research questions), structured surveys, interventions