1. 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for Aging in Diverse Communities

2. 2 Overview u Integrating quantitative and qualitative methods to develop surveys u Focus groups u Cognitive interview methods u Using results of cognitive interviews to make decisions about items

3. 3 When are Qualitative Methods Used? u Open-ended interviews typically used in new areas of study u Provide in-depth knowledge and testable insights about health issues u Especially critical in cross-cultural studies due to dearth of information

4. 4 Nature of Qualitative Data u Exploratory, descriptive u Identify unanticipated phenomena and influences u Generate hypotheses about processes underlying observed relationships u Many types: ethnography, direct observation, focus groups, in-depth interviews

5. 5 Meaning of Events Richness of Qualitative Data Contextual Influences on Behavior Processes Underlying Observed Relationships

6. 6 Sampling in Qualitative Research u Purposive, nonprobabilistic u Deliberately select settings, persons or events to best answer research questions u Set up contrasts to examine differences between settings or individuals

7. 7 Integrating Qualitative and Quantitative Methods to Develop Surveys u Goal: develop survey items with similar meaning across groups u Quantitative methods (descriptive statistics, reliability, validity, missing data) identify ethnic differences in response patterns u Qualitative methods uncover cognitive processes used to answer questions; conceptual relevance u Iterative process

8. 8 Integrating Qualitative and Quantitative Methods to Develop Surveys (cont.) u Prior to quantitative: to develop concepts, framework, hypotheses, and content for structured survey items or interventions u After quantitative: –to help identify reasons for survey items not performing well quantitatively –to explore possible explanations for unexpected results

9. 9 Applications in Survey Development To develop: u Concepts and their definitions (FG) u Measures of these concepts (FG) u Pre-test measures (CI)

10. 10 Qualitative Methods to Develop Surveys u Focus Groups –Open-ended guided group discussion with probes u Cognitive interviews –Open-ended probes to assess how items are interpreted and adequacy of response choices u Expert panels provide input on relevant concepts u Interviewer debriefing

11. 11 Focus Groups to Develop Surveys u Assess the universality of concepts and measures u Identify experiences that may be specific to one group (e.g., AA experiences of discrimination) u Pretest clarity, relevance, adequacy of existing measures u Use phrases from transcripts to develop new measures

12. 12 Focus Groups u Open-ended guided group discussion with probing of responses u Led by experienced moderator; usually 1 - 2 hours u Purposeful sampling of 6-10 homogenous participants per group u Participants stimulate comments of others u Audio-record and transcribe discussion

13. 13 Focus Groups-Logistics u Moderator skills: listening, communication, negotiation, cultural similarity to participants u Costs of group: $600 - $1000 per group (incentives, audio-taping, transcription, translation, food) u Convenient and hospitable community setting u In-person recruitment works best with telephone/mail reminders

14. 14 Focus Groups u Advantages –group stimulates fruitful discussion –spontaneity leads to discovery of new issues and factors u Disadvantages: scheduling, skilled moderator, group setting may be inappropriate

15. 15 Example: Measures of Cultural Sensitivity of Clinicians u Conducted 19 focus groups with 163 participants u 61 African Americans, 45 Latinos and 55 non-Latino Whites u Asked about the influence of cultural factors on the quality of encounters

16. 16 Focus Group Questions u What does the word “culture” mean to you? u What do or don’t your doctors understand about your culture or health beliefs that might affect your visits? u Embedded in more general discussion of communication and decision making in medical encounters

17. 17 Identify Themes (Codes): How Culture Might Affect Health Care and Outcomes u CAM u Discrimination u Doctor Culture u Ethnicity of MD u Family u Immigration u Language fluency u Modesty u Nutrition u Patient submissiveness u Spirituality

18. 18 Define Domains of Each Cultural Sensitivity Domain: CAM u Definition: MD’s knowledge and acceptance of non-Western, non- biomedical, holistic approaches to health or healing “When I told her I was on estrogen, it was refreshing to hear suggestions about alternative types of herbal treatments.” AA woman > 50

19. 19 Develop Items Based on Definition and Wording Used by Participants Over the past 12 months, how often did doctors…..ask if you would be interested in hearing more about alternative types of herbal treatments? (CAM domain)

20. 20 Cognitive Interviews u Derived from social and cognitive psychology to explore processes respondents use to answer survey questions u Diagnostic tool for pretesting survey questions

21. 21 Cognitive Interviews Examine 4 Steps in Answering Questions u Comprehension of the question –as intended by the researchers u Retrieve the information –various strategies used to access memory u Judgment formation - formulate an answer –calculate or judge the correct information u Edit response - decide what to report –is answer embarrassing, socially undesirable?

22. 22 Purpose of Cognitive Interviews To learn … u if respondents understand words and phrases as intended (meaning) u about the process of answering the questions u whether items are unacceptable u about the usefulness of response choices

23. 23 Two Types of Cognitive Interviews u Probe interviews –Interviewer asks specific questions to elicit how respondent answered question –Scripted and spontaneous probing u Think aloud interviews –Respondent asked to think aloud as they answer question u Think alouds - greater respondent burden

24. 24 Two Types of Probe Interviews u Concurrent probing –Ask probes immediately after respondent has given answer to survey item –Advantage-information is fresh in respondent’s mind u Retrospective probing –Ask probes after entire interview –Advantage-able to assess standard administration of items

25. 25 Steps Involved in Cognitive Interview Pretesting u Decide on a final item pool u Develop scripted “probes” for a subset of items u Translate interview and probe questions u Recruit sample for cognitive interviews u Conduct cognitive interviews u Analyze results u Revise items based on results u Cognitive testing of revised items

26. 26 Reducing Item Pool u Reduce items from item pool (subjective process among research team) u Criteria: –maintain breadth of concept, multiple items/concept –reduce redundancy (but OK to test alternate versions of items) –eliminate items that are unclear, complex, require high verbal skills, lack face validity, or will not translate well

27. 27 Writing Probe Questions u From reduced set of items, select potentially problematic items for pretesting u Write open-ended scripted probe questions –worded to reveal if suspected problem with a specific item is present

28. 28 Types of Cognitive Probes u General –Tell me what you were thinking when you answered that question –How easy or difficult was it to answer that question? Why? u Explore meaning of word or phrase –I asked you how often doctors take a genuine interest in you. What does the phrase “genuine interest” mean to you?

29. 29 Types of Cognitive Probes (cont.) u Retrieval –How did you remember that? u Judgment –Why did you pick that number for your answer? u Response –Do you think that most people answer this question honestly? Collins D. Quality of Life Research 2003. 12:229-38.

30. 30 Types of Cognitive Probes (cont.) u Redundancy –How is the phrase “give you advice about your diet and exercise” different from the phrase “talk to you about your diet and exercise”? u Acceptability –When I asked you how often you felt discriminated against by doctors because of your race or ethnicity, you answered (read answer given). Were you offended by this question?

31. 31 Types of Cognitive Probes (cont.) u Cultural appropriateness –I asked you how often doctors asked you about your health beliefs? What does the term ‘health beliefs’ mean to you?

32. 32 Sampling for Cognitive Interviews u Usually do not use representative samples u Include respondents from major segments of population to be sampled for main survey u Approximately 5-15 interviews/group, but may involve several rounds

33. 33 Recruiting for Cognitive Interviews u Explain how their help fits into larger study, process of creating questions u Explain their role clearly: –“help us learn how to ask better questions” –“help us make questions clearer for others” –“help us to identify problems with questions” u Pay subjects $25 - $50, interview is demanding u If survey is long, pretest different sections on different subjects

34. 34 Conducting Cognitive Interviews u Individual face-to-face, in-depth interviews u Standard administration of closed-ended items u Administer probe questions at the end (or concurrently) u Typically 1 hr interview u Each interview audiotaped and transcribed

35. 35 Data Analysis of Cognitive Interviews u Create summary of most significant problems: –Annotated questionnaire: use electronic version of survey to enter comments for each item directly under each question –Aggregate item-by-item comments over multiple interviews

36. 36 Analysis of Cognitive Interviews u Direct analysis of transcripts u Two approaches for transcripts –Behavioral coding –Use qualitative data analysis software to perform content analysis

37. 37 Behavioral Coding u Review transcripts to identify problems with standard administration of items –respondent and interviewer “problem” behaviors u Assign “problem behavior” codes to each item using pre-determined categories u Summarize results for each item: –proportion of interviews with each problematic behavior for each item »e.g., 7/48 respondents requested clarification of item 10

38. 38 Examples of Behavioral Codes Interviewer behavior u Hard to read - interviewer experiences difficulty reading question Respondent behavior u Repeat question - respondent asks to have the question repeated

39. 39 Content Analysis of Items and Probes u Using qualitative analysis software, review all dialogue during standard administration of closed-ended items and open-ended probes –can reveal source of problems –can help in deciding whether to keep, modify or drop items u Allows you to examine dialogue by item –within groups –across groups/languages

40. 40 Example of Results: Interpreting the Question Original item: How satisfied are you with the amount of stress or worries in your life? Probe: Did you answer this question in terms of stress, worries, or both? u AA and Spanish-speakers - tended to answer in terms of worries u Revised item –How satisfied are you with the amount of worries in your life? Warnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38.

41. 41 Example of Results: Unclear Phrase Original item: Have you had any medical tests or procedures in past year? –26% of respondents asked for clarification Probe: What did you include as medical tests or procedures? u Medical test or procedures unclear (e.g., asked if it included dental or cosmetic procedures) u Item revised to include examples: –Have you had any medical tests or procedures, such as blood tests, x-rays, or cancer screening tests?

42. 42 Example of Results: Lack of Relevance Original item: How satisfied are you with your ability to travel on vacations? Probe: What do you think we meant by vacation? u Neither concept of “vacation” nor “travel for pleasure” had relevance to lifestyle of AA and MA; travel for family reasons u Dropped item Warnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38.

43. 43 Example of Results: Response Sets Original scale: “Very unimportant to Very Important” on 0-100 scale (bipolar) u Problem: No direct Spanish translation for “unimportant” –bilinguals understood translation “sin importancia” as equivalent to unimportant –Spanish monolinguals did not understand it as the negative pole of the scale u Solution: Changed English to unipolar scale “not at all important” to “very important” to correspond to best Spanish Warnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38

44. 44 Example of Results: Redundancy of Items Original items: –How often did doctors explain what was causing your health problem? –How often did doctors explain your diagnosis? Probe: What do the words health problem and diagnosis mean to you? u Respondents viewed them as the same u Some respondents did not know meaning of diagnosis u Dropped item with word diagnosis

45. 45 Example of Results: Cultural Differences Original item: –How often did doctors ask you if you wanted to include your family when making decisions? Probe: When would you include your family in making decisions about your health care? u AAs and WHs viewed question as irrelevant; only in cases of genetic or terminal illness u Latinos more likely to include family in less serious cases u Dropped item

46. 46 Summary u Greatest problems are with question interpretation –usually due to need to write at lower level of verbal comprehension u Sometimes English concepts not meaningful in other languages, are irrelevant for certain groups, or differ in meaning across groups u Need to pretest response sets

47. 47 Advantages of Cognitive Interviews u Complement other field test methods (e.g. where problems are identified by missing or truncated answers) u Identifies where responses might be affected by cultural or group experiences u Suggests ways to revise items, responses u Improves validity of questions

48. 48 Disadvantages of Cognitive Interviews u Flags problems, but significance of the problem is a subjective judgment –When do you need to revise or drop items? u Based on small number of respondents u Time and labor intensive

49. 49 Conclusions: Usefulness of Qualitative Methods u To identify relevant constructs, items, unanticipated mechanisms of disparities u To pretest items for conceptual adequacy u Even limited cognitive interviewing can reveal significant issues overlooked by survey developers

50. 50 How To Resources u Ward H, Atkins J. From their lives: a manual on how to conduct focus groups of low-income parents. Institute for Child and Family Policy, Edmund S. Muskie School of Public Health, University of Southern Maine, 2002: http://muskie.usm.maine.edu/focusgroupmanual/manual.htm http://muskie.usm.maine.edu/focusgroupmanual/manual.htm u Willis GB. Cognitive Interviewing: A “How To” Guide. Research Triangle Institute, 1999:http://appliedresearch.cancer.gov/areas/cognitive/interview.pdf