1. 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for Aging in Diverse Communities

2. 2 Overview u Integrating quantitative and qualitative methods to develop surveys u Focus groups u Cognitive interview methods u Using results of cognitive interviews to make decisions about item revisions/deletions

3. 3 When are Qualitative Methods Used? u Open-ended interviews typically used in new areas of study u Useful for in-depth knowledge about issues, especially in less studied groups u Especially critical in cross-cultural studies due to lack of information

4. 4 What are Qualitative Methods? u Data consist of words, not numbers u Richly descriptive, open-ended u Focus on inductive analytic approaches u Many types: ethnography, participant- observation, direct observation, focus groups, in-depth interviews

5. 5 When are Qualitative Methods Useful? u To understand the meaning of participants’ events, situations, and actions u To understand contextual influences on participants’ actions u To identify unanticipated phenomena and influences (e.g., exploratory studies to design questionnaires and identify variables for study) u To understand the processes underlying observed relationships between variables

6. 6 Meaning of Events Richness of Qualitative Data Contextual Influences on Behavior Processes Underlying Observed Relationships

7. 7 Integrating Qualitative and Quantitative Methods to Develop Surveys u Goal: develop survey items that mean the same thing across groups u Quantitative methods (descriptive statistics, reliability, validity, missing data) identify ethnic differences in response patterns u Qualitative methods uncover cognitive processes used to answer questions; conceptual relevance u Iterative quantitative and qualitative methods

8. 8 Combining Qualitative and Quantitative Methods u Prior to quantitative: to develop concepts, framework, hypotheses, and content for structured survey items or interventions u After quantitative: –to help identify reasons for survey items not performing well quantitatively –to explore possible explanations for unexpected results

9. 9 Applications To develop: u Concepts and their definitions u Measures of these concepts u Pre-test measures

10. 10 Qualitative Methods to Develop Surveys u Focus Groups –Open-ended guided group discussion with probing of responses u Cognitive interviews –Open-ended probes to assess how items are interpreted and adequacy of response choices –Respondent is expert u Expert panels provide input on relevant concepts u Interviewer debriefing

11. 11 Focus Groups to Develop Surveys u To assess the universality of concepts and measures – funnel approach, concept to term u Facilitate culturally sensitive models (e.g., how being AA shapes experiences of discrimination) u Use phrases from transcripts to develop items u Pretest clarity and relevance of existing items

12. 12 Focus Groups u Open-ended guided group discussion with probing of responses u Led by experienced moderator; usually 1 - 2 hours u Purposeful sampling of 6-10 homogenous participants per group u Participants stimulate comments of others u Audio-record and transcribe discussion

13. 13 Focus Groups-Logistics u Moderator skills: listening, communication, negotiation, cultural similarity to participants u Costs of group: $600 - $1000 per group (incentives, audio-taping, transcription, translation, food) u Convenient and hospitable community setting u In-person recruitment works best with telephone/mail reminders

14. 14 Focus Groups u Advantages –group stimulates fruitful discussion –spontaneity leads to discovery of new issues and factors u Disadvantages: scheduling, skilled moderator, group setting may be inappropriate

15. 15 Sampling in Qualitative Research u Purposive, nonprobabilistic u Deliberately select settings, persons or events to best answer research questions u Set up contrasts to examine differences between settings or individuals

16. 16 Example: Measures of Cultural Sensitivity of Clinicians u Conducted 19 focus groups with 163 participants u 61 African Americans, 45 Latinos and 55 non-Latino Whites u Asked about the influence of cultural factors on the quality of encounters

17. 17 Focus Group Questions u What does the word “culture” mean to you? u What do or don’t your doctors understand about your culture or health beliefs that might affect your visits? u Embedded in more general discussion of communication and decision making in medical encounters

18. 18 Identify Themes (Codes): How Culture Might Affect Health Care and Outcomes u CAM u Discrimination u Doctor Culture u Ethnicity of MD u Family u Immigration u Language u Modesty u Nutrition u Patient submissiveness u Spirituality

19. 19 Define Domains of Each Cultural Sensitivity Domain: CAM u Definition: MD’s knowledge and acceptance of non-Western, non- biomedical, holistic approaches to health or healing “When I told her I was on estrogen, it was refreshing to hear suggestions about alternative types of herbal treatments.” AA woman > 50

20. 20 Develop Items Based on Definition and Wording Used by Participants Over the past 12 months, how often did doctors…..ask if you would be interested in hearing more about alternative types of herbal treatments? (CAM domain)

21. 21 Cognitive Interviews u Derived from social and cognitive psychology to explore processes respondents use to answer survey questions u Diagnostic tool for pretesting survey questions

22. 22 Cognitive Interviews Examine 4 Steps in Answering Questions u Comprehension of the question –as intended by the researchers u Retrieve the information –various strategies used to access memory u Judgment formation - formulate an answer –calculate or judge the correct information u Edit response - decide what to report –is answer embarrassing, socially undesirable?

23. 23 Purpose of Cognitive Interviews To learn.. u.. if respondents understand words and phrases as intended (meaning) u.. about the process of answering the questions u.. whether items are unacceptable u.. about the usefulness of response choices –whether response choices are adequate –how they use the response choices

24. 24 Two Types of Cognitive Interviews u Think aloud interviews –Respondent asked to think aloud as they answer question u Probe interviews –Interviewer asks specific questions to elicit how respondent answered question –Scripted and spontaneous probing u Think alouds - greater respondent burden

25. 25 Two Types of Probe Interviews u Concurrent probing –Ask probes immediately after respondent has given answer to survey item –Advantage-information is fresh in respondent’s mind u Retrospective probing –Ask probes after entire interview –Advantage-able to assess standard administration of items

26. 26 Steps Involved in Cognitive Interview Pretesting u Decide on a final item pool u Develop scripted “probes” for a subset of items u Translate interview and probe questions u Recruit sample for cognitive interviews u Conduct cognitive interviews u Analyze results u Revise items based on results u Cognitive testing of revised items

27. 27 Reducing Item Pool u Reduce items from item pool (subjective process among research team) u Criteria: –maintain breadth of concept, multiple items/concept –reduce redundancy (but OK to test alternate versions of items) –eliminate items that are unclear, complex, require high verbal skills, lack face validity, or will not translate well

28. 28 Writing Probe Questions u From reduced set of items, select potentially problematic items for pretesting u Write open-ended scripted probe questions –worded to reveal if suspected problem with a specific item is present

29. 29 Types of Cognitive Probes u General –Tell me what you were thinking when you answered that question –How easy or difficult was it to answer that question? Why? u Explore meaning of word or phrase –I asked you how often doctors take a genuine interest in you. What does the phrase “genuine interest” mean to you?

30. 30 Types of Cognitive Probes (cont.) u Retrieval –How did you remember that? u Judgment –Why did you pick that number for your answer? u Response –Do you think that most people answer this question honestly? Collins D. Quality of Life Research 2003. 12:229-38.

31. 31 Types of Cognitive Probes (cont.) u Redundancy –How is the phrase “give you advice about your diet and exercise” different from the phrase “talk to you about your diet and exercise”? u Acceptability –When I asked you how often you felt discriminated against by doctors because of your race or ethnicity, you answered (read answer given). Were you offended by this question?

32. 32 Types of Cognitive Probes (cont.) u Cultural appropriateness –I asked you how often doctors asked you about your health beliefs? What does the term ‘health beliefs’ mean to you?

33. 33 Sampling for Cognitive Interviews u Usually do not use representative samples u Include respondents from major segments of population to be sampled for main survey u Approximately 5-15 interviews/group, but may involve several rounds

34. 34 Recruiting for Cognitive Interviews u Explain how their help fits into larger study, process of creating questions u Explain their role clearly: –“help us learn how to ask better questions” –“help us make questions clearer for others” –“help us to identify problems with questions” u Pay subjects $25 - $50, interview is demanding u If survey is long, pretest different sections on different subjects

35. 35 Conducting Cognitive Interviews u Individual face-to-face, in-depth interviews u Standard administration of closed-ended items u Administer probe questions at the end (or concurrently) u Typically 1 hr interview u Each interview audiotaped and transcribed

36. 36 Data from Cognitive Interviews Two general methods for compiling data u Use electronic version of survey to enter comments on each item directly under each question OR u Transcripts are data

37. 37 Analysis of Cognitive Interviews u For method where entered comments under each question –Annotated questionnaire - aggregate item-by- item comments over multiple interviews –Summary of most significant problems u Two approaches for transcripts –Behavioral coding –Use qualitative data analysis software to perform content analysis

38. 38 Behavioral Coding u Review transcripts to identify problems with standard administration of items –respondent and interviewer “problem” behaviors u Assign “problem behavior” codes to each item using pre-determined categories u Summarize results for each item: –proportion of interviews with each problematic behavior for each item »e.g., 7/48 respondents requested clarification of item 10

39. 39 Examples of Behavioral Codes Interviewer behavior u Hard to read - interviewer experiences difficulty reading question Respondent behavior u Repeat question - respondent asks to have the question repeated

40. 40 Content Analysis of Items and Probes u Using qualitative analysis software, review all dialogue that ensued during standard administration of closed-ended items and open-ended probes –can reveal source of problems –can help in deciding whether to keep, modify or drop items u Allows you to examine dialogue for each item –within groups –across groups/languages

41. 41 Example of Results: Interpreting the Question Original item: How satisfied are you with the amount of stress or worries in your life? Probe: Did you answer this question in terms of stress, worries, or both? u AA - tended to answer in terms of worries u No term in Spanish for “stress” u Item revised –How satisfied are you with the amount of worries in your life? Warnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38.

42. 42 Example of Results: Unclear Phrase Original item: Have you had any medical tests or procedures in past year? –26% of respondents asked for clarification Probe: What did you include as medical tests or procedures? u Medical test or procedures unclear (e.g., asked if it included dental or cosmetic procedures) u Item revised to include examples: –Have you had any medical tests or procedures, such as blood tests, x-rays, or cancer screening tests?

43. 43 Example of Results: Information Retrieval Original item: How satisfied are you with your ability to travel on vacations? Probe: What do you think we meant by vacation? u Neither concept of “vacation” nor “travel for pleasure” had relevance to lifestyle of AA and MA; travel for family reasons u Dropped item Warnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38.

44. 44 Example of Results: Response Sets Original scale: “Very unimportant to Very Important” on 0-100 scale (bipolar) u Problem: No direct Spanish translation for “unimportant” –bilinguals understood translation “sin importancia” as equivalent to unimportant –Spanish monolinguals did not understand it as the negative pole of the scale u Solution: Changed English to unipolar scale “not at all important” to “very important” to correspond to best Spanish Warnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38

45. 45 Example of Results: Redundancy of Items Original items: –How often did doctors explain what was causing your health problem? –How often did doctors explain your diagnosis? Probe: What do the words health problem and diagnosis mean to you? u Respondents viewed them as the same u Some respondents did not know meaning of diagnosis u Dropped item with word diagnosis

46. 46 Example of Results: Cultural Differences Original item: –How often did doctors ask you if you wanted to include your family when making decisions? Probe: When would you include your family in making decisions about your health care? u AAs and WHs viewed question as irrelevant; only in cases of genetic or terminal illness u Latinos more likely to include family in less serious cases u Dropped item

47. 47 Summary u Greatest problems are with question interpretation –usually due to need to write at lower level of verbal comprehension u Sometimes English concepts not meaningful in other languages, are irrelevant for certain groups, or differ in meaning across groups u Need to pretest response sets –Ethnic groups may use them differently

48. 48 Advantages of Cognitive Interviews u Complement other field test methods (e.g. where problems are identified by missing or truncated answers) u Identifies where responses might be affected by cultural or group experiences u Suggests ways to revise items, response sets u Improves validity of questions

49. 49 Disadvantages of Cognitive Interviews u Flags problems, but significance of the problem remains matter of subjective judgment –at which point need to revise or drop items u Based on small number of respondents u Time and labor intensive

50. 50 Conclusions: Usefulness of Qualitative Methods u Qualitative methods are necessary component of research in diverse groups u To identify relevant constructs, items, unanticipated mechanisms of disparities u To pretest items for conceptual adequacy u Even limited cognitive interviewing can reveal significant issues overlooked by survey developers

51. 51 How To Resources u Ward H, Atkins J. From their lives: a manual on how to conduct focus groups of low-income parents. Institute for Child and Family Policy, Edmund S. Muskie School of Public Health, University of Southern Maine, 2002: http://muskie.usm.maine.edu/focusgroupmanual/manual.htm http://muskie.usm.maine.edu/focusgroupmanual/manual.htm u Willis GB. Cognitive Interviewing: A “How To” Guide. Research Triangle Institute, 1999:http://appliedresearch.cancer.gov/areas/cognitive/interview.pdf