1. Copyright  2000, Richard B. Warnecke QUALITATIVE APPROACHES TO QUESTIONNAIRE DESIGN: METHODS AND APPLICATIONS RICHARD B. WARNECKE, Ph.D. UNIVERSITY OF ILLINOIS AT CHICAGO MINORITY HEALTH PROJECT UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL JUNE 15, 2000 Copyright  2000, Richard B. Warnecke

2. ANSWERING QUESTIONS u INTERPRETATION u RECALL u JUDGMENT FORMATION u RESPONSE FORMATTING u EDITING

3. Copyright  2000, Richard B. Warnecke LABORATORY METHODS u FOCUSED INTERVIEWS u THINK-ALOUDS u PARAPHRASING u CONFIDENCE RATINGS u RESPONSE LATENCY u SORTS

4. Copyright  2000, Richard B. Warnecke INTERPRETING THE QUESTION u LITERAL MEANING u INTENDED MEANING u RESPONSE ALTERNATIVES

5. Copyright  2000, Richard B. Warnecke INTERPRETATION PROBES u PARAPHRASE QUESTION. u DEFINE A WORD OR PHRASE. u WHAT DOES A RESPONSE MEAN TO YOU? u BOUNDARIES (WHICH OF THE FOLLOWING WOULD ALSO BE …..?) u LABEL PREFERENCE u INTERPRETATION DIFFICULTY

6. Copyright  2000, Richard B. Warnecke WHAT TYPES OF PHYSICAL ACTIVITY OR EXERCISE DID YOU PERFORM DURING THE PAST MONTH? ________________________________ ________________

7. Copyright  2000, Richard B. Warnecke WHAT DID YOU THINK WE MEANT WHEN WE SAID “PHYSICAL ACTIVITY?

8. Copyright  2000, Richard B. Warnecke WHICH, IF ANY, OF THE FOLLOWING WOULD YOU (ALSO) CONSIDER TO BE PHYSICAL ACTIVITY? WHAT ABOUT... WALKING, HOUSEWORK, SUCH AS CLEANING THE BATHROOM OR KITCHEN, WORK-RELATED ACTIVITY, SUCH AS CONSTRUCTION YARD WORK, SUCH AS MOWING THE LAWN OR RAKING LEAVES?

9. Copyright  2000, Richard B. Warnecke LOGISTIC REGRESSION ANALYSIS OF PHYSICAL ACTIVITY QUESTION MEANING PROBES (UNSTANDARDIZED COEFFICENTS REPORTED) Behavioral frequency0.22*0.020.070.12 Male (1=Yes) Male-0.91***-0.36**0.41* -0.43** (1 = Yes) African American0.070.042**-0.01 0.59** (1 = Yes) Mexican American0.480.250.360.43 (1 = Yes)

10. Copyright  2000, Richard B. Warnecke LOGISTIC REGRESSION ANALYSIS OF PHYSICAL ACTIVITY QUESTION MEANING PROBES (UNSTANDARDIZED COEFFICENTS REPORTED) cont’d. Puerto Rican0.260.38*-0.090.34 (1 = Yes) Age0.08**0.01-0.000.00 Education0.120.10-0.12-0.03 Model chi-square30.11***18.42**8.21 18.00** Sample Size365372371371

11. Copyright  2000, Richard B. Warnecke RECALLING AND RETRIEVING u RECALL PRIOR INFORMATION u COMPUTE u ESTIMATE u USE A SCHEMA (A PATTERN OF ASSOCIATIONS AROUND WHICH THINGS ARE RECALLED, I.E. PAP SMEAR AS PART OF A PHYSICAL EXAM).

12. Copyright  2000, Richard B. Warnecke RETRIEVAL PROBES u RECALL STRATEGY u RECALL INTERVAL u SEARCH STRATEGY (PROXIMAL, DISTAL, ANYWHERE) u LONG TERM RECALL--LINK TO EVENTS TO HELP REMEMBER u RECALL FRAME OF REFERENCE-- WHAT KINDS OF THINGS HELPED YOU REMEMBER?

13. Copyright  2000, Richard B. Warnecke HOW MANY TIMES PER WEEK OR PER MONTH DID YOU TAKE PART IN (FIRST ACTIVITY MENTIONED)? _______________ TIMES PER WEEK ____________________ TIMES PER MONTH

14. Copyright  2000, Richard B. Warnecke HOW DID YOU REMEMBER THE NUMBER OF TIMES YOU DID THIS? (PERFORM PHYSICAL ACTIVITY)

15. Copyright  2000, Richard B. Warnecke IN A TYPICAL WEEK, HOW MANY TIMES DO YOU TALK ON THE TELEPHONE WITH FAMILY, FRIENDS, OR NEIGHBORS? ______ TIMES PER DAY ______ TIMES PER WEEK

16. Copyright  2000, Richard B. Warnecke IN RECALLING THIS INFORMATION ABOUT HOW MANY TIMES YOU TALK ON THE TELEPHONE, DID YOU SPECIFICALLY THINK ABOUT THE MOST RECENT WEEK OR DID YOU AVERAGE TOGETHER EXPERIENCES YOU’VE HAD OVER A NUMBER OF WEEKS, OR DID YOU THINK ABOUT IT IN SOME OTHER WAY? u THE MOST RECENT WEEK u AVERAGE WEEK u OTHER (SPECIFY) __________________ u DON’T KNOW

17. Copyright  2000, Richard B. Warnecke JUDGMENT FORMATION u FIT A REFERENCE PERIOD u FIT A DEFINITION u RELATE TO QUESTIONNAIRE CONTEXT-- ORDER EFFECTS

18. Copyright  2000, Richard B. Warnecke JUDGMENT PROBES u JUDGMENT EXPLANATION –HOW ARRIVED AT ANSWER u JUDGMENT CERTAINTY –SCALE OF CERTAINTY (1-7) u ANCHORING STRATEGIES –COMPARATIVE u ATTITUDE CRYSTALLIZATION –THOUGHT ABOUT BEFORE –FREQUENCY OF PREVIOUS THOUGHT –DISCUSS WITH OTHERS

19. Copyright  2000, Richard B. Warnecke HOW LIKELY DO YOU THINK IT IS THAT YOU WILL HAVE SERIOUS HEALTH PROBLEMS FROM SMOKING? DO YOU THINK IT IS….. VERY LIKELY SOMEWHAT LIKELY SOMEWHAT UNLIKELY VERY UNLIKELY DEFINITELY NOT POSSIBLE? DON’T KNOW

20. Copyright  2000, Richard B. Warnecke BEFORE WE ASKED YOU THE QUESTION ABOUT SERIOUS HEALTH PROBLEMS, HAD YOU THOUGHT ABOUT IT BEFORE? YES NO

21. Copyright  2000, Richard B. Warnecke ON A SCALE FROM 1 TO 7, WHERE 1 IS ONLY RARELY AND 7 IS VERY OFTEN, HOW OFTEN HAVE YOU THOUGHT ABOUT IT BEFORE? 1 2 3 4 5 6 7 ONLY VERY RARELY OFTEN

22. Copyright  2000, Richard B. Warnecke ORDER EFFECTS-I u MINIMIZING ORDER EFFECTS –GENERAL BEFORE SPECIFIC (FUNNEL) –SPECIFIC BEFORE GENERAL (INVERTED FUNNEL) –USE ORDER TO REDUCE EFFECTS OR ITEM NON RESPONSE

23. Copyright  2000, Richard B. Warnecke ORDER EFFECTS -2 u DEVELOP A LOGICAL FLOW TO HELP RESPONDENT –SEGMENT BY TOPIC –ASK ABOUT RELATED TOPICS TOGETHER –ASK RECALL FROM MOST RECENT TO LEAST RECENT –USE TRANSITIONS TO CHANGE TOPICS

24. Copyright  2000, Richard B. Warnecke ORDER EFFECTS-III u ESTABLISH RAPPORT –START WITH EASY, NON- THREATENING QUESTIONS –DON’T START WITH DEMOGRAPHICS END WITH THEM –DON’T START WITH KNOWLEDGE AND AWARENESS

25. Copyright  2000, Richard B. Warnecke EXAMPLE OF USE OF ORDER TO RESOLVE CONFUSION u ORIGINAL QUESTION: HOW SATISFIED ARE YOU WITH YOUR RELATIONSHIP WITH YOUR SPOUSE/SIGNIFICANT OTHER? u PROBE: WHAT PARTS OF THE RELATIONSHIP DID YOU THINK ABOUT?

26. Copyright  2000, Richard B. Warnecke THE PROBLEM u QUESTION WORKED WELL IN ENGLISH. u HOWEVER IN SPANISH THE WORD FOR RELATIONS IS THE WORD FOR SEXUAL RELATIONS

27. Copyright  2000, Richard B. Warnecke SOLUTION u REVERSED ORDER OF QUESTIONS. ASKED ABOUT SATISFACTION WITH SEXUAL RELATIONS FIRST. u THEN ASKED ABOUT OTHER ASPECTS OF MARITAL RELATIONSHIP.

28. Copyright  2000, Richard B. Warnecke FORMATTING A RESPONSE u RESPONSE CATEGORIES u RATING SCALES u RESPONSE SET –EXTREME CHOICES –ACQUIESCENCE

29. Copyright  2000, Richard B. Warnecke African American (N=102) Mexican American (N=106) Puerto Rican (N=84) White non- Hispanic (N=101) Mean**7.788.098.696.7 Standard deviation 4.653.994.204.42 **F-ratio p<.01. Note: Means with different superscripts differ significantly (p<.05) on the basis of multiple range test. Mean Group Responses to Measure of Extreme Response Style

30. Copyright  2000, Richard B. Warnecke African American (N=102 Mexican American (N=106) Puerto Rican (N=84) White non- Hispanic (N=101) Mean**10.009.969.438.76 Standard deviation 2.842.913.122.71 **F-ratio p<.01. Note: Means with different superscripts differ significantly (p<.05) on the basis of multiple range test. Mean Group Responses to Acquiescence Scale

31. Copyright  2000, Richard B. Warnecke African American (N=110) Mexican American (N=110) Puerto Rican (N=87) White non- Hispanic (N=108) Mean**5.575.925.474.85 Standard deviation 2.472.302.162.29 **F-ratio p<.01. Note: Means with different superscripts differ significantly (p<.05) on the basis of multiple range test. Mean Group Responses to the Marlowe-Crowne Social Desirability Scale

32. Copyright  2000, Richard B. Warnecke SCALE OF SATISFACTION WITH QUALITY OF LIFE u QUALITY OF LIFE SCALE WITH RANGE OF VALUES. –VERY SATISFIED –SATISFIED –SOMEWHAT SATISFIED –A LITTLE SATISFIED –NEITHER SATISFIED NOR DISSATISFIED –A LITTLE DISSATISFIED –SOMEWHAT DISSATISFIED –DISSATISFIED –VERY DISSATISFIED

33. Copyright  2000, Richard B. Warnecke ASSESSMENT SATISFACTION RANGE u VERY 100-60 u SATISFIED 100-50 u SOMEWHAT 90-45 u A LITTLE 70-30 DISSATISFIED RANGE u VERY 0-80 u DISSATISFIED 0-80 u SOMEWHAT 10-54 u A LITTLE 10-53

34. Copyright  2000, Richard B. Warnecke SOLUTION u USED A BAR GRAPH WITH SINGLE SCALE WHERE BLANK EQUALS “NOT AT ALL SATISFIED” AND 6 EQUALS “VERY SATISFIED.” u SIMPLIFIED RESPONSE ON ONE DIMENSION u AVOIDED LABELS THAT WERE CONFUSING.

35. Copyright  2000, Richard B. Warnecke EDITING A RESPONSE u SOCIAL DESIRABILITY u SELF PRESENTATION u EFFECT OF INTERVIEWER –GENDER –AGE –CULTURE/ ETHNICITY

36. Copyright  2000, Richard B. Warnecke RESPONSE EDITING PROBES u PERCEIVED SOCIAL DESIRABILITY u PERCEIVED EMBARRASSMENT u PERCEIVED HONESTY u PERCEIVED INTERVIEWER EFFECTS –ETHNICITY –AGE –GENDER

37. Copyright  2000, Richard B. Warnecke INTERVIEWER EFFECTS: AGE u ORIGINAL QUESTION: HAVE YOU EVER SMOKED MARIJUANA, HASH OR HASH OIL OR TAKEN THEM IN SOME OTHER WAY? u PROBE: DO YOU FEEL THIS IS A QUESTION THAT A PERSON YOUR AGE WOULD BE COMFORTABLE OR UNCOMFORTABLE TALKING ABOUT TO A SURVEY INTERVIEWER OF THE SAME AGE OR WHO WAS MUCH YOUNGER OR OLDER?

38. Copyright  2000, Richard B. Warnecke Interviewer age Respondent age Younger Older (18-29) (30-50) Same age 77.578.6 Younger or Older 30.354.6 Projected Comfort Level Discussing Marijuana Use with Hypothetical Interviewer of Same Age/Different Age

39. Copyright  2000, Richard B. Warnecke INTERVIEWER EFFECTS: GENDER u ORIGINAL QUESTION: DURING THE PAST WEEK, HOW OFTEN HAVE YOU HAD CRYING SPELLS? u PROBE: DO YOU FEEL THIS IS A QUESTION THAT (RESPONDENT’S GENDER) WOULD BE COMFORTABLE TALKING ABOUT TO A SURVEY INTERVIEWER OF THE SAME SEX? OF THE OPPOSITE SEX?

40. Copyright  2000, Richard B. Warnecke Interviewer genderMaleFemale Same gender 34.193.3 Opposite gender 28.141.3 Projected Comfort Level Discussing Crying Spells with Hypothetical Interviewer of Same Sex/Opposite Sex

41. Copyright  2000, Richard B. Warnecke INTERVIEWER EFFECTS: ETHNICITY u ORIGINAL QUESTION: DO YOU THINK OF YOURSELF AS HETEROSEXUAL, HOMOSEXUAL OR BISEXUAL? u PROBE: DO YOU FEEL THIS IS A QUESTION THAT (RESPONDENT’S ETHNIC GROUP) WOULD BE COMFORTABLE DISCUSSING WITH AN INTERVIEWER OF THE SAME (ETHNIC GROUP)? A SURVEY INTERVIEWER OF A DIFFERENT ETHNIC GROUP?

42. Copyright  2000, Richard B. Warnecke Comfort Level African American (N=101) Mexican American (N=100) Puerto Rican (N=82) White Non- Hispanic (N=99) Topic: Sexual orientation Interviewer race/ethnicity: Same race/ethnicity 72.364.065.482.8 Different race/ethnicity 61.051.554.971.4 Projected Comfort Level of Respondents by Topic, By Culture of Hypothetical Interviewer, And By Culture of Respondent (Percent Comfortable)