Copyright  2000, Richard B. Warnecke QUALITATIVE APPROACHES TO QUESTIONNAIRE DESIGN: METHODS AND APPLICATIONS RICHARD B. WARNECKE, Ph.D. UNIVERSITY OF ILLINOIS AT CHICAGO MINORITY HEALTH PROJECT UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL JUNE 15, 2000 Copyright  2000, Richard B. Warnecke

ANSWERING QUESTIONS u INTERPRETATION u RECALL u JUDGMENT FORMATION u RESPONSE FORMATTING u EDITING

Copyright  2000, Richard B. Warnecke LABORATORY METHODS u FOCUSED INTERVIEWS u THINK-ALOUDS u PARAPHRASING u CONFIDENCE RATINGS u RESPONSE LATENCY u SORTS

Copyright  2000, Richard B. Warnecke INTERPRETING THE QUESTION u LITERAL MEANING u INTENDED MEANING u RESPONSE ALTERNATIVES

Copyright  2000, Richard B. Warnecke INTERPRETATION PROBES u PARAPHRASE QUESTION. u DEFINE A WORD OR PHRASE. u WHAT DOES A RESPONSE MEAN TO YOU? u BOUNDARIES (WHICH OF THE FOLLOWING WOULD ALSO BE …..?) u LABEL PREFERENCE u INTERPRETATION DIFFICULTY

Copyright  2000, Richard B. Warnecke WHAT TYPES OF PHYSICAL ACTIVITY OR EXERCISE DID YOU PERFORM DURING THE PAST MONTH? ________________________________ ________________

Copyright  2000, Richard B. Warnecke WHAT DID YOU THINK WE MEANT WHEN WE SAID “PHYSICAL ACTIVITY?

Copyright  2000, Richard B. Warnecke WHICH, IF ANY, OF THE FOLLOWING WOULD YOU (ALSO) CONSIDER TO BE PHYSICAL ACTIVITY? WHAT ABOUT... WALKING, HOUSEWORK, SUCH AS CLEANING THE BATHROOM OR KITCHEN, WORK-RELATED ACTIVITY, SUCH AS CONSTRUCTION YARD WORK, SUCH AS MOWING THE LAWN OR RAKING LEAVES?

Copyright  2000, Richard B. Warnecke LOGISTIC REGRESSION ANALYSIS OF PHYSICAL ACTIVITY QUESTION MEANING PROBES (UNSTANDARDIZED COEFFICENTS REPORTED) Behavioral frequency0.22* Male (1=Yes) Male-0.91***-0.36**0.41* -0.43** (1 = Yes) African American ** ** (1 = Yes) Mexican American (1 = Yes)

Copyright  2000, Richard B. Warnecke LOGISTIC REGRESSION ANALYSIS OF PHYSICAL ACTIVITY QUESTION MEANING PROBES (UNSTANDARDIZED COEFFICENTS REPORTED) cont’d. Puerto Rican * (1 = Yes) Age0.08** Education Model chi-square30.11***18.42** ** Sample Size

Copyright  2000, Richard B. Warnecke RECALLING AND RETRIEVING u RECALL PRIOR INFORMATION u COMPUTE u ESTIMATE u USE A SCHEMA (A PATTERN OF ASSOCIATIONS AROUND WHICH THINGS ARE RECALLED, I.E. PAP SMEAR AS PART OF A PHYSICAL EXAM).

Copyright  2000, Richard B. Warnecke RETRIEVAL PROBES u RECALL STRATEGY u RECALL INTERVAL u SEARCH STRATEGY (PROXIMAL, DISTAL, ANYWHERE) u LONG TERM RECALL--LINK TO EVENTS TO HELP REMEMBER u RECALL FRAME OF REFERENCE-- WHAT KINDS OF THINGS HELPED YOU REMEMBER?

Copyright  2000, Richard B. Warnecke HOW MANY TIMES PER WEEK OR PER MONTH DID YOU TAKE PART IN (FIRST ACTIVITY MENTIONED)? _______________ TIMES PER WEEK ____________________ TIMES PER MONTH

Copyright  2000, Richard B. Warnecke HOW DID YOU REMEMBER THE NUMBER OF TIMES YOU DID THIS? (PERFORM PHYSICAL ACTIVITY)

Copyright  2000, Richard B. Warnecke IN A TYPICAL WEEK, HOW MANY TIMES DO YOU TALK ON THE TELEPHONE WITH FAMILY, FRIENDS, OR NEIGHBORS? ______ TIMES PER DAY ______ TIMES PER WEEK

Copyright  2000, Richard B. Warnecke IN RECALLING THIS INFORMATION ABOUT HOW MANY TIMES YOU TALK ON THE TELEPHONE, DID YOU SPECIFICALLY THINK ABOUT THE MOST RECENT WEEK OR DID YOU AVERAGE TOGETHER EXPERIENCES YOU’VE HAD OVER A NUMBER OF WEEKS, OR DID YOU THINK ABOUT IT IN SOME OTHER WAY? u THE MOST RECENT WEEK u AVERAGE WEEK u OTHER (SPECIFY) __________________ u DON’T KNOW

Copyright  2000, Richard B. Warnecke JUDGMENT FORMATION u FIT A REFERENCE PERIOD u FIT A DEFINITION u RELATE TO QUESTIONNAIRE CONTEXT-- ORDER EFFECTS

Copyright  2000, Richard B. Warnecke JUDGMENT PROBES u JUDGMENT EXPLANATION –HOW ARRIVED AT ANSWER u JUDGMENT CERTAINTY –SCALE OF CERTAINTY (1-7) u ANCHORING STRATEGIES –COMPARATIVE u ATTITUDE CRYSTALLIZATION –THOUGHT ABOUT BEFORE –FREQUENCY OF PREVIOUS THOUGHT –DISCUSS WITH OTHERS

Copyright  2000, Richard B. Warnecke HOW LIKELY DO YOU THINK IT IS THAT YOU WILL HAVE SERIOUS HEALTH PROBLEMS FROM SMOKING? DO YOU THINK IT IS….. VERY LIKELY SOMEWHAT LIKELY SOMEWHAT UNLIKELY VERY UNLIKELY DEFINITELY NOT POSSIBLE? DON’T KNOW

Copyright  2000, Richard B. Warnecke BEFORE WE ASKED YOU THE QUESTION ABOUT SERIOUS HEALTH PROBLEMS, HAD YOU THOUGHT ABOUT IT BEFORE? YES NO

Copyright  2000, Richard B. Warnecke ON A SCALE FROM 1 TO 7, WHERE 1 IS ONLY RARELY AND 7 IS VERY OFTEN, HOW OFTEN HAVE YOU THOUGHT ABOUT IT BEFORE? ONLY VERY RARELY OFTEN

Copyright  2000, Richard B. Warnecke ORDER EFFECTS-I u MINIMIZING ORDER EFFECTS –GENERAL BEFORE SPECIFIC (FUNNEL) –SPECIFIC BEFORE GENERAL (INVERTED FUNNEL) –USE ORDER TO REDUCE EFFECTS OR ITEM NON RESPONSE

Copyright  2000, Richard B. Warnecke ORDER EFFECTS -2 u DEVELOP A LOGICAL FLOW TO HELP RESPONDENT –SEGMENT BY TOPIC –ASK ABOUT RELATED TOPICS TOGETHER –ASK RECALL FROM MOST RECENT TO LEAST RECENT –USE TRANSITIONS TO CHANGE TOPICS

Copyright  2000, Richard B. Warnecke ORDER EFFECTS-III u ESTABLISH RAPPORT –START WITH EASY, NON- THREATENING QUESTIONS –DON’T START WITH DEMOGRAPHICS END WITH THEM –DON’T START WITH KNOWLEDGE AND AWARENESS

Copyright  2000, Richard B. Warnecke EXAMPLE OF USE OF ORDER TO RESOLVE CONFUSION u ORIGINAL QUESTION: HOW SATISFIED ARE YOU WITH YOUR RELATIONSHIP WITH YOUR SPOUSE/SIGNIFICANT OTHER? u PROBE: WHAT PARTS OF THE RELATIONSHIP DID YOU THINK ABOUT?

Copyright  2000, Richard B. Warnecke THE PROBLEM u QUESTION WORKED WELL IN ENGLISH. u HOWEVER IN SPANISH THE WORD FOR RELATIONS IS THE WORD FOR SEXUAL RELATIONS

Copyright  2000, Richard B. Warnecke SOLUTION u REVERSED ORDER OF QUESTIONS. ASKED ABOUT SATISFACTION WITH SEXUAL RELATIONS FIRST. u THEN ASKED ABOUT OTHER ASPECTS OF MARITAL RELATIONSHIP.

Copyright  2000, Richard B. Warnecke FORMATTING A RESPONSE u RESPONSE CATEGORIES u RATING SCALES u RESPONSE SET –EXTREME CHOICES –ACQUIESCENCE

Copyright  2000, Richard B. Warnecke African American (N=102) Mexican American (N=106) Puerto Rican (N=84) White non- Hispanic (N=101) Mean** Standard deviation **F-ratio p<.01. Note: Means with different superscripts differ significantly (p<.05) on the basis of multiple range test. Mean Group Responses to Measure of Extreme Response Style

Copyright  2000, Richard B. Warnecke African American (N=102 Mexican American (N=106) Puerto Rican (N=84) White non- Hispanic (N=101) Mean** Standard deviation **F-ratio p<.01. Note: Means with different superscripts differ significantly (p<.05) on the basis of multiple range test. Mean Group Responses to Acquiescence Scale

Copyright  2000, Richard B. Warnecke African American (N=110) Mexican American (N=110) Puerto Rican (N=87) White non- Hispanic (N=108) Mean** Standard deviation **F-ratio p<.01. Note: Means with different superscripts differ significantly (p<.05) on the basis of multiple range test. Mean Group Responses to the Marlowe-Crowne Social Desirability Scale

Copyright  2000, Richard B. Warnecke SCALE OF SATISFACTION WITH QUALITY OF LIFE u QUALITY OF LIFE SCALE WITH RANGE OF VALUES. –VERY SATISFIED –SATISFIED –SOMEWHAT SATISFIED –A LITTLE SATISFIED –NEITHER SATISFIED NOR DISSATISFIED –A LITTLE DISSATISFIED –SOMEWHAT DISSATISFIED –DISSATISFIED –VERY DISSATISFIED

Copyright  2000, Richard B. Warnecke ASSESSMENT SATISFACTION RANGE u VERY u SATISFIED u SOMEWHAT u A LITTLE DISSATISFIED RANGE u VERY 0-80 u DISSATISFIED 0-80 u SOMEWHAT u A LITTLE 10-53

Copyright  2000, Richard B. Warnecke SOLUTION u USED A BAR GRAPH WITH SINGLE SCALE WHERE BLANK EQUALS “NOT AT ALL SATISFIED” AND 6 EQUALS “VERY SATISFIED.” u SIMPLIFIED RESPONSE ON ONE DIMENSION u AVOIDED LABELS THAT WERE CONFUSING.

Copyright  2000, Richard B. Warnecke EDITING A RESPONSE u SOCIAL DESIRABILITY u SELF PRESENTATION u EFFECT OF INTERVIEWER –GENDER –AGE –CULTURE/ ETHNICITY

Copyright  2000, Richard B. Warnecke RESPONSE EDITING PROBES u PERCEIVED SOCIAL DESIRABILITY u PERCEIVED EMBARRASSMENT u PERCEIVED HONESTY u PERCEIVED INTERVIEWER EFFECTS –ETHNICITY –AGE –GENDER

Copyright  2000, Richard B. Warnecke INTERVIEWER EFFECTS: AGE u ORIGINAL QUESTION: HAVE YOU EVER SMOKED MARIJUANA, HASH OR HASH OIL OR TAKEN THEM IN SOME OTHER WAY? u PROBE: DO YOU FEEL THIS IS A QUESTION THAT A PERSON YOUR AGE WOULD BE COMFORTABLE OR UNCOMFORTABLE TALKING ABOUT TO A SURVEY INTERVIEWER OF THE SAME AGE OR WHO WAS MUCH YOUNGER OR OLDER?

Copyright  2000, Richard B. Warnecke Interviewer age Respondent age Younger Older (18-29) (30-50) Same age Younger or Older Projected Comfort Level Discussing Marijuana Use with Hypothetical Interviewer of Same Age/Different Age

Copyright  2000, Richard B. Warnecke INTERVIEWER EFFECTS: GENDER u ORIGINAL QUESTION: DURING THE PAST WEEK, HOW OFTEN HAVE YOU HAD CRYING SPELLS? u PROBE: DO YOU FEEL THIS IS A QUESTION THAT (RESPONDENT’S GENDER) WOULD BE COMFORTABLE TALKING ABOUT TO A SURVEY INTERVIEWER OF THE SAME SEX? OF THE OPPOSITE SEX?

Copyright  2000, Richard B. Warnecke Interviewer genderMaleFemale Same gender Opposite gender Projected Comfort Level Discussing Crying Spells with Hypothetical Interviewer of Same Sex/Opposite Sex

Copyright  2000, Richard B. Warnecke INTERVIEWER EFFECTS: ETHNICITY u ORIGINAL QUESTION: DO YOU THINK OF YOURSELF AS HETEROSEXUAL, HOMOSEXUAL OR BISEXUAL? u PROBE: DO YOU FEEL THIS IS A QUESTION THAT (RESPONDENT’S ETHNIC GROUP) WOULD BE COMFORTABLE DISCUSSING WITH AN INTERVIEWER OF THE SAME (ETHNIC GROUP)? A SURVEY INTERVIEWER OF A DIFFERENT ETHNIC GROUP?

Copyright  2000, Richard B. Warnecke Comfort Level African American (N=101) Mexican American (N=100) Puerto Rican (N=82) White Non- Hispanic (N=99) Topic: Sexual orientation Interviewer race/ethnicity: Same race/ethnicity Different race/ethnicity Projected Comfort Level of Respondents by Topic, By Culture of Hypothetical Interviewer, And By Culture of Respondent (Percent Comfortable)