Using Context to Evaluate Survey Items Presented by Michael Burke, MPH, PhD RTI International Presented at AEA Evaluation 2009 Orlando, FL Phone 202-974-7805 • Fax 202-728-2095 • e-mail mburke@rti.org

Overview Background What is a cognitive interview? Methods Advantages/disadvantages Types of problems found Types of verbal probes Why we care Test Future research

Pretesting Surveys Pretesting has been around since the 1940s or earlier Typically a dry run of the survey with a smaller sample size Look for non-response, examine distributions, discuss the survey with interviewers, etc. Cognitive interviewing has been around since the 1980s 3

Cognitive Interviews Arose from dialogue between survey researchers and cognitive psychologists Typically assume a 4 step cognitive process* Interpretation and comprehension Retrieval of pertinent information Integration and evaluation of retrieved info (judgment) Selection of a response category (reporting) Cognitive interviewing has not greatly expanded to include other academic disciplines and theories of human behavior *Tourangeau, 1984 Cognitive Aspects of Survey Methodology conference in 1984 Census Bureau, Bureau of Labor Statistics, and academic and commercial research organizations have cognitive laboratories. Census had a lab before the 1980 census to evaluate items, but the orientation for the laboratory came from marketing research rather than cognitive psychology. One of the methods used was talk-aloud as they filled out questionnaires. Presser et. al 2004. 4

What is a cognitive interview? Interview designed to reveal some of the mental processes elicited by questions Method to examine how respondents perceive and interpret questions and detect possible sources of response error Method to understand the mental processes involved in responding to a given message that may result in miscommunication A method of increasing recall of an event for the purpose of providing information to law enforcement (Fisher and Geiselman, 1992) 5

Criticisms Lack of standardization Definition of a cognitive interview Methods employed Means of analysis Little methodological research to determine relative effectiveness of various methods Focus on finding problems with questions and meaning of responses rather than on cognitive processes 6

Cognitive Interview Methods Asking a respondent to think-aloud and say what you are thinking (see Ericsson and Simon 1984, 1993, 1998) Verbal probing by an interviewer to reveal or clarify the subject’s thinking (see Willis, 1994, 1999, 2005) Cognitive probes Confirmatory probes Expansive probes Functional remarks Feedback (Beatty, 2004) Contextual probes 7

Think-aloud Asking the participant to verbalize their thinking Works better: verbal information (not spatial or nonverbal) problem oriented questions conscious processing of information Less effective: understanding of terms May rely on short term memory recall Should not be a relational exercise 8

Verbal Probing Asking questions immediately after a question is answered, or after the whole questionnaire is completed Many different types of probes Scripted and unscripted Concurrent and retrospective General and specific Redirection or elaboration Redirection probes should be asked prior to elaboration probes Works better when more specific information is needed 9

Advantages & Disadvantages Think-aloud Less interviewer bias Less interviewer training Open-ended format – Need to subject training Less control More difficult to tell if respondent can answer Potential bias in information processing Verbal Probing More control Topics Depth Little training of subject – Greater risk of reactivity Potential for bias Need for interviewer training 10

Problems (Conrad and Blair, 1996) Vocabulary (Lexical) Not knowing the meaning of a word or expression Words or concepts that have different connotations in one culture than another Inclusion/Exclusion Not knowing or variation in what is included in a group Time (Temporal) Not understanding the time period referenced Not knowing or being able to estimate the amount of time spent on an activity 11

Problems (Conrad and Blair, 1996) Logical Issues related to presuppositions and connected items (e.g., double barreled questions) Computational Issues related to memory demand and overall processing… a bit of a catch all category Translation Words or concepts that are present in one language, but not another The improper translation from one language to another 12

Why do we care? Response errors lead to lower reliability Validity is called into question Construct validity Content validity Intentional validity Better survey questions lead to better results 13

So let’s look a little closer at Verbal Probing Examples 14

Verbal Probes (Beatty, 2004) Cognitive Probes Computational processes Level of difficulty in answering Interpretation of terms Confirmatory Probes Accuracy of information provided “What were you thinking about while answering that?” “How did you come up with your answer?” “What does [term] mean to you?” “So for the last 14 days you did not drive aggressively?” 15

Verbal Probes (continued) Expansive Probes Additional detail Better explanation Functional Remarks Redirection to obtain answer Clarifications Feedback Reinforce response behavior Conversation bridges “Tell me more about how you do your homework.” “When did you decide to tell someone?” “And how many times did you do that?” “Yes, including just one puff of smoke.” “Thanks, that’s useful information.” “I know answering some questions can be tough.” 16

Verbal Probes (continued) Contextual Probes Obtain frame of reference Investigation of terms Probable strategies for action “Tell me about the last time your children had to stay home because of illness? What happened?” “Can you tell me about a time when you used drugs topically?” “In your mind, what doesn’t count as ‘having sex’?” “Do you know anyone who was forced to have sex? Could you tell me about it?” 17

Why Contextual Probes? More control and focus than think-aloud Less demand on subjects than explanation and definition Less reliance on subjects knowledge and forthrightness Find possible sources of error/misunderstanding You can relay subjects’ thoughts without having to know what the designer/client thinks Contextual probes don’t replace all other probes, but are a complement to them 18

Test – What contextual probes can you create for the following? Q1: Which group does the majority of your active patient population belong to? Q2:Did you refer any of your of colleagues to the campaign materials? Q3: Do you consider yourself to be male, female, or transgender? Q4: In the past 12 months, have you ever lived as a woman?  White, non-Hispanic  Asian/Pacific Islander  White, Hispanic  Native American/Alaskan native  African-American, non-Hispanic  Multiracial  African-American, Hispanic  Unsure 19

Future Research Cognitive interviews as the focus of the research Comparisons between cognitive interview techniques Comparisons between types verbal probes Number and types of problems Information provided by contextual probes versus cognitive or expansive probes Introduction of more theories of human behavior 20