1. Using Context to Evaluate Survey Items
Presented by Michael Burke, MPH, PhD
RTI International Presented at AEA Evaluation Orlando, FL
Phone • Fax •

2. Overview Background What is a cognitive interview? Methods
Advantages/disadvantages
Types of problems found
Types of verbal probes
Why we care
Test
Future research

3. Pretesting Surveys
Pretesting has been around since the 1940s or earlier
Typically a dry run of the survey with a smaller sample size
Look for non-response, examine distributions, discuss the survey with interviewers, etc.
Cognitive interviewing has been around since the 1980s 3

4. Cognitive Interviews
Arose from dialogue between survey researchers and cognitive psychologists
Typically assume a 4 step cognitive process*
Interpretation and comprehension
Retrieval of pertinent information
Integration and evaluation of retrieved info (judgment)
Selection of a response category (reporting)
Cognitive interviewing has not greatly expanded to include other academic disciplines and theories of human behavior
*Tourangeau, 1984
Cognitive Aspects of Survey Methodology conference in 1984
Census Bureau, Bureau of Labor Statistics, and academic and commercial research organizations have cognitive laboratories. Census had a lab before the 1980 census to evaluate items, but the orientation for the laboratory came from marketing research rather than cognitive psychology. One of the methods used was talk-aloud as they filled out questionnaires. Presser et. al 2004. 4

5. What is a cognitive interview?
Interview designed to reveal some of the mental processes elicited by questions
Method to examine how respondents perceive and interpret questions and detect possible sources of response error
Method to understand the mental processes involved in responding to a given message that may result in miscommunication
A method of increasing recall of an event for the purpose of providing information to law enforcement (Fisher and Geiselman, 1992) 5

6. Criticisms Lack of standardization Definition of a cognitive interview
Methods employed
Means of analysis
Little methodological research to determine relative effectiveness of various methods
Focus on finding problems with questions and meaning of responses rather than on cognitive processes 6

7. Cognitive Interview Methods
Asking a respondent to think-aloud and say what you are thinking (see Ericsson and Simon 1984, 1993, 1998)
Verbal probing by an interviewer to reveal or clarify the subject’s thinking (see Willis, 1994, 1999, 2005)
Cognitive probes
Confirmatory probes
Expansive probes
Functional remarks
Feedback (Beatty, 2004)
Contextual probes 7

8. Think-aloud Asking the participant to verbalize their thinking
Works better:
verbal information (not spatial or nonverbal)
problem oriented questions
conscious processing of information
Less effective:
understanding of terms
May rely on short term memory recall
Should not be a relational exercise 8

9. Verbal Probing Asking questions
immediately after a question is answered, or
after the whole questionnaire is completed
Many different types of probes
Scripted and unscripted
Concurrent and retrospective
General and specific
Redirection or elaboration
Redirection probes should be asked prior to elaboration probes
Works better when more specific information is needed 9

10. Advantages & Disadvantages
Think-aloud
Less interviewer bias
Less interviewer training
Open-ended format –
Need to subject training
Less control
More difficult to tell if respondent can answer
Potential bias in information processing
Verbal Probing
More control
Topics
Depth
Little training of subject –
Greater risk of reactivity
Potential for bias
Need for interviewer training 10

11. Problems (Conrad and Blair, 1996)
Vocabulary (Lexical)
Not knowing the meaning of a word or expression
Words or concepts that have different connotations in one culture than another
Inclusion/Exclusion
Not knowing or variation in what is included in a group
Time (Temporal)
Not understanding the time period referenced
Not knowing or being able to estimate the amount of time spent on an activity 11

12. Problems (Conrad and Blair, 1996)
Logical
Issues related to presuppositions and connected items (e.g., double barreled questions)
Computational
Issues related to memory demand and overall processing… a bit of a catch all category
Translation
Words or concepts that are present in one language, but not another
The improper translation from one language to another 12

13. Why do we care? Response errors lead to lower reliability
Validity is called into question
Construct validity
Content validity
Intentional validity
Better survey questions lead to better results 13

14. So let’s look a little closer at Verbal Probing
Examples 14

15. Verbal Probes (Beatty, 2004)
Cognitive Probes
Computational processes
Level of difficulty in answering
Interpretation of terms
Confirmatory Probes
Accuracy of information provided
“What were you thinking about while answering that?”
“How did you come up with your answer?”
“What does [term] mean to you?”
“So for the last 14 days you did not drive aggressively?” 15

16. Verbal Probes (continued)
Expansive Probes
Additional detail
Better explanation
Functional Remarks
Redirection to obtain answer
Clarifications
Feedback
Reinforce response behavior
Conversation bridges
“Tell me more about how you do your homework.”
“When did you decide to tell someone?”
“And how many times did you do that?”
“Yes, including just one puff of smoke.”
“Thanks, that’s useful information.”
“I know answering some questions can be tough.” 16

17. Verbal Probes (continued)
Contextual Probes
Obtain frame of reference
Investigation of terms
Probable strategies for action
“Tell me about the last time your children had to stay home because of illness? What happened?”
“Can you tell me about a time when you used drugs topically?”
“In your mind, what doesn’t count as ‘having sex’?”
“Do you know anyone who was forced to have sex? Could you tell me about it?” 17

18. Why Contextual Probes? More control and focus than think-aloud
Less demand on subjects than explanation and definition
Less reliance on subjects knowledge and forthrightness
Find possible sources of error/misunderstanding
You can relay subjects’ thoughts without having to know what the designer/client thinks
Contextual probes don’t replace all other probes, but are a complement to them 18

19. Test – What contextual probes can you create for the following?
Q1: Which group does the majority of your active patient population belong to?
Q2:Did you refer any of your of colleagues to the campaign materials?
Q3: Do you consider yourself to be male, female, or transgender?
Q4: In the past 12 months, have you ever lived as a woman?
 White, non-Hispanic  Asian/Pacific Islander
 White, Hispanic  Native American/Alaskan native
 African-American, non-Hispanic  Multiracial
 African-American, Hispanic  Unsure 19

20. Future Research Cognitive interviews as the focus of the research
Comparisons between cognitive interview techniques
Comparisons between types verbal probes
Number and types of problems
Information provided by contextual probes versus cognitive or expansive probes
Introduction of more theories of human behavior 20