Cognitive Testing: Process and Results Roger Levine, Ph.D. American Institutes for Research (AIR) CAHPS AIR Team

Cognitive Interviewing Interviews intended to provide insights into the respondent’s thought processes as they read (or hear) and respond to questions.

Purposes of Cognitive Testing Determine if survey items are working as intended – Will they elicit valid information? – Will data be useful analytically? Response variance Can also be used to test and refine cover letters and telephone introductions

Advantages of Cognitive Interviewing Enables the detection of unexpected problems Provides an underlying model of the questionnaire item response process –This model not only facilitates detection of problems, but also informs the item refinement process

Model of the Questionnaire Item Response Process Comprehension Retrieval –Where were you on the night of May 7, 2000? Response formation/Response synthesis –How many times have you seen a doctor in the last two years? Response selection

Cognitive Interviewing Tools Think-alouds Asking questions Observations Any, or all, of the above may be used

Consent Form Will you be recording me today? Yes. With your permission, we plan to videotape the interview today. Your responses will be strictly confidential and you have the right to ask us to stop videotaping at any time. The tape may be edited and shown to others for informational and instructional purposes. We will erase any parts of the interview that you do not want other people to see. Also, any information that you provide that is shared with people outside of the project team will not reveal your name.

Selected Findings (1) Misunderstanding of key concepts –Kidney doctor –Treatment choices –Health provider –Dialysis center staff

Selected Findings (2) Dialysis center staff (DCS): intended to include only non-M.D. care staff –DCS defined by focal behavior MDs included in items dealing with “talking about access site care,” “talking about kidney transplant,” “listening carefully to you” MDs, drivers, secretaries, receptionists included in item about overall satisfaction with the DCS Items about “inserting needles without causing too much pain or discomfort” interpreted properly

Selected Findings (3) “Dialysis center” means “facility” to some and “facility and staff” to others “Weighed out within 15 minutes after your dialysis was over” was confusing due to different interpretations of “after your dialysis was over” and unfamiliarity with the term “weighed out”

General Findings (1) For “long-timers,” many items were not appropriate indicators of their satisfaction nor of quality of service. –Getting as much information about dialysis as you wanted –DCS talking to you about access site care –DCS giving you as much information as you wanted about taking care of kidney disease –DCS talking to you about treatment choices

General Findings (2) Time reference periods worked fairly well “Yes/No” items generally were easier to answer than “Never/sometimes/usually/ always” items 0 – 10 rating scale worked fairly well

Summary Learned a great deal Items are being revised Another round of cognitive testing is planned