1. Washington Group Cognitive Test Kristen Miller Questionnaire Design Research Lab National Center for Health Statistics, USA 1) Purpose of cognitive test 2) Structure of protocol 3) Types of cognitive test data & possible analyses 4) Things to keep in mind

2. Purpose of Cognitive Testing Interviews: Designed to examine the stages of the question response process – Comprehension, retrieval, judgment, response Analysis: Identifies… – potential response errors: (e.g. vague concepts, recall problems, inadequate response options) – patterns of interpretation

3. Traditional Cognitive Testing Protocol Semi-structured Data collected are qualitative Qualitative methodologies for analysis Small samples: approx. 10 – 30 participants

4. Washington Group Protocol Structured Questionnaire Designed to illustrate: 1. Administration ease 2. Interpretations 3. Factors considered by respondent 4. Degree of consistency with physical abilities 5. Differences between subgroups

5. Cognitive Interview Protocol I. Demographic Section: Country, language, gender, age, SES II. Question Testing Sections A. Core Question B. Interviewer Coding C. Open-ended Follow-up Probe D. Cognitive Follow-up Probe E. Functioning Follow-up Probe III. Health Questions: subjective health, chronic condition list IV. Interviewer Debriefing

6. Interviewer Code: Did the respondent… need you to repeat any part of the question? Yes/No have any difficulty using the response options? Yes/No ask for clarification or qualify their answer? Yes/No

7. Percent of respondents with no response problems Vision83.89% Hearing88.47% Cognitive85.72% Mobility91.03% Self-care93.65% Communication86.70% N=1,287

8. Number of response problems by age 01 or 23+n 18-2558.29%12.56%29.15%199 26-3058.2618.3922.99174 31-4069.4214.0516.53242 41-5061.5419.4619.00221 51-6064.4317.4518.12149 61+57.6715.8726.46189 1174 Chi-square=19.74, p<.05

9. Open-ended Follow-up Probe 1. Can you tell me what you were thinking? Why did you answer [fill with respondent’s answer to the core question]?

10. Qualitative Data Varied in quality A lot missing Very brief Provides some insight – E.g. “Because I wear glasses”

11. Cognitive Follow-up Probes: 1. How often they had difficulty 2. The amount of effort they needed to put into the activity 3. The degree of concern or worry they had regarding the difficulty 4. Other domain-specific questions 1. Communication: Shyness question 2. Cognitive: Reason question

12. Cognition Cognitive Probe Do you believe that your memory or concentration difficulties are… [Mark all that apply]  because you have too many things to do?  because you getting older?  or, because of something else? Only busy: 28.78% Aging:25.84% Something else:43.07%

13. Reason by degree of problem SometimesA lotCan’t don Only busy89.05%10.95%0.00%137 Aging77.2422.760.00123 Something Else 73.6624.401.95205

14. Functioning Follow-up Probes 1. Presence of a condition 2. Use of an assistive device 3. Degree of difficulty doing related activities

15. Analysis of Functioning Data Examine the consistency between: – Washington Group Question responses and – Follow-up functioning questions responses Goal: explain the discrepancies – Misunderstood word? – Cultural difference? – Translation problem? – Interviewer error? – Error in the follow-up questions? Other data assist in explanation

16. Response patterns: Vision PatternWears Glasses Functioning Problem Reports Disability ANo BYesNo (corrected)No C YesYes, disability DYesYes (not corrected)Yes, disability EYesNo (corrected)Yes, disability FNo Yes, disability GYesYes (not corrected)No H YesNo

17. Response patterns: Hearing PatternAidDisabilityMissed WordsFunctioning Problem ANo B Yes CNo YesNo D Yes ENoYesNoYes FNoYes No G Yes HNoYesNo IYesNoYes J K No LYes NoYes M No NYesNo Yes O NoYesNo PYes No