1. SPECA Regional Wrokshop on Disability Statistics, Dec 13-15, 2006 Issues Related to Disability Measurement: Cognitive testing and mode Jennifer Madans for the Washington Group on Disability Statistics

2. Part 1: Cognitive testing Adapted from work of Kristen Miller, NCHS, USA

3. Cognitive testing Designed to help us understand how respondents comprehend, retrieve, judge, respond to questions Designed to help us understand how respondents comprehend, retrieve, judge, respond to questions Illustrate how aspects of respondents’ lives impact their understanding of the question Illustrate how aspects of respondents’ lives impact their understanding of the question Through this examination, can identify Through this examination, can identify –Potential response errors –Patterns of interpretation

4. Components of the process A good question is: Relevant to the research agenda Relevant to the research agenda Relevant to each potential respondents’ experience and knowledge Relevant to each potential respondents’ experience and knowledge Respondents: Are informants Are informants Reference aspects of their lives Reference aspects of their lives Do not know or understand the research question Do not know or understand the research question Do not use scientific or abstract concepts Do not use scientific or abstract concepts

5. Response errors When questions are not optimal or the respondent does not understand, error is introduced into the data When questions are not optimal or the respondent does not understand, error is introduced into the data –False positives –False Negatives –An entirely different phenomena is measured than intended by the research agenda

6. Cognitive stages: responding to questions Comprehension– the respondent interprets the question Comprehension– the respondent interprets the question Retrieval– the respondent searches memory for relevant information Retrieval– the respondent searches memory for relevant information Judgment– the respondent assesses the completeness and relevance of memories and makes an estimation Judgment– the respondent assesses the completeness and relevance of memories and makes an estimation Response: Maps judgment onto response category; may need to edit response to fit the category Response: Maps judgment onto response category; may need to edit response to fit the category

7. Cognitive response problems Respondent cannot remember the correct answer Respondent cannot remember the correct answer –“In the past month, how many times did you eat rice?” Respondent makes an error when forming an estimate Respondent makes an error when forming an estimate –“What is the total value of all your financial assets?” Respondent’s response does not match response categories Respondent’s response does not match response categories –“When you go to work, do you drive, bicycle or walk?”

8. Socio-cultural factors impact question response Level of education Level of education Socio-economic status Socio-economic status Cultural beliefs Cultural beliefs Racial, ethnic, gender identities Racial, ethnic, gender identities Access to knowledge and resources Access to knowledge and resources

9. Translation Problems Bad literal translations Bad literal translations Words vary by region Words vary by region Formal words may not be understood Formal words may not be understood Multiple meanings of words Multiple meanings of words

10. Cognitive Interviews Semi-structured Semi-structured Core questions Core questions –the questions that will be tested –interview organized by the core questions Probe questions Probe questions –the questions that will provide information about respondents’ understanding of the core questions –Probe questions are open-ended, spontaneous, not pre-scripted, based on information that respondent provides

11. Cognitive Interviews Data are qualitative Data are qualitative Qualitative methodologies used to analyze data Qualitative methodologies used to analyze data Small samples of roughly 10 – 30 participants are used Small samples of roughly 10 – 30 participants are used

12. Probes Probing for story: Why did the respondent answer the question the way that they did? Why did the respondent answer the question the way that they did? Does this story match with the intent of the question? Does this story match with the intent of the question? Common probes: Why did you answer that way? Why did you answer that way? Can you tell me about that? Can you tell me about that? When the question asks about “difficulty walking,” how are you understanding that? When the question asks about “difficulty walking,” how are you understanding that?

13. Most Common Probes How so? How so? In what way? In what way? Can you say more? Can you say more? What do you mean by that? What do you mean by that? [Silence] [Silence]

14. Issues related to disability Conceptual confusion around disability Conceptual confusion around disability Cognitive testing allows you to gain insight into what respondent is thinking and whether this varies by socio-demographic characteristics such as education, age, and gender Cognitive testing allows you to gain insight into what respondent is thinking and whether this varies by socio-demographic characteristics such as education, age, and gender

15. Part 2: Mode of data collection Mail (paper and pencil) Mail (paper and pencil) Telephone Telephone In-person In-person Web-based Web-based

16. Issues related to disability Mail (paper and pencil) Mail (paper and pencil) –Literacy level –Insure availability of Braille, large print, or assistant if needed –Cannot clarify questions if respondent unclear Telephone Telephone –Presence of phone in the household may bias who responds –TTY/TDD or assistant for persons with hearing impairment –Can clarify questions if needed

17. Issues related to disability In-person In-person –May require assistant if respondent has communication or cognitive difficulties (allows for including respondents who would usually be excluded) –Sign language interpreter or alternate accommodations for persons with hearing impairments –Can offer additional explanation of question if needed Web-based Web-based –Presence of computer in the household may bias who responds –Bias in educational level of respondents who are able to respond to web-based survey

18. Issues related to disability Levels of interview assistance Direct personal interview: Respondent participates directly; no interpretation or facilitation is needed Direct personal interview: Respondent participates directly; no interpretation or facilitation is needed Interpreted interview: Someone interprets the questions to the respondent and interprets the responses back to the interviewer; interpreter is an intermediary Interpreted interview: Someone interprets the questions to the respondent and interprets the responses back to the interviewer; interpreter is an intermediary Facilitated interview: Someone else helps the respondent understand the questions and even answers some of the questions on their behalf Facilitated interview: Someone else helps the respondent understand the questions and even answers some of the questions on their behalf Proxy interview: Someone else completes the questionnaire for the respondent because he / she is unable to do so Proxy interview: Someone else completes the questionnaire for the respondent because he / she is unable to do so

19. Refusals and non- responses Disabled people may not be seen as valid respondents by family members or ignored and therefore not enumerated in the household Disabled people may not be seen as valid respondents by family members or ignored and therefore not enumerated in the household Not acceptable to have non-response on basis of disability in disability surveys - losing out on target population Not acceptable to have non-response on basis of disability in disability surveys - losing out on target population Training of interviewers and provision of support is crucial (e.g. sign language interpreters, assistants, Braille versions, etc.) Training of interviewers and provision of support is crucial (e.g. sign language interpreters, assistants, Braille versions, etc.)