United Nations Economic Commission for Europe Statistical Division The problem of identifying persons with disabilities – the importance of questionnaire design Angela Me, Chief Social and Demographic Statistics Section

- UNECE Statistical Division Bishkek, December 2006 Slide 2 Challenge  How to measure a wider experience of disability through a limited number of questions?

- UNECE Statistical Division Bishkek, December 2006 Slide 3 Design an instrument to identify the defined population with disability  The difficult part is: To logically convert/translate objectives into measurement instruments and to link definitions with questions

- UNECE Statistical Division Bishkek, December 2006 Slide 4 Design an instrument to identify the defined population with disabilities - a Census Example  Definition: Any restriction or lack (resulting from an impairment) of ability to perform an activity in the manner or within the range considered normal for a human being  Question: Is there anybody in the household who is disabled?

- UNECE Statistical Division Bishkek, December 2006 Slide 5 Definition-measurement instrument BFS limitations Activity limitations Paralyzed Deaf …..

- UNECE Statistical Division Bishkek, December 2006 Slide 6 Design an instrument to identify the defined population with disabilities - a Census Example Despite a definition based on activity limitations, the questions identified only persons with most severe impairments 2.5%

- UNECE Statistical Division Bishkek, December 2006 Slide 7 From theory to practice  Estimates of prevalence of disability are highly sensitive to the measures used

- UNECE Statistical Division Bishkek, December 2006 Slide 8 Measurement Issues  Design of the study Method of data collection Question design (wording, place, length, …) Interview process: o interviewer effect o Respondent effect  Socio-cultural Determinants

- UNECE Statistical Division Bishkek, December 2006 Slide 9 US Survey Example The following questions and results were obtained in an American survey % 'Yes' 53 Have you ever heard of the Taft-Pepper Bill concerning veteran's housing (no such bill!) Have you ever heard of the famous writer, John Woodson? (no such writer!) Have you ever heard of the Midwestern Life Magazine? (no such magazine!) Do you recall that, as a good citizen you voted last December in the special election for your state representative? (no election!) 8 Have you ever heard the word AFROHELIA? (no such word!)

- UNECE Statistical Division Bishkek, December 2006 Slide 10 Example: Australian Survey Average number of sex partners reported By women who were watched as they filled in their survey answers: 2.6; By women who knew they were completely anonymous: 3.4; By women who thought they were attached to a lie detector: 4.4 Sydney Morning Herald, August 31, 2003

- UNECE Statistical Division Bishkek, December 2006 Slide 11 Wording  The most detailed disability survey, using a carefully designed and relatively complete set of questions covering a wide range of topics, is limited when the initial questions used to identify the persons with disability is poorly designed

- UNECE Statistical Division Bishkek, December 2006 Slide 12 Developing instruments to identify persons with disabilities Disability is a dynamic complex related to: Individual attributes Environment Time Two persons with the same impairment may have a different perception of disability

- UNECE Statistical Division Bishkek, December 2006 Slide 13 Issues that we need to consider Particular attention is needed to measure disability through an interview process People may be unwilling to talk about their problems Difficulty in defining what is meant by disability and its various aspects Stigma

- UNECE Statistical Division Bishkek, December 2006 Slide 14 Issues that we need to consider In an interview process: Easier to measure activity limitations (day- to-day activities) and participation

- UNECE Statistical Division Bishkek, December 2006 Slide 15 Developing instruments to identify the complexity of disability  Requirement: Multiple questions to set context, clarify terminology, define multiple domains  Resource availability Short questions

- UNECE Statistical Division Bishkek, December 2006 Slide 16 Developing instruments to identify the complexity of disability  Long instruments/modules High number of questions, more opportunities to capture the different dimensions, intensity  Short instrument/modules 1-5 questions to identify persons with disabilities Careful design of the question(s) to make sure that all persons with disabilities that we want to identify can indeed be properly identified

- UNECE Statistical Division Bishkek, December 2006 Slide 17 What defines a good instrument  The instrument measures the concept it is supposed to measure (Validity- Accuracy)  Repeated measurements of the same instruments give the same results (Reliability-Precision)

- UNECE Statistical Division Bishkek, December 2006 Slide 18 Design an instrument to identify the target population- A Census Example  Definition: Any restriction or lack (resulting from an impairment) of ability to perform an activity in the manner or within the range considered normal for a human being  Question: Is there anybody in the household who is disabled?

- UNECE Statistical Division Bishkek, December 2006 Slide 19 An example: U.K. Census 1991  Do you have any long-standing illness, health problem or handicap which limits your daily activities or the work you can do? Include problems which are due to old age

- UNECE Statistical Division Bishkek, December 2006 Slide 20 Conditions that effect the output of a question  Wording  Context  Self/Proxy  Response categories  Mode of data collection  Method of data collection  Overall survey topics  Survey sponsor

- UNECE Statistical Division Bishkek, December 2006 Slide 21 Wording: what does affect comprehension?  Ambiguous syntax  Complicated syntax  Unfamiliar terms  Vague concepts  Assumptions about respondent’s knowledge

- UNECE Statistical Division Bishkek, December 2006 Slide 22 Wording Language: Clear Unambiguous Simple Terms such as long-term, disabilities, handicaps are viewed as extremely negative and tend to underreport disabilities (Langlois, 2001)

- UNECE Statistical Division Bishkek, December 2006 Slide US Census Because of a physical, mental, or emotional condition lasting 6 months or more, does this person have any difficulty in doing any of the following activities… d. Working at a job or business?  Multiple health domains (explicit enumeration)  Duration  “Difficulty”  capacity  participation  “Working”

- UNECE Statistical Division Bishkek, December 2006 Slide 24 Question components for a short question  Preamble  Health condition (as cause)  Duration (long/short term disability?)  ICF domain  Functioning  Activities  Participation

- UNECE Statistical Division Bishkek, December 2006 Slide 25 Question components for a short question Keep it relevant and valid but SIMPLE  If possible split the different components of the measure into different questions

- UNECE Statistical Division Bishkek, December 2006 Slide 26 Question components for a short question Introduction: make the respondent think about the outputs of an health condition and set the duration (conditions that last for 3 months, 6 months, 12 months, …) Depending on the domain we want to identify:  Do you have difficulties walking?  Do you have difficulties concentrating?  Do you have difficulties participating in daily activities  …..

- UNECE Statistical Division Bishkek, December 2006 Slide 27 Question components for a short question Questions based on activities and participation rather than impairments provide a broader view of disability and they facilitate the identification of persons with disabilities by providing a more simple and natural language

- UNECE Statistical Division Bishkek, December 2006 Slide 28 Response process: judgment and response formulation  Evaluation of retrieved information Perception of accuracy Motivation  Evaluation of response options  Communication of response

- UNECE Statistical Division Bishkek, December 2006 Slide 29 Response categories  Response categories set the context of the question  Avoid to force the respondent to identify him/herself with a socially-defined label (stigma)  Scale Response instead of a dichotomy None/A little/A lot Yes, sometimes/Yes, often/No

- UNECE Statistical Division Bishkek, December 2006 Slide 30 Response categories  It has been proven that scaled responses improve the respondents’ ability to report having disabilities (Statistics Canada, Austrian Bureau of Statistics, Research in the USA)  If the respondent can not choose among multiple dimensions, he/she is likely to misreport his/her disability status Disability is not a yes/no phenomenon but rather a status that varies on a continuum in terms of intensity and time

- UNECE Statistical Division Bishkek, December 2006 Slide 31 Mode of data collection  Self reported or Proxy?  Avoid if possible proxy responses  The disability process relates to the individual’s experience and can be accurately described only by the individual him/her self

- UNECE Statistical Division Bishkek, December 2006 Slide 32 Type of question  Avoid household-based questions (Is there anybody in the household who has difficulties walking?)  Use person-based question (Do you have difficulties walking?)  It has been proven that person-based questions identify more persons with functional limitations (USA, UN)

- UNECE Statistical Division Bishkek, December 2006 Slide 33 Context  Place of the question in the questionnaire  Example: disability and economic characteristics questions  Introduction to the question