Methodology of Disability Statistics EDSIM testing evaluation project Department of Health Sciences Methodology of Disability Statistics EDSIM testing evaluation project Revision progress Amanda Wilmot and Howard Meltzer Eurostat, Luxembourg, 28/29June 2010

Project objective To produce an updated version of the EDSIM (questionnaire and explanatory notes) based on: synthesis and analysis of test results from 10 Member States comments from DG Employment recommendations from Nordic report suggestions by TF members

Testing in 10 Member States Cognitive test Czech Republic Estonia Finland Greece Hungary Latvia Slovak Republic Spain Pilot test Bulgaria Czech Republic Estonia Greece Hungary Latvia Malta

Objectives of the testing To highlight any translational issues To highlight questions in the module which caused difficulties To examine whether the questionnaire as a whole caused difficulties To make recommendations for possible improvement of the questions and/or questionnaire

Evaluation Methodology Qualitative approach to analysis and reporting Collation and synthesis presented in Excel spreadsheets Two stages: Methodology used by MS Reported findings synthesised (section by section, question by question)

Consideration of additional questions with reference to: Stand-alone survey Consideration of additional questions with reference to: 16 Core Social Variables MEHM from EU-SILC Health, basic activity questions from 2011 LFS disability module Health, basic activity, ADL and IADL questions from EHIS

Key findings: Generally positive comments “ Generally questions easily understood for both disabled and nondisabled respondents. Answer categories were comprehensive” [Bulgaria] “In general the questionnaire was suitable and interviewers mentioned that respondents were responsive” [Latvia] “ Interviewers mentioned that the questionnaire was pretty straightforward and most interviews did not take long to complete” [Malta] “ The pretest revealed that generally the questions worked well and respondents were actively interested in the questionnaire” [Spain] “None of the respondents felt themselves uncomfortable while answering and none of the EDSIM questions were perceived as too sensitive to answer “ [Finland]

Key findings: Sensitivity Issue: Respondents in some MS uncomfortable with certain sections Action: Interviewer training important - some guidance included in interviewer instructions

Key findings: Conceptual issues (1) Survey concept Issue: Not sufficiently conveyed Action: Improve guidance, title and question formulation

Key findings: Conceptual issues (2) Concepts of ‘activity limitation’ and ‘health condition, illness or disease’ Issue: Too complex and not sufficiently conveyed Action: Review concepts and improve guidance, titles and question formulation

Key findings: Conceptual issues (3) Section concepts Issue: Too complex and not sufficiently conveyed Action: Review concepts and improve guidance, titles and question formulation

Key findings: Section introductions Issue: Clear and concise introductions or preambles should be provided each time there is a change of topic/section Action: Introductions added/reviewed Example: ‘This section is about how easy or difficult you are finding it to pay for the essential things in life such as food, clothing, medicine, housing and transport.’ [Economic life]

Key findings: Section order (1) Issue: Change section order - to engage respondents as quickly as possible - to take account the relative importance of topics - to take account of section order effects - to improve the flow of interview

Key findings: Section order (2) Action: Section order changed to: 1 Mobility 2 Transport 3 Accessibility to buildings 4 Education and training 5 Employment 6 Internet use 7 Social contact and support 8 Leisure pursuits 9 Economic life 10 Attitudes and behaviours

Key findings: Standardisation Barrier question and response codes ‘Is there anything which prevents you from... whenever you want to? Please use this card as a guide and choose all that apply.’ Financial reasons (lack of money, can’t afford it) Too busy (with work, family, other responsibilities) Lack of knowledge or information (such as don’t know area, lack of street signs) A health condition, illness, or disease Difficulties with basic activities (such as seeing, hearing, concentrating, moving around) Lack of convenient or available transport Unsuitable surroundings (such as hills, slopes, steps, footpath design) Difficulties accessing or using buildings Lack of self confidence or attitudes of other people Other reasons No, nothing prevents me from ...whenever I want to

Key Findings: Supplementary questions ‘Lack of special aids or equipment’ and ‘lack of personal help or assistance’ only asked of those indicating a health condition or activity limitation as a barrier. Example:  APPLIES IF EdPrv = 4 or 5 ‘health condition or difficulties with basic activities’ [*] May I just check, does the [lack of special aids or equipment] [lack of personal help or assistance] prevent you from studying (for a qualification) at the moment? Yes/No [Education]

Routing Review of questionnaire routing conducted Example: Employment: APPLIES IF Self declared labour status NE 31 ‘students’, 32 ‘retired’ or 34 ‘in compulsory military or community service’

Key findings: Length and complexity Aim: to reduce the length and complexity of the questioning where indicated

Example of reduction: Leisure pursuits “From the entire module it was this set of questions that most confused respondents” [Hungary] “Answering questions in this section was time consuming as respondents took longer to process the meaning of given responses and the differences between them ...Reducing the number of questions in this section is suggested as respondent often did not see much difference between them “ [Czech Republic] “The number of questions in this section could be reduced ...The answer options involve two concepts: frequency and opinion of that frequency. It’s difficult for respondents to choose an option“ [Italy] “Consider reducing the number of activities asked about in this section” [Nordic project] “Problems with understanding answer categories” [Finland)

Example of reduction: Leisure pursuits Intro: Leisure pursuits ‘The next section is about how you spend your leisure time.’ Intro: Hobbies and interests ‘I would like to ask you about hobbies or interests that involve spending time with other people. For example, belonging to a club or association, or taking part in sporting and fitness activities.’ Question: HobPrv [*] ‘Is there anything which prevents you from pursuing hobbies or interests whenever you want to? (Please use this card as a guide and choose all that apply.)’

Summary Improved comprehension and reduced burden by: Reviewing section concepts Reducing overlap Changing the section order Standardising questions and terminology where appropriate Reviewing the questionnaire routing Reducing the length and complexity of the questions and sections Providing more information and clarification to interviewers

Conclusion The approach of asking about lack of social integration and the relationship with health and non-heath factors has been retained Findings from testing taken on board to produce a simplified and more coherent questionnaire that can be implemented cross-nationally