1. Methodology of Disability Statistics EDSIM testing evaluation project
Department of Health Sciences
Methodology of Disability Statistics
EDSIM testing evaluation project
Revision progress
Amanda Wilmot and Howard Meltzer
Eurostat, Luxembourg, 28/29June 2010

2. Project objective
To produce an updated version of the EDSIM (questionnaire and explanatory notes) based on:
synthesis and analysis of test results from 10 Member States
comments from DG Employment
recommendations from Nordic report
suggestions by TF members

3. Testing in 10 Member States
Cognitive test
Czech Republic
Estonia
Finland
Greece
Hungary
Latvia
Slovak Republic
Spain
Pilot test
Bulgaria
Czech Republic
Estonia
Greece
Hungary
Latvia
Malta

4. Objectives of the testing
To highlight any translational issues
To highlight questions in the module which caused difficulties
To examine whether the questionnaire as a whole caused difficulties
To make recommendations for possible improvement of the questions and/or questionnaire

5. Evaluation Methodology
Qualitative approach to analysis and reporting
Collation and synthesis presented in Excel spreadsheets
Two stages:
Methodology used by MS
Reported findings synthesised (section by section, question by question)

6. Consideration of additional questions with reference to:
Stand-alone survey
Consideration of additional questions with reference to:
16 Core Social Variables
MEHM from EU-SILC
Health, basic activity questions from 2011 LFS disability module
Health, basic activity, ADL and IADL questions from EHIS

7. Key findings: Generally positive comments
“ Generally questions easily understood for both disabled and nondisabled respondents. Answer categories were comprehensive” [Bulgaria]
“In general the questionnaire was suitable and interviewers mentioned that respondents were responsive” [Latvia]
“ Interviewers mentioned that the questionnaire was pretty straightforward and most interviews did not take long to complete” [Malta]
“ The pretest revealed that generally the questions worked well and respondents were actively interested in the questionnaire” [Spain]
“None of the respondents felt themselves uncomfortable while answering and none of the EDSIM questions were perceived as too sensitive to answer “ [Finland]

8. Key findings: Sensitivity
Issue: Respondents in some MS uncomfortable with certain sections
Action: Interviewer training important - some guidance included in interviewer instructions

9. Key findings: Conceptual issues (1)
Survey concept
Issue: Not sufficiently conveyed
Action: Improve guidance, title and question formulation

10. Key findings: Conceptual issues (2)
Concepts of ‘activity limitation’ and ‘health condition, illness or disease’
Issue: Too complex and not sufficiently conveyed
Action: Review concepts and improve guidance, titles and question formulation

11. Key findings: Conceptual issues (3)
Section concepts
Issue: Too complex and not sufficiently conveyed
Action: Review concepts and improve guidance, titles and question formulation

12. Key findings: Section introductions
Issue: Clear and concise introductions or preambles should be provided each time there is a change of topic/section
Action: Introductions added/reviewed
Example: ‘This section is about how easy or difficult you are finding it to pay for the essential things in life such as food, clothing, medicine, housing and transport.’ [Economic life]

13. Key findings: Section order (1)
Issue: Change section order
- to engage respondents as quickly as possible
- to take account the relative importance of topics
- to take account of section order effects
- to improve the flow of interview

14. Key findings: Section order (2)
Action: Section order changed to:
1 Mobility
2 Transport
3 Accessibility to buildings
4 Education and training
5 Employment
6 Internet use
7 Social contact and support
8 Leisure pursuits
9 Economic life
10 Attitudes and behaviours

15. Key findings: Standardisation
Barrier question and response codes
‘Is there anything which prevents you from... whenever you want to?
Please use this card as a guide and choose all that apply.’
Financial reasons (lack of money, can’t afford it)
Too busy (with work, family, other responsibilities)
Lack of knowledge or information (such as don’t know area, lack of street signs)
A health condition, illness, or disease
Difficulties with basic activities (such as seeing, hearing, concentrating, moving around)
Lack of convenient or available transport
Unsuitable surroundings (such as hills, slopes, steps, footpath design)
Difficulties accessing or using buildings
Lack of self confidence or attitudes of other people
Other reasons
No, nothing prevents me from ...whenever I want to

16. Key Findings: Supplementary questions
‘Lack of special aids or equipment’ and ‘lack of personal help or assistance’ only asked of those indicating a health condition or activity limitation as a barrier.
Example:  APPLIES IF EdPrv = 4 or 5 ‘health condition or difficulties with basic activities’
[*] May I just check, does the [lack of special aids or equipment] [lack of personal help or assistance] prevent you from studying (for a qualification) at the moment?
Yes/No [Education]

17. Routing Review of questionnaire routing conducted
Example: Employment: APPLIES IF Self declared labour status NE 31 ‘students’, 32 ‘retired’ or 34 ‘in compulsory military or community service’

18. Key findings: Length and complexity
Aim: to reduce the length and complexity of the questioning where indicated

19. Example of reduction: Leisure pursuits
“From the entire module it was this set of questions that most confused respondents” [Hungary]
“Answering questions in this section was time consuming as respondents took longer to process the meaning of given responses and the differences between them ...Reducing the number of questions in this section is suggested as respondent often did not see much difference between them “
[Czech Republic]
“The number of questions in this section could be reduced ...The answer options involve two concepts: frequency and opinion of that frequency. It’s difficult for respondents to choose an option“ [Italy]
“Consider reducing the number of activities asked about in this section” [Nordic project]
“Problems with understanding answer categories”
[Finland)

20. Example of reduction: Leisure pursuits
Intro: Leisure pursuits
‘The next section is about how you spend your leisure time.’
Intro: Hobbies and interests
‘I would like to ask you about hobbies or interests that involve spending time with other people. For example, belonging to a club or association, or taking part in sporting and fitness activities.’
Question: HobPrv
[*] ‘Is there anything which prevents you from pursuing hobbies or interests whenever you want to? (Please use this card as a guide and choose all that apply.)’

21. Summary Improved comprehension and reduced burden by:
Reviewing section concepts
Reducing overlap
Changing the section order
Standardising questions and terminology where appropriate
Reviewing the questionnaire routing
Reducing the length and complexity of the questions and sections
Providing more information and clarification to interviewers

22. Conclusion
The approach of asking about lack of social integration and the relationship with health and non-heath factors has been retained
Findings from testing taken on board to produce a simplified and more coherent questionnaire that can be implemented cross-nationally