1. MEADOW Questionnaire Validation G. Sirilli & A. Arundel

2. MEADOW questionnaires Designed for computer assisted telephone interviews (CATI) No more than 20 to 30 minutes per interview Two separate questionnaires –Employers –Employees

3. Goals of question validation 1.Almost all respondents understand each question in the way that it is intended to be understood Includes consistency of understanding across countries 2.A large majority of respondents can answer each question Varies by the questionnaire method – mailed, by telephone, etc. 3.Minimize response error 1.Identify appropriate response scales, remove redundant questions, identify questions susceptible to ‘satisficing’ responses, etc.

4. Validation methods Testing among MEADOW participants: –Can identify many problems, but ‘expert’ familiarity often results in a failure to identify problems Cognitive testing: face-to-face interviews of a variety of types of respondents. –Random sampling of respondents is not needed –Ask respondents how they understand each question, possible problems with answering, and probe for understanding of terms. Pilot survey: Random survey of target population under normal conditions. –Provides better data on ability of respondents to reply to each question, checks for logical inconsistencies, etc. –Provides better data on response option categories

5. MEADOW testing Cognitive testing through 247 interviews (approximately 1 hr each) Employer questionnaire: –118 interviews in 11 countries Employee questionnaire: –129 interviews in 11 countries Questionnaires tested in Denmark, Finland, France, Hungary, Germany, Italy, Netherlands, Sweden, Ukraine, UK, and US.

8. Two rounds of interviews Phase 1: Approximately 2/3rds of interviews –Due to questionnaire length, each questionnaire divided into two logically consistent versions (A and B) and tested separately –Identified problem questions, questions revised Phase 2: 1/3 of interviews –Only retested one version consisting of revised questions –Questions revised (slightly) again if needed or deleted

9. Examples of deletions Questions on the use of specific software systems (performance tracking software, etc). –most respondents were unfamiliar with the software; can only be asked of IT specialists Question on policy of guaranteed job security for groups of employees –Respondents strongly believed that no jobs today are guaranteed –question no longer relevant.

10. Question revisions Revisions extensive: –Employer questionnaire: approximately 25 questions revised (out of 90) –Employee questionnaire: approximately 20 questions revised (out of 115) In addition, many minor changes to response categories and wording.

11. Conclusions Extensive cognitive testing: 247 interviews instead of the norm of 20 – 30 – partly required to ensure consistency in responses across countries and to test two questionnaires –Several questions deleted plus many revisions Result: much improved question reliability Next step: Pilot survey