1. LFS Workshop Reykjavik, May 2018
Swiss LFS multimode project CAWI (optimized for smartphone) / CATI
State of play, lessons learned and challenges involved
Sylvie Regli Greub

2. Content of the presentation
Context
Development process
Questionnaire
Main results of usability tests of the CAWI questionnaire
Lessons learned
Issues

3. Context continuous LFS since 2010
2 (2) 2 rotation scheme, 4 interviews in 1.5 year
voluntary, 4 languages
CATI only
person-base interview
120’000 interviews in a year
field outsourced

4. Context (cont.) response rate
good sampling frame, but difficulty to get fixed phone lines or to get in touch with people  coverage
new needs from participants
 change of mode: multimode CAWI / CATI
CAWI optimized for smartphones

5. Development process Get knowledge from literature and other surveys
Work on the questionnaire (content + formulation)
Choose technical aspects
Run first tests on questionnaire (CAWI / CATI)
Work on the field processes (multimode survey)
Run a pilot (wave 1, then wave 2)

6. Content Formulation Technical aspects
Questionnaire
Content
National needs
Users needs
New IESS reglementation
Formulation
Questions / Answers
Instructions
Multimode / Multilingual
Technical aspects
RWD
Design
Usability

7. Questionnaire: Paradigms for formulation
Master questionnaire: German vs French for the «nuances»
Formulation for CAWI, but having in mind the new CATI
Smartphone = small screens  no interviewer  instructions?
3 types of questions:
Those for which no new formulation is needed,
Those for which a small adaptation is needed,
Those for which a bigger adaptation is needed.

8. Questionnaire: playing with the design
1 character font for the question, same font for the answer lists, apart from special answer (grey and lower), if existing
other colour, smaller text for instructions

9. Questionnaire: playing with the design
Infobutton: only when people have a doubt

10. Questionnaire: more technical aspects
Don’t know options: to avoid the excessive use of “don’t know” “cannot say” or “don’t want to answer”, which we call don’t know options  not visible is the answer list, per default. If the person doesn’t answer a question and goes further, then a don’t know option appears, or an estimation question comes, depending on the type of question.
Hard checks: apart from filter questions, for which we really need an answer, we decided not to force any answer (even don’t knows)
Validation checks: only when the answer is not convenient (ex: improbable years).

11. Main results of usability tests of the CAWI questionnaire:
1 is for the worst evaluation, 7 for the best one. n = 101.

12. Lessons learned
As good results of the first round of test show, it was useful to consider the lessons learned from what literature says and from other already active surveys.
Tests show that the length of questionnaire remains reasonable in a CAWI mode (~22 minutes / shorter with smartphone than laptop or tablet).
It takes much time to develop such a project; resources are to be seriously taken into account.
Even though our external partner is a professional of the RWD, the LFS remains a challenging questionnaire to build in a smartphone CAWI mode.
Every single change  test on each device. Smartphones imply different browsers and screen sizes, which do no act the same way: what works on Android may not work with IOs.

13. Main issues and challenges
Formulation of questions Is not trivial when you have a multimode survey. Even if you begin with the CAWI questionnaire, you have to keep in mind that you will have a CATI version. This difficulty is enhanced in a country like in Switzerland, which is multilingual: a formulation, which works in German, may not work in French. Even if you use the same concepts or definitions, it has to be explained differently according to the language.

14. Main issues and challenges (cont.)
New IESS regulation:
undefinite planning: starts in 2020, 2021, ? + unfixed list of variables  not easy to coordinate with the project
implication of CAWI is minimized: model questionnaires proposed in the explanatory notes cannot be implemented in a CAWI mode
minimum (but still large) set of variables for all the members of the household  very difficult in a CAWI, 2 risks:
 dropouts
 PROXIES  bias

15. Issue: Effect of proxy-interviewing on a selection of indicators in the EU-LFS 2015
Switzerland strongly encourages Eurostat to reconsider the question of collecting information on every household member to open the door to CAWI. This position is also comforted by an analysis on Eurostat’s microdata that show that proxy-interviewing is to be suspected to bring a bias in results. Suspicion reinforced after having applied statistical regressions to control for selection effects.
Variables included in the regressions in addition to proxy: gender, citizenship, age group, country and - specifically for some indicators - status in employment, education, second job, full-time/part-time
All EU/EFTA countries but:
- Denmark, Finland, Iceland, Norway, Sweden, Switzerland: these countries do not survey all household members or only in a subsample => proxies are of a different kind
- Croatia, Germany, Ireland, Liechtenstein, Malta, Slovenia: not present in the micro-data
- Spain: important percentage of missings in the variable proxy/direct interview

16. Thank you for your attention!

17. Annexes

18. Questionnaire: Radio buttons vs Drop-down: variation of drop-down answers by browser

19. Collecting information on all household members:
May seem attractive in CAPI (face-to-face): budgetary reasons + proxies help limit the risk of low response rates or incomplete households.
Is already much less attractive in CATI: low cost gains, burden for people responding for all, risk of lower response rates and incomplete households.
Is a very bad idea in CAWI: dropouts highly related to duration of the questionnaire. Consequences of having to reply for 3-4 household members (1-2 hours)? Questions asked on other members should be limited to a few (e.g. some demographic data if not known from registers, ILO-status, occupation rate and education).