1. Adaptive mixed-mode design WP1
ROME
April 11th | 12th
MIMOD
Mixed-Mode Designs for Social Surveys
FINAL WORKSHOP
Adaptive mixed-mode design
WP1
Barry Schouten
Statistics Netherlands (CBS)

2. Adaptive mixed-mode survey design (ASD)
Adaptive survey design optimizes quality-cost trade-offs by differentiating effort to different (relevant) population strata.
In MIMOD, differentiation of effort is focussed at the choice of mode strategy per population stratum.
Objectives of WP1 mixed-mode ASD:
Make an inventory of ASD implementations in ESS countries;
Structure the decisions/steps towards an mixed-mode ASD;
Illustrate using two case studies;
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

3. WP1 MIMOD survey Findings from WP1 survey:
Mixed-mode ASD implemented only at Stat Netherlands;
ASD is a relatively unknown strategy to balance quality and costs. Eight countries indicated in survey they were unsure whether ASD is applied;
Potential reasons:
Implementation demands flexible case management system across modes;
Relatively weak available auxiliary data to stratify the population;
Mostly theoretical approach without many success stories;
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

4. ASD steps to implementation
Key ingredients of ASD:
Explicit quality and costs metrics;
Relevant auxiliary data;
Design features/interventions
In general: All possible elements of data collection strategy;
In MIMOD: Modes;
Optimization strategy, e.g.
Case prioritization;
Mathematical optimization;
Stopping rules based on quota
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

5. ASD steps to implementation – checklist
Identify priorities;
Identify major risks:
Consider risk of incomparability in time;
Consider risk of incomparability between subgroups;
Consider risk of budget overrun and heavy interviewer workloads in follow-up modes;
Define quality and cost indicators;
Consider nonresponse indicators;
Consider measurement error indicators;
Consider cost indicators;
Define decision rules from:
Trial-and-error;
Case prioritization;
Quota;
Mathematical optimization;
Modify the survey design and monitor the outcomes;
Develop a dashboard for survey errors;
Develop a dashboard for survey costs;
Compute estimates;
Document;
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

6. ASD case study – Health Survey/EHIS – priorities and risks
Key design feature: yes/no F2F follow-up to web nonrespondents
Main priorities:
Acceptable and similar response rates among relevant population subgroups
Sufficient precision on annual survey estimates
Costs satisfying a specified budget
Main risks:
Incomparability in time
Unpredictable CAPI workload due to varying monthly and annual web response rate
Incomparability between different population subgroups of interest
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

7. ASD case study – Health Survey/EHIS – quality and costs
Objective:
Maximize coefficient of variation (CV) of response propensities (combines R-indicator and response rate)
Response propensities modelled by age, income, urbanization, type of household, ethnicity
Constraints:
A minimum annual total number of about 9500 respondents was requested
An upper limit of 8000 was imposed to the number of nonrespondents that are sent to CAPI, as a proxy for a budget constraint
An upper limit of persons was set to the sample size
TO DO: Inclusion of constraint on mode-specific measurement bias
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

8. ASD case study – Health Survey/EHIS – optimization
Stratification based on classification tree of web response
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

9. ASD case study – Health Survey/EHIS – optimization
Optimal allocation probabilities of web nonrespondents to F2F follow-up were determined based on mathematical optimization. Per month allocation probabilities are rescaled to guarantee a fixed F2F workload.
CV uniform design = 0.158
CV adaptive design = 0.116
Stratum
RR web
Alloc F2F
RR F2F
RR tot 1
42.6
56,4
48.5
58.1 2
24.8
95.1
47.1
56.4 3
37.6
71.4
42.3
57.1 4
23.1
100.0
25.4 5
39.3
43.5
64.8
57.2 6
36.3
66.5
47.6
57.4 7
19.3
30.8
44.4 8
20.2
38.7
52.3 9
28.9
65.8
58.3
57.0
total
35.9
71.5
42.5
55.7
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

10. Concluding remarks
Adaptive mixed-mode survey design offers a flexible way to balance quality and budget
Holds true especially in sequential designs with more expensive (interviewer) modes as optional.
Further within mode differentiation (timing and number of calls/visits) is possible.
However:
Account of both representation and measurement is crucial;
A flexible case management system and monitoring is required;
Future:
May be combined with re-interview designs (WP2)
May be combined with sensor measurements/data (WP5)
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019