1. Mode bias/mode effect and its adjustment: General Overview WP2
ROME
April 11th | 12th
MIMOD
Mixed-Mode Designs for Social Surveys
FINAL WORKSHOP
Mode bias/mode effect and its adjustment: General Overview
WP2
Orietta Luzi
Istat

2. Mode bias/mode effect and its adjustment: General Overview
Session organization
1) General overview (Orietta Luzi)
2) Key issues and main decisions in mode effect detecion and adjustment (Orietta Luzi)
3) Re-interviews – a case study (Barry Schouten)
4) Experimenting methods to assess and adjust mode effect when a single mode control survey is available as a benchmark: a case study (Claudia De Vitiis)
5) Discussant (Pawel SZymankiewicz)
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

3. Mode bias/mode effect and its adjustment: General Overview
Involved Countries
ISTAT Italy
Orietta Luzi (coordinator)
Claudia De Vitiis, Francesca Inglese, Roberta Varriale, Alessio Guandalini, Marco Terribili
CBS Netherlands
Barry Schouten, Bart Buelens, Jan van den Brakel
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

4. Mode bias/mode effect and its adjustment: General Overview
Main objectives and performed activities
1) Updated review of methodologies to:
Assess mode effects and evaluate the comparability of data collected by different modes (assessing the measurement equivalence)
Deal with mode effects to ensure accurate results by properly estimating mode bias/mode effect
2) Evaluation of the suitability of selected methodologies to deal with selection errors and to adjust for measurement errors in current MM surveys
Re-interview designs
Selected methods at the estimation phase
3) Production of evidence-based guidelines for the use of methods to deal with mode effects in MM surveys, with a discussion of assumptions, advantages and disadvantages of the most common approaches
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

5. Title of the deliverable
Mode bias/mode effect and its adjustment: General Overview
Key outputs/deliverables #
Title of the deliverable
Overall Contents
Author 1
Current methodologies to deal with mode effects and mode bias in mixed-mode designs
Overview on current methodologies adopted at the ESS NSIs to deal with mode bias/mode effects in MM designs (literature review & query)
CBS 2
A cost-benefit analysis of re-interview designs for mode-specific measurement bias
Results of the analyses performed on re-interview designs 3
Experimenting methods to assess and adjust mode effect when a single mode control survey is available as a benchmark: a case study on the Italian Aspects of daily life survey
Results of the applications of selected methods at the estimation phase on MM social surveys
Istat 4
Methodological Report
Summary of all the performed analyses.
General guidelines on the methodological solutions which can be adopted to deal with mode effects in MM designs
Istat&CBS
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

6. Mode bias/mode effect and its adjustment: General Overview
Overview on current methodologies to deal with mode effect in MM surveys
MIMOD survey
Specific section of the questionnaire devoted to collect information on methods/strategies currently adopted in the ESS Countries for mode effect assessment and adjustment
Recent literature review
Updated overview on methodologies for mode effects assessment and adjustment in MM designs
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

7. Key issues and main decisions in mode effect detection and adjustment
MIMOD survey: main results on methodologies to deal with mode bias/mode effects (1)
Objective
Activity undertaken
% countries
Assessing mode effects
Pre-tests, experiments on questionnaire design
48 %
Pilot surveys
42 %
Differences in distributions of socio-demographic or target variables
39 %
Differences in quality indicators (e.g. total or item non response rates, break-off rates, …)
35 %
Pre-tests, experiments on sensitive or core questions
Previous and new data collection strategies running simultaneously (independent sampling)
32 %
Separating selection, nonresponse and measurement effects
26 %
Calculation of representativeness indicators of various designs
23 %
Pre-tests, experiments on split sample approach
19 %
Subsampling of groups receiving different data collection strategies (e.g. control group)
Pre-tests, experiments on the use of different devices (smartphones, tablets, …)
Re-interview studies
6 %
Other types of pre-tests and/or experiments
3 %
Other activities
No activity conducted in recent years
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

8. Key issues and main decisions in mode effect detection and adjustment
MIMOD survey: main results on methodologies to deal with mode bias/mode effects (2)
Objective
Activity undertaken
% countries
Adjusting for mode effects
Weight adjustments
26 %
Calibration to fixed mode distributions
13 %
Estimate measurement errors and correct responses to a benchmark mode
10 %
Other
No measure taken
61 %
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

9. Mode bias/mode effect and its adjustment: General Overview
Overview on current methodologies to deal with mode effect in MM surveys: Main evidences from the MIMOD survey
31 responding Countries
Assessment of mode effects:
Most of the activities aim to assess the total mode effect
About 32% did not conduct any assessment activity in MM social surveys
Adjustment for mode effects:
About 60% did not conduct any adjustment activity in MM social surveys
Future plans:
About 50% of Countries reported to have future plans for research in into mode effect
Most of theese plans focus on assessment and to a lesser extent on adjustment
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

10. Mode bias/mode effect and its adjustment: General Overview
Overview on current methodologies to deal with mode effect in MM surveys: Main evidences from the Literature review
Mode assessment studies are often limited to quantifying the total mode effect
Accordingly with the MIMOD survey, literature review highlights that mode effect assessment methods are more widespread than mode effect adjustment techniques:
Adjustment methods require mode effects to be separable into selection and measurement effects and this is not always done in the analyzed literature
An important reason is that it is difficult to separate selection from measurement effects, but easy to assess their combined effect. The two effects are said to be confounded
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

11. Key issues and main decisions in mode effect detection and adjustment
ROME
April 11th | 12th
MIMOD
Mixed-Mode Designs for Social Surveys
FINAL WORKSHOP
Key issues and main decisions in mode
effect detection and adjustment
WP2
Orietta Luzi
Istat

12. Key issues and main decisions in mode effect detection and adjustment
Methodological strategies to deal with mode effect: key issues
MM is used to contrast declining response rates and coverage and to reduce the total survey costs
Main methodological drawbacks of MM:
Difficulty of controlling over mode effects and their biasing effects on survey estimates
The confounding of selection and measurement effects
Mode effect refers strictly to measurement differences due to the mode of survey administration
The selection effect, due to the differences in the distributions of respondents to the different modes
Especially in repeated surveys, estimates must be consistent and comparable, for ensuring that changes in the time series are exclusively due to real changes of the observed phenomena
Appropriate methodological strategies are necessary to properly assess and adjust mode effects to ensure accurate and consistent estimates
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

13. Key issues and main decisions in mode effect detection and adjustment
General “guidelines”
Methodological strategies to deal with mode bias/mode effect: key decisions
Methodological strategies to deal with mode bias/mode effect : key elements
Methods to deal with mode effects
Assessing and adjusting mode effects: some key aspects
Check-list for the design of a strategy to deal with mode bias/mode effects
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

14. Key issues and main decisions in mode effect detection and adjustment
Methodological strategies to deal with mode bias/mode effect: Key decisions
Deciding if and how to estimate mode effects and/or to adjust for their biasing effects
quality criterion (e.g. the MSE) against a cost limit: how to assess whether mode effect adjustment is beneficial?
multi-dimensionality of a survey: what key estimates and population parameters of interest need to be evaluated?
time perspective: is the survey repeated, can effects be assumed constant, which actions are planned for future survey repetitions?
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

15. Key issues and main decisions in mode effect detection and adjustment
Methodological strategies to deal with mode bias/mode effect: key elements
Main requirements to be defined:
A “design” to control for mode bias/mode effects
Appropriate auxiliary variables (from administrative data/frame data/paradata) referred to as covariates that are mode-insensitive and informative about
Mode selection
Mode measurement
A set of assumptions (depend on the type of auxiliary data and type of design)
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

16. Key issues and main decisions in mode effect detection and adjustment
The design: strategies to control for mode bias/mode effect
Experimental designs allow controlling for selection effects, and hence the unbiased assessment of measurement differences between modes. Experimental designs are rather “rare” because of costs.
Observational studies require covariates that explain the selection mechanisms. If available, differences between mode groups are attributed to measurement differences, conditional on the (error free) covariates.
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

17. Key issues and main decisions in mode effect detection and adjustment
The design: strategies to control for mode bias/mode effect - A general scheme
Design type
Objective
Experimental
Re-interview study - repeated measurement designs (Deliverable 2)
To assess mode effect
To adjust for mode effect
Parallel independent designs - single mode and mixed mode samples - (Deliverable 3)
Other (Embedded experiments, Split sample designs)
Non-experimental
Observational studies (mixed-mode design only)
To control for selection effect
To adjust for measurement effect (Required auxiliary variables)
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

18. Key issues and main decisions in mode effect detection and adjustment
Methods to deal with mode effects
Reference schemes that outline the methods which can be applied for different:
Survey/experimental contexts
Objectives of the study
Analysis of total mode effect (1)
Analysis to disentangle measurement and selection effects (2)
Adjustment for selection and measurement effects (3)
Where appropriate, additional elements about requirements, assumptions, advantages/drawbacks
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

19. Assessing total mode effect (1)
Key issues and main decisions in mode effect detection and adjustment
Methods to deal with mode effects
Assessing total mode effect (1)
Objective of study: Assessing differences between estimates based on data collected through different survey designs (single-mode and mixed-mode), in order to evaluate the total mode effect and the measurement equivalence
Method
Analysis
Context / Conditions
Regression modelling (Martin and Lynn, 2011)
Univariate analysis of items to evaluate the impact on marginal distributions
Parallel independent surveys
Appropriate statistical models and tests ……
Multi-group confirmatory factor analysis
(Martin and Lynn, 2011; Hox et al., 2015)
Analysis of the measurement equivalence
Mixed mode survey design
Identification of the latent structure of the phenomenon, Selection effect control
Schemes - examples
Objective of study: Analysing the response processes and evaluation of the bias caused by the total nonresponse (selection errors)
Method
Analysis
Context / Conditions
R-indicator
(Klausch et al., 2015; Schouten et al., 2011; Shlomo and Schouten, 2013; Schouten, et al., 2017)
Analysis of the representative response/absolute selection error
Parallel independent surveys
Single and mixed mode designs
MAR assumption for Response model ……
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

20. Disentangle measurement and selection effects (2)
Key issues and main decisions in mode effect detection and adjustment
Methods to deal with mode effects
Disentangle measurement and selection effects (2)
Objective of study: Assessing mode effect - disentangling measurement and selection effects
Method
Analysis
Conditions
Context
Weighting
(Vandenplas et al., 2016; Rosenbaum and Rubin, 1983; Vannieuwenhuyze, et al., 2014)
Analysis based on response model to control for respondent characteristics (comparable samples in MM)
MAR assumption
Mode-insensitive auxiliary variables
Mixed mode survey designs (observational studies) ……
Instrumental variable approach
(Vannieuwenhuyze et al., 2010)
Analysis based on benchmark single-mode design
Validity of comparability and representativity assumptions
Parallel independent surveys
Re-interview
(Biemer, 2001)
Analysis based on re-interview data, administrative data and paradata
Re-interview does not affect measurement behavior of respondent
Re-interview of subset of mixed-mode respondents
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

21. Key issues and main decisions in mode effect detection and adjustment
Methods to deal with mode effects
Adjust for selection and measurement effects (3)
Objective of study: Adjustment methods
Method
Data requirements
Assumptions
Advantages/Disadvantages
Standard Covariate-based adjustment 
Sampling frame data
Paradata
Survey responses 
Missing at random potential outcomes (MAR)
Exogeneity of auxiliary data
Too strong assumptions in many settings (-)
Adjustment on individual level possible (+) ……
Re-interview method
(Klausch et al., 2017)
Objective of study: Adjusting selection/measurement effects in MM (observational studies)
Method
Aim
Conditions
Weighting
To equate samples
To correct selection effect
MAR assumption, Mode-insensitive auxiliary variables, Measurement error negligible ……
Multiple imputation
1.Multiple (standard) imputation
To predict counterfactual data
To correct measurement error
Benchmark mode, MAR assumption
2. ……
Sequential design and two modes, benchmark mode, also NMAR assumption
3.Fractional multiple imputation proposed by Park et al. (2016)
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

22. Key issues and main decisions in mode effect detection and adjustment
Assessing mode effects: key aspects
Assessments are most sensibly conducted with respect to a benchmark
In assessment studies, the representativity of respondents, the response rates etc. can provide insight into the selection mechanism. Selection effect is a desirable effect of MM strategies as it could reduce selection bias on survey estimates
Assessment of total mode effect it is relatively easy and sometimes can be sufficient, but when detected, undesired mode effects need to be properly estimated and adjusted for (measurement effects)
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

23. Key issues and main decisions in mode effect detection and adjustment
Adjusting for mode effects: key aspects
Adjustment methods are necessary to correct estimates for undesidered mode effects
Appropriate adjustment methods require the effective separation of selection and measurement effects.
Disentangling and estimating mode effect components can be a difficult task
It requires that covariates are available which explain the selection mechanism and which are assumed mode-insensitive and informative on mode selection/mode measurement
Both assessment and adjustment strategies are most reliable and less dependent on assumptions when conducted in experimental settings
When applying adjustments, a reference mode has to be chosen as benchmark
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

24. Key issues and main decisions in mode effect detection and adjustment
Check-list for the design of a strategy to deal with mode bias/mode effect
Identify the main quality and cost criteria
Decide whether mode effect estimation serves explanation only, design choice or adjustment
Identify available auxiliary data that is informative about mode selection/mode measurement
Evaluate anticipated validity of assumptions for mode selection, mode measurement and absence of experimental influences
Decide whether an experimental design (such as re-interview or parallel run) is required and feasible to serve the purposes of the mode effect estimation
Conduct experimental designs if deemed feasible and necessary
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019

25. THANK YOU FOR YOUR ATTENTION
Key issues and main decisions in mode effect detection and adjustment
THANK YOU FOR YOUR ATTENTION
MIMOD project - Mixed-Mode Designs in Social Surveys
Rome, April 2019