1. Detecting and understanding interviewer effects on survey data using a cross-classified mixed-effects location scale model
Ian Brunton-Smith, University of Surrey
Patrick Sturgis, University of Southampton
George Leckie, University of Bristol
7th ESRC Methods Festival, 5th July, Bath, UK

2. Motivation
Interviewers can substantially inflate variance of estimators
idiosyncrasies in administration of questions
interaction of personal characteristics with those of respondents
Induce within-interviewer dependency
Akin to clustering
Standard approach: mixed effects model with random intercept

3. Another source of interviewer error?

4. Mixed effects approach to interviewer effects
Two-level (e.g. individuals nested in interviewers) random intercept model:
where ,
Enables estimation of interviewer ICC

5. Location-scale model where
Extend the standard model by modelling the level-1 variance as a log-linear function of covariates and a further random effect (Hedeker et al., 2008)
Mean function:
Level-1 variance function:
where ,

6. Location-scale model where
Extend the standard model by modelling the level-1 variance as a log-linear function of covariates and a further random effect (Hedeker et al., 2008)
Mean function:
Level-1 variance function:
where , ,
Extended to the cross-classified case to adjust for area confounding (e.g. Vassallo et al., 2016; Durrant et al., 2010)

7. Data and measures
Wave 3 of UKHLS general population sample ( ) n = 17,471
Respondent age and gender
matched to ‘understanding society interviewer survey’ n = 303
Interviewer gender, age, experience, beliefs about the value of surveys, and personality (big 5)
Adjusted for area clustering (MSOA) n = 3,473
Ethnic diversity, socio-economic disadvantage, urbanicity, population mobility, age/housing structure (census)

8. Results
“People in this neighbourhood generally don’t get along with each other” (5-point likert scale)

9. Identifying differences between interviewers
Design effect between 2.5 and 4.9 across middle 95%

10.  Scale equation
 Interviewer effects
Mean SD
2.50%
97.50%
[Intercept]
-0.701
0.112
-0.915
-0.466
Male
0.09
0.051
-0.009
0.191
Age
0.01
0.027
-0.044
0.062
Worked on another survey
0.097
0.048
0.005
0.192
Non-survey interviewing
-0.016
0.049
-0.111
0.08
Public interaction
-0.04
0.057
-0.155
0.073
Survey participation self-interest
0.043
0.047
-0.048
0.135
Surveys conducted responsibly
-0.269
-0.457
-0.077
Surveys correct
0.125
0.085
0.294
Agreeableness
0.012
0.025
-0.039
Conscientiousness
0.035
-0.013
0.084
Extravert
0.058
0.026
0.106
Neuroticism
0.003
-0.049
0.054
Openness
0.014
-0.035
0.065

11. Quantifying the effect
Interviewer scoring 1SD below mean on extraversion, had worked on other surveys, but does not believe surveys conducted responsibly: DEFF = 3.2
As above but 1SD above mean on extraversion: DEFF = 2.4

12. Sense check: A self completion item
“The friendships and associations I have with other people in my neighbourhood mean a lot to me”

13. Setting up the model – a template for Stat-JR

17. Discussion
Interviewer can have effect on variability of responses as well as mean
Location-scale model provides a flexible approach to identify these effects (implemented in Stat-JR)
Can be linked to specific interviewer characteristics to better understand effects
Full paper available: Brunton-Smith, I., Sturgis, P., and Leckie, G. (online) ‘Detecting and understanding interviewer effects on survey data by using a cross-classified mixed effects location-scale model.’ Journal of the Royal Statistical Society Series A.