1. NCRM is funded by the Economic and Social Research Council 1 Interviewers, nonresponse bias and measurement error Patrick Sturgis University of Southampton Research Methods Festival, Oxford, 2-5 July 2012

2. NCRM is funded by the Economic and Social Research Council Co-authors Ian Brunton-Smith (University of Surrey) Joel Williams (TNS-BMRB)

3. NCRM is funded by the Economic and Social Research Council Background and Motivation

4. NCRM is funded by the Economic and Social Research Council Common Causes of Survey Error Recent attention has focused on common causes of nonresponse and measurement error (cf. 2010 special issue of POQ on Total Survey Error) Agencies often target field resources at persuading reluctant respondents to meet response rate targets These respondents are less motivated and potentially less able to complete questionnaire accurately 4

5. NCRM is funded by the Economic and Social Research Council Error trade-offs So potential reduction in nonresponse bias my be offset by increase in measurement error Growing evidence that this does happen in practice (Kreuter et al 2010; Sakshaug et al 2010) This work has focused on respondents so far What about interviewers? 5

6. NCRM is funded by the Economic and Social Research Council Interviewers as common cause? Interviewers can cause nonresponse bias and measurement error Success in obtaining contact and cooperation related to interviewer characteristics: –Tailoring –Maintaining an interaction –Personality –Attitudes and beliefs Some interviewers don’t get interviews, where ‘better’ interviewers would If these ‘lost’ respondents are different on survey variables, result is biased population estimates 6

7. NCRM is funded by the Economic and Social Research Council Interviewer variance Interviewers cause measurement error through the way they administer the questionnaire At the individual level, these can be considered biases (response different to true value) But across respondents and interviewers the result is larger variances

8. NCRM is funded by the Economic and Social Research Council Interviewer Variance E.g. interviewer always reads same question incorrectly Across interviews, this creates within- interviewer correlation – same as geographical clustering Interviewer contribution to variance of estimator denoted ρ Int 8

9. NCRM is funded by the Economic and Social Research Council How is ρ Int related to nonresponse? Anecdotal evidence that more successful interviewers on the doorstep are less diligent at sticking to questionnaire wording and instructions Alternatively, some interviewers are good at what they do, others are not so good Either way, we should anticipate a correlation between response rate and ρ Int ?

10. NCRM is funded by the Economic and Social Research Council Conceptual model 1 Implications for total survey error MSE int = bias 2 + variance int Agreeable- ness Response rate Deviation from questionnaire wording and instructions + ρInterviewer Tailoring + + Nonresponse bias VarianceInt + - + - 10

11. NCRM is funded by the Economic and Social Research Council Conceptual model 2 Implications for total survey error MSE int = bias 2 + variance int Conscientiou- sness Response rate Deviation from questionnaire wording and instructions + ρInterviewer Tailoring + + Nonrespon se bias VarianceI nt + - + + 11

12. NCRM is funded by the Economic and Social Research Council Analytical approach Fit cross-classified multilevel models to face to face interview data Partition ρ into area and interviewer components Examine variation in ρ Int across distribution of measures of interviewer success in obtaining contact and cooperation seperately 12

13. NCRM is funded by the Economic and Social Research Council Measuring interviewer success Average response rate problematic as indicator of interviewer success on the doorstep Our measure of interviewer success –Calculate ‘expected’ response propensity for all original issue cases based on geodemographic characteristics and paradata –Take mean of ratio of expected to observed rate across all cases for each interviewer –Do this separately for contact and cooperation –Group into ‘success quantiles’ 13

14. NCRM is funded by the Economic and Social Research Council Modelling strategy Adjusts estimates for clustering of interviews within sample points Models also include individual, interviewer and area controls to account for non-random allocation of respondents to interviewers Cross-classified multilevel models with a complex interviewer error structure Allows simultaneous estimation of separate ρInt for each interviewer success quintile

15. NCRM is funded by the Economic and Social Research Council Data and analysis British Crime Survey (2005/06) –43,465 respondents, 472 interviewers, 3,782 areas 36 items asked of all respondents which were non-factual and included probes and/or show-cards Cross-classified multilevel model with complex error term at interviewer level Controls –Individual - Gender, age, ethnicity, education –Interviewer - Gender, age, ethnicity, experience level (months worked) –Area - Socio-economic disadvantage, urbanisation, ethnic diversity, housing structure, age profile, population turnover 15

16. NCRM is funded by the Economic and Social Research Council 16 Results I: Example Model

17. NCRM is funded by the Economic and Social Research Council Results II: Overall estimates For contact measure bottom quintile has largest variance on 25/36 items (15 at p<0.05) For cooperation measure bottom quintile has largest variance on 20/36 items (13 at p<0.05) For contact, bottom group had 74% higher variance across all items compared to top group For cooperation, bottom group had 34% higher variance across all items compared to top group 17

18. NCRM is funded by the Economic and Social Research Council Mean Interviewer variance components across 36 items by contact and cooperation success quintiles

19. NCRM is funded by the Economic and Social Research Council ITEMS WITH A DOWNWARD TREND ASSOCIATION BETWEEN CONTACT SUCCESS AND INTERVIEWER VARIANCE

20. NCRM is funded by the Economic and Social Research Council ITEMS WITH A U-SHAPED ASSOCIATION BETWEEN COOPERATION SUCCESS AND INTERVIEWER VARIANCE

21. NCRM is funded by the Economic and Social Research Council Are these differences really due to interviewers deviating from the script? Could also arise due to differential nonresponse bias We find no differences across quintile groups on a range of background variables If due to nonresponse bias, should observe uniform gradients over different question types that vary the degree of interviewer involvement An alternative explanation? 21

22. NCRM is funded by the Economic and Social Research Council Results III: by question type 22

23. NCRM is funded by the Economic and Social Research Council Discussion Historical measure of contact and cooperation success negatively correlated with interviewer variance Different pattern for contact and cooperation – contact broadly linear, cooperation evidence of u-shaped distribution Pattern of findings across question types suggest effect is due to interviewer behaviour in questionnaire administration Relatively small group of ‘poor’ interviewers make disproportionate contribution to total survey error Suggests response rate may be used as indicator of potential problems with interviewer behaviour/ training 23

24. NCRM is funded by the Economic and Social Research Council Some extensions

25. NCRM is funded by the Economic and Social Research Council Conceptual model Agreeableness Response rate Deviation from questionnaire wording and instructions + ρInterviewer Tailoring + + Nonresponse bias VarianceInt + - + - 25