11 How Much of Interviewer Variance is Really Nonresponse Error Variance? Brady T. West Michigan Program in Survey Methodology University of Michigan-Ann Arbor Kristen Olson Survey Research and Methodology Program University of Nebraska-Lincoln June 14, 2010 International Total Survey Error Workshop 2010

22 Interviewer Variance: The Problem An undesirable product of the data collection process, given interpenetrated sample designs Responses for same interviewer are more similar than responses for different interviewers Leads to inflation of variance in survey estimates due to intra-interviewer correlation, ρ int ρ int = 0.01, 30 cases per interviewer  13.6% increase in SE of estimates ρ int usually less than 0.02, but can be larger

33 Research Question Does ρ int arise from complex interviewer- respondent interactions / probing for hard items? High estimates of ρ int (0.03–0.12) for factual (easy) and self-completion items in literature… There is also consistent empirical evidence of interviewer variance in response rates One estimates ρ int with respondent data only, ignoring contributions of NR error variance How much of interviewer variance can be attributed to nonresponse error variance?

44 The Wisconsin Divorce Study (WDS) SRS of divorce records from four Wisconsin counties in 1989 and 1993 Sampled divorce records included official information also collected in a survey The present study focuses on data collected using CATI: interviewer effects are likely attenuated relative to CAPI n = 733 cases randomly sampled, and 355 CATI interviews performed by 31 trained interviewers

55 WDS Data Six Survey Variables of Interest  Length of Marriage in Months  Time since Divorce in Months  Time since Marriage in Months  Number of Marriages including the Divorce  Age at Marriage  Age at Divorce Date of Divorce was recorded by an official body; other frame measures were reported by one member of the couple (possible errors)

66 Assigning Nonrespondents Ideally, cases would not be worked by multiple interviewers (e.g., Singer and Frankel, 1982) WDS used refusal conversion, and there were frequent changes in interviewers working non-finalized cases This complicates the process of assigning nonrespondents to interviewers

77 Assigning Nonrespondents Assumption: Interviewers working a particular shift work a random subsample of cases The focus of this study is on interviewer variance within a shift, rather than across shifts Persons with different characteristics are likely to be contacted at different times of the day Account for shift: avoids possible confounding of nonresponse error with differences across shifts

88 Assigning Nonrespondents Definitions of shifts:  Weekday, 9-5pm (Shift 1: 26.8% of calls)  Weekday, after 5pm (Shift 2: 44.1% of calls)  Weekend, any time (Shift 3: 29.1% of calls) Similar to work of Stokes and Yeh (1988) Interviewers worked multiple shifts Alternative shifts were also considered, and the study results did not change

99 Assigning Nonrespondents Respondents were assigned to the interviewer completing the interview Contacted refusals were assigned to: 1)the first interviewer receiving a refusal, or 2)the last interviewer to make contact Non-contacts were assigned to the last interviewer making a call to the case Random assignment of non-contacts was also considered; no change in results

10 Assigning Nonrespondents Limited power: interviewers worked each shift, based on assignments Large variability in assigned workloads across interviewers within a shift Between 8 and 15 cases per interviewer within a shift, on average Response rates lowest during the week, and especially before 5pm (41.1%)

11 Analytic Approach Examine interpenetration assumptions Estimate ρ int for each survey variable within each shift, based on respondent data  Test interviewer variance for significance Estimate all variance components of the MSE of the respondent mean (possible with WDS data) Estimate interviewer effects on (and interviewer contributions to) the variance components Compute the proportion of interviewer- contributed variance due to NR error variance

12 Example Derivation (Groves and Magilavy, 1984) MSE of respondent mean (u = # of refusals): Example: Refusal error component

13 Example Derivation, cont’d Estimate variance components using linearized variance estimators  Accounts for clustering due to interviewers and unequal workloads Estimate interviewer effects on variances based on estimates of intra-interviewer correlations in true values (Census Bureau, 1985) 13

14 Example Derivation, cont’d Estimated total contribution of interviewers to refusal error variance: The estimated contribution is a function of intra-interviewer correlations in true values, for respondents and refusals Similar derivations for other components

15 Results: Interviewer Variance Based on Respondent Data Interpenetration evident in each shift Four variable / shift pairs were found to have unusually large estimates of ρ int :  Age at Divorce, Shift 2 (ρ int = 0.08, p = 0.05)  Age at Divorce, Shift 3 (ρ int = 0.10, p = 0.11)  Age at Marriage, Shift 2 (ρ int = 0.11, p = 0.01)  Mths. since Marr., Shift 2 (ρ int = 0.05, p = 0.13)

16 Results: Sources of Interviewer Variance in Age at Divorce (Shift 2) Estimated intra-interviewer correlations in  Response errors:  True values for respondents:  True values for refusals:  True values for noncontacts: Total estimated variance of R mean: Total estimated variance contributed by interviewers: Additional variance contributed by interviewers is due to intra-interviewer correlations in true values for respondents!

17 Additional Results Similar findings for age at divorce in shift 3 Response error variance was main contributor for age at marriage in shift 2 and months since marriage in shift 2 Response error variance may arise from outliers, as shown in the following graph

18 Illustration of Variance Sources

19 Conclusions Interviewer variance on key survey variables may arise from nonresponse error variance among interviewers Interviewers may successfully obtain cooperation from different pools of respondents (e.g., older vs. younger) Liking theory could be one explanation: variance in interviewer ages, voices  variance in respondent ages (F. Conrad)

20 Implications for Practice / Future Work Monitoring Strategies: managers can continuously compare available features of R and NR for each interviewer, and intervene when large differences arise Findings need to be replicated in a face-to- face setting with interpenetrated subsamples assigned to interviewers Access to interviewer features would also enable use of multilevel modeling

21 Thank You! Bob Groves, Mick Couper, Frauke Kreuter and Paul Biemer have provided very helpful feedback and comments Thanks to Vaughn Call for providing access to the WDS data Please for these slides or a draft of the